This is a question which is often overlooked when testing. The first thing you would normally consider when you get a new feature is what tests you will run and in what areas. Why bother thinking about what you won’t test?
My answer is stated in the context of carrying out exploratory testing for new features, as opposed to scripted or automated testing. In scripted testing, the test cases are usually explicitly written (normally beforehand), have an expected outcome and a check-box to tick. Once the tests have been written, there is no need to think about what and what not to test, or how to test it. The ‘Tester’ (checker) can simply follow the instructions without diversion and tick or cross their boxes; their mind remaining unexpanded.
This discussion also excludes automation (a bit like a computer running the human scripted tests). I would expect basic happy path and failure (sad path) cases to be automated.
This leaves Exploratory Testing. This is where the world is the testers oyster and they can let their creative juices flow (this is why we all love testing right?). There are no limits as to what he or she chooses to test. It is up to the tester to use their mind (expanding all the while) to work out what to test, and what not to test.
This question can’t be properly answered unless we get the answers to some further questions.
- What are the acceptance criteria? (most if not all of these should already be covered by automation)
- What part of the product is being changed? (and what other parts of the product are touched by this area?)
- What is the deadline, i.e. how long can you spend testing? (If time is very limited you might want to stick to risk based testing and think about the most fundamental areas first, i.e. those which would impact the customers most if they did not work)
- What level of quality (for want of a better word) are customers expecting? (Minimum Viable Product (MVP) or super shiny finished product?)
- Are 3rd parties involved? (i.e. does the code interact with 3rd parties and does your integration with those need testing?)
- Are there dependencies? (e.g. does the new feature need to be backwards compatible with older versions of the product)
Talk to the developers to get their opinion of what parts of the product their code is likely to impact. Don’t forget to take what they say with a pinch of salt (https://patterson2a.wordpress.com/2012/07/01/take-it-with-a-pinch-of-salt), as they may be adamant that you can disregard a certain area of the product (they may also be adamant their code never contains bugs!)
Ask the developers to show you the code and the tests they’ve written. You are unlikely to understand it as well as they do but it shouldn’t do you any harm in seeing it, even better if they can talk you through it. It may lead onto more test ideas and help answer the question of what not to test.
One method I like to use is to create a mind map for the story or feature (I try to pair on this with a fellow tester using mindmup.com) The mind map normally contains areas of the product I am considering testing for the story and I may include some questions. I then share this mind map with the rest of the team (developers, testers and product manager) asking for feedback. Preferably this happens before any development (coding) has begun. I want feedback in regard to whether they feel (in their opinion) the areas I’ve mentioned are appropriate for testing or whether any areas I’ve mentioned have no bearing on the story and I could consider not testing them.
Once you have all this information it is up to you to decide what you will and will not test. There is no secret formula but at least you can make a decision armed with information which will aid your decision.