Thoughts of a new(ish) Software tester

Leave a comment

Work Smarter Not Harder

Do you ever feel like you’re under pressure as a tester?  Are you comfortable with the amount and quality of testing that features undergo?  Do you feel like corners are being cut?

This may not be an issue with your people but things may improve if you changed some of your processes.

There are several ceremonies (for want of a better word), which, when executed properly, should result in more reliable software being released to customers.  Carrying out these tasks may take longer in the short term, for example, a story may take 5 days to be finished rather than 3 or 4.
However, in the long term, this should result in customers giving a lot less (negative) feedback about those features.

Some of the resistance to this way of working comes from our innate desire to have everything now, or even yesterday or last month.  So it can be hard to change the short term mindset from ‘We can give this to our customers today’ (and sweep the risks under the carpet), to ‘Lets spend an extra week on this, and make sure it’s of higher quality and minimal risk’.  The first mentality may keep your customer happy for a few weeks, until the edge cases start to creep in and come back to bite you as customer complaints. The second option may frustrate some customers in the short term but you will reap the long term rewards of happy customers and a priceless reputation for producing quality software.

Described in these terms it looks like a no-brainer to choose the second approach.  However, when you’re in the heat of battle, so to speak, with product managers and customers begging for features now, it can be hard not to succumb to the pressure.

To put into context what I’ve talked about, the practices we incorporate into our development cycle are very briefly described below.  A lot of people will say ‘We already do that’ but complacency is the enemy of progress, so it does no harm to constantly evaluate whether you are really sticking to these positive behaviours.

Sprint Kickoff – So everyone, including developer, testers and product managers, can start thinking about upcoming work

Story Planning – To get realistic estimations so customer expectations can be set.  Formulate test ideas to share with the Developers

Team Huddle (Developers and testers get together) – So the whole team can sing from the same hymn sheet on requirements and acceptance criteria before any coding starts.  Specification By Example tests in the Given, When, Then format should be written in readiness for automation.

Automation – Using Specification By Example
You really need to have your developers on board for automation to become a success.  Done properly this should significantly reduce the manual testing required once this reaches the tester. That way you can have confidence the majority of tests have been covered and can focus your attention on edge cases and Exploratory Testing

Demo to Test – Before committing their code the Developer should demo the feature on their machine if possible and allow the tester to test on it.  That way any bugs can be dealt with and fixed before the software reaches the test environment.  When bugs are caught early this can save a lot of time in multiple deploys.

Exploratory Testing (ET)  – This should not be the exception, but the rule for the tester.  If all the tasks above have been successfully completed the testers main form of testing should only need to be the most interesting – ET

It is our job as testers to champion and promote the practices above.  If executed well, it should make all of our lives easier, especially the customers.


Leave a comment

Purposeful Regression Testing

Regression testing is a term that all testers ought to be familiar with but does it mean the same thing to all of us?

My view of regression testing has gradually been changing since I started as a software tester 8 months ago.  When I first started out one of the first things I was introduced to was the regression tests we run for every new kit.  This was a good place to start as it helped me build up my knowledge of the product and also get familiar with quite a wide coverage of tests.  With each run of the regression tests I became more and more confident and found I could run through the tests quicker each time and with less of a need to read the test steps word for word.  Although this was all good and positive I could also feel myself getting a bit too comfortable and did not feel I was putting enough thought into my regression testing.

It felt as though this testing was gradually turning into checking but at the same time I thought this must be the nature of the beast.  However, running the same manual tests time after time is not what I’d call fun.

So what can we do to change things?
Of course, automation is one tool we can use to reduce the number of regression tests we run manually. Automation is especially useful when the tests are literally run in the same way following identical steps every time and the results are simple enough for a computer to understand.
Some of the tests cannot be automated, or if they were automated they could not be interpreted reliably by a computer.  This may be due to timing issues in running the test or audio quality assessment of the output which we cannot (yet?) trust a computer to make a valid judgement on.

However, as well as these special cases, I believe that there will always be fundamental manual tests that must be carried out by a human to ensure confidence that the kit has the basics right, before moving on to the ‘more interesting’ testing.

Every tester knows that it is not possible to run every test variant on a product of any reasonable size, let alone get close to this with every set of regression tests.  For this reason regression tests need to be carefully chosen to be of maximum use specifically to the kit in question.  To continue to be useful I believe several factors need to be taken into account when selecting what tests to run as part of a regression suite.  I would include the following in my areas for consideration:

  • Recent new product features – Areas of the product which have recently been created.  Even though they may have had a thorough testing, in general, I would still want to focus on a new area rather than an older one.  I would expect more bugs to be found lurking there as they are likely to have had less exposure to as many different variants of input as opposed to more established parts of the product.
  • Customer impact – If I had to choose between testing only area A and area B and was told both were of equal size and that area A was considered essential to 70% of customers and area B only important to 30% I would choose to test area A.
  • Time – We can obviously only perform a finite amount of testing within whatever time frame we have.  So, looking at the example above, in simple terms if I had 100 minutes to test I would spend 70 minutes on area A and 30 minutes on area B.
  • Bug fixes – Even when bugs have been fixed and successfully retested I still like to focus on these areas as they are effectively newer areas of code now so the same principle applies to that of new feature areas.
  • Refactoring – Again for the same reasons as new features and bug fixes.

Targeted and meaningful regression testing should include the core tests which cannot be automated but also take into account all of the factors above.  The examples above are obviously very simplified so deciding what to test and for how long is not straightforward.  I think part of the skill of being a tester is to weigh up all the factors and choose one’s tests wisely rather than blindly following a regression test script.  I’d like to think that’s how regression testing is done in the majority of companies but I fear it may not.

Leave a comment

April 2012 – Time is Precious

One of the natural consequences of working in software development is that requirements can change at any given time.  Because we’re an Agile team working in an Agile environment this is not a big problem as it means we are prepared and able to react quickly to the changing demands of management and customers.  If we were working in a Waterfall environment (which I’ve heard about but not experienced for myself) I imagine we would find it a lot harder to adapt to such short notice changes.  Therefore, we are not always in control of how much time we will have to carry out development and testing.  I don’t see this as a negative as it just means we have to be adaptive and flexible.  I have also found that it brings out the creativity in the team and makes for a challenging and rewarding experience.

On a team level and also on a personal level this means that we have to make the best use of the, sometimes limited, time we have.  In an ideal world customers would work to dates we could set them and not mind if the dates kept slipping out so we could take our time to create even more polished applications.  Obviously, in reality, this would never be the case as customers want to move fast and beat the competition to market.  They want to have use of a product we can provide them before they find a similar product created by one of our competitors.

To achieve this we must have all areas of the company working together to meet the customers requirements.  When this happens and we all pull together to realise what the customer wants and when they want it, it is very satisfying knowing that we are driving the company forward by meeting customer expectations and improving our reputation.

One of the conclusions I am coming to is that there’s nothing like a deadline to focus the mind.  I’m not thinking of this in terms of cramming everything in, such as doing your revision last minute on the night before the exam.  But rather planning what needs to be done today, what needs to be done this week, and also what contingency plans do we have if requirements change.  I don’t think deadlines necessarily need to be dictated by customers either.  When the implementation of a piece of work is not directly customer dependent it should not mean that we rest on our laurels until the next ‘real’ deadline comes along.  If we can commit to a target date for all tasks undertaken then we should be able to have a gauge, at least in our minds if not visible to the team, of how our tasks are progressing in relation to the deadline we have committed to.  Commitment based tasks are something we are considering where I work so the results should be interesting.  The theory is that our time management and work rate ought to improve as we have a defined static target to aim at.

Leave a comment

March 2012 – Phases of Testing

As it becomes more instinctive for me to decide what part of our application to test next, e.g. smoke testing a new kit, what bug fixes to concentrate on, or new story features to explore; I can start to focus more attention on exactly how I’m going about my testing.  This includes the preparation/planning, the main phase of testing itself, and the reporting of my findings afterwards.

I have started to get into the habit of using Mind Maps (XMind) to get the main topics down in black and white.  Although it’s a very straight forward premise I find it very useful as I can easily forget even the brightest idea pretty quickly if I don’t write it down.  Mind Maps also double as useful information for future reference.  So anyone who wishes to test a feature they haven’t seen before can view the Mind Map for test ideas to get them started. I find them of particular use when testing a brand new feature or area of the product with which I am less familiar.

When I first started testing I was paying more attention to what was going on in front of me on the screen rather than making notes of what I saw.  This was very bad practice as memory tends to be a lot less reliable than a written word.  The smallest changes in steps to reproduce a bug or unusual behaviour will mean you can no longer produce the desired result.  However memorable it seems at the time there is no substititute for recording your findings.  I have now started to try and make an effort to make notes when I test now (using Notepad++), especially when testing something new in an exploratory way as it could prove invaluable when trying to recall steps in producing any bugs found.
I also tend to use Firefox a little more than other browsers so that I can have firebug running in the background.  Then I can include the firebug error output along with my description when I make a bug report, which can sometimes be useful for developers in pinpointing the origin or cause of the defect.
Screen grab tools and videos are also very useful to back up a description and can often make things a lot clearer than a written description alone in visualising what’s gone on.

Under this heading I include retesting in the sense of Regression testing and also Re-Testing bugs which are believed to have been fixed.
When I run regression tests I do not make as many notes since the paths followed are already well trodden and the steps taken are already recorded step by step.  However I use the same methods for recording findings if a bug is found whilst running regression tests.
When Re-Testing bug fixes it can be very easy to fall into the trap of only testing the ‘happy path’, or the scenario in which the bug was originally found.  It’s also important to test around the edges of the happy path and test the bug scenario with similar but not identical variations in order to be confident the bug(s) have been fully fixed and will not resurface later on.  The number of different paths you can test is obviously time dependent but the more you have time for the better.

Reporting Test Findings
Once testing is complete it is important to quantify what areas of the product have been tested and in what way.  This is especially important when describing how to reproduce a bug as if this is not accurate then the developers may not be able to reproduce it for themselves.  The steps to reproduce are also important when the bug is fixed and you need to repeat the original steps used to create the bug.  You must also make sure you include all relevant setup information such as the browser which was used, the kit version, user credentials, scenario, and any accompanying event logs.  This will give the developers the best chance of identifying and reproducing the bug and reduces the chance they will need to come back to you with further questions.  It should also minimise the chance of the hearing the well known ‘It works on my machine’ response.


Leave a comment

February 2012 – Why I made the right choice

I have now been a Software Tester for two months. It seems like only yesterday I started, as the time has gone so fast, but at the same time, I have probably learnt more in the last two months than I did during the last year at my previous job. This says a lot about the difference between my previous employer and my new employer and how they make me feel about going to work.
I think the word Agile probably sums up the difference. Although this is generally a Software Development methodology I think the values below (taken from the Agile Manifesto) can and should apply to multiple industries:Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan

I think the list above highlights quite accurately why the way things are done where I work now make much more sense than the monotonous planning, processes, documents and contracts my former bosses seemed disinclined to get us away from.
Where I now work there is a lot of emphasis on learning both through colleagues but also by self driven education. When there is a positive atmosphere where everyone wants to learn and use their knowledge to help others it instills in me a desire to improve my own knowledge outside of work. I think it helps that Software Development and Testing are such a vast and ever expanding area which the world is becoming increasingly reliant upon, as companies go online and start using the cloud, rather than the high street, to sell their wares. In hindsight I probably would have gone down the Computer Science rather than Sports Science route at University.
As a tester, one of the main lessons I’ve learnt so far is that there are some core aspects of working in a Test team which must be right to allow a smooth and efficient delivery of work. For me, two of the most important aspects are communication and time management.
Communication so that everyone has a good idea what other individuals in the team are working on and therefore tasks are not duplicated. Also, to be aware of what other teams are working on and how that may affect what your own team is doing. We achieve this through daily team stand ups followed by a scrum of scrums stand up. This means that every individual need not listen to every other individuals update which saves a lot of valuable time.
It really helps that we sit in very close proximity to the developers so that as soon as a bug or issue surfaces we can immediately grab one to demonstrate the problem and they can look into it straight away. Sometimes bugs are known about and a fix is already in the pipeline, or they can be put down to a limitation with the system or feature which hasn’t been built yet rather than a fault in the software. I’m sure as time goes on I will come to learn more of these possibilities and will become one less bug for the developers myself.
I’ve also learnt that effective time management is very important in testing as time can very easily be eaten up by going off on tangents. By this I mean you have to be focused on the task at hand, whether that is testing a bug is fixed, testing a new feature or exploring a new area of the software. In any of these situations you can easily spot a bug or unusual behaviour and decide to investigate a little further. Before you know it you have lost half an hour and wonder where it went. This can have a doubly negative effect in that you have lost focus on your original task and you are probably not 100% focused on the new quirk you have found as you know you have digressed and ought to get back to what you should be doing. Now, when I spot something new I will make a quick note of it as something to look into later when I can allocate time specifically to it, as tempting as it may be to pursue at the time of discovery.
After two months Testing I really feel part of the Development team and appreciate the way the team works. I’ve learnt a lot and am sure I won’t stop learning whilst I’m still a Software Tester. I’m quite sure I have not written anything revelationary as yet. Hopefully over time my posts will offer more insightful views and I’ll be able to talk more about experiences that other testers may value.