It doesn’t matter how your company is run, this percentage can still be obtained. You could be running your IT (Information Technology) department agile, scrum, straight SDLC (software development life cycle) with rules and regulations, you could have Unit, QA, or UAT (User Acceptance Testing). It does not matter, you just work with what you have and if something in the calculation does not apply to your team, just omit it or replace it with what process your team does use.
How to make the calculation: First and fore most you must find the number of ‘true defects’ that made its way into your PRODUCTION environment. True defects are those sneaky little devils. If you have no formal forms of testing and no customer support ticket system, then you will just have to count the number of customer complaint phone calls about the release and bounce this off already existing known issues and this will be your number of ‘true defects.’
If you have some signs of formal testing and no customer support ticket system, you will have to identify with the testing defects that were released into production and have not been previously logged elsewhere. This would be your number of ‘true defects.’
Some points to consider while reviewing the defect and issues logs:
Hopefully now you can understand the complexity of trying to figure out if a defect is a ‘true defect’ or not. Even if you have fields in your bug tracking system to mark defects as such, you can’t rely on if it was filled out properly or did someone change the value later on down the road? This is why you need to have familiarization with your application and heavy knowledge on defects to even get close to an accurate percentage of effectiveness. Otherwise you are just throwing all your open defects opened after ‘this date’ into the calculation and you will be providing an inaccurate percentage. I’ve seen it happen time and time again.
Now that we have spent days shifting through the logs and have our accurate number of ‘true defects’ we can now begin the calculation for the percentage of effectiveness.
100 – [Total # of ‘true defects’] = ______ % of testing effectiveness.
If you have over a 100 ‘true defects’ use the following formula:
1000 – [Total # of ‘true defects’]= ______ % of testing effectiveness.
10
When to make this calculation?
A real life example: For this example we will use both a defect bug tracking tool as well as a customer complaint tracking tool. The Release number is 9.
| # | Testers Defects | Status | True Defect? |
| 001 | Release 9: Feed Cat page – Cat Food Type DDLB displays incorrect options | Open | YES |
| 002 | Release 9: Feed Cat page – Highlight the word ‘cat’ in header | Assigned | YES |
| 003 | Release 9: Sign up page – Highlight the word ‘cat’ in header | Open | YES |
| 004 | Release 9: Sign up page – Can’t see text on first button | New | YES |
| 005 | Release 9: Display page – Req. state 3 columns, we have 2 | Open | YES |
| 006 | Release 8: Display page Regression – Login page button display issue | Closed | NO – because it’s from a prior release and the defect is closed |
| # | Customer Care Call Log | Status | True Defect? |
| 001 | Release 9: Feed Cat page – Cat Food Type DDLB displays incorrect options | New | NO – duplicate from QA |
| 002 | Release 9: Feed Cat page – Highlight the word ‘cat’ in header | New | NO – duplicate from QA |
| 003 | Release 9: Sign up page – Highlight the word ‘cat’ in header | New | NO – duplicate from QA |
| 004 | Release 9: Sign up page – Can’t see text on third button | New | YES – QA has not found this one |
| 005 | Release 9: Display page – Wants to add a new forth row | New | NO – Customer Enhancement Request |
| 006 | Release 9: Display page – Login page header issue | New | YES – QA has not found this one |
So from both logs we have a combination of 7 ‘true defects’ and if we plug that into our equation we would get a 93% testing effectiveness rating for Release 9.
QA Soft-e 