Reading a QA case study: what is checkable
- Find out who is speaking.A figure in the vendor's own text is a vendor claim; a figure in a client press release is a client claim.
- Check whether the client is named.A named client allows a reference call; an anonymised client caps what the case can prove.
- Look for the same figure outside the vendor's site.A client statement or press article that repeats the metric is what this site marks verifiable yes.
- Ask what the figure was measured against.A reduction needs a baseline and a method; without them a percentage cannot be compared with another case.
- Note the dates and the duration.A result that held for one release differs from one that held for a year, and undated cases hide that.
- List what the case leaves out.Team size, cost and scope boundaries are usually absent; listing them shows what to ask next.
- Treat outlier figures as questions, not answers.A figure far above the rest of the data set is printed here with a label and never used in a summary.
Download this checklist
The checklist file holds the 7 steps above, one per row, with a column for your notes on each vendor.
reading-a-qa-case-study-what-is-checkable-checklist.csvCSV1 KBCC BY 4.0What the data on this site shows
- Across the 95 cases analysed on this site, 413 metrics were recorded: 0 verifiable yes, 45 partial and 368 no.
- 13 of the 95 cases name the client; 0 have a client statement or press confirmation.
- The gaps found most often: date of the work (90 cases), cost (83 cases), measurement method (75 cases).
Published by the editor of retailqavendors.com. Last reviewed .