QA Madness and an anonymised online retailer: Magento functional testing case analysed
Verdict: what is checkable
The evidence grade (this site's label for how a case is sourced: primary with client or press confirmation, secondary when vendor-hosted only.) is secondary: the only source is the vendor's own page; of 4 metrics, 0 are verifiable yes, 0 partial and 4 no. [ev-1187]
Client profile as published
- Client
The UK merchant owner of two e-commerce websites
(the vendor's description; the vendor does not name the client)- Segment
- Ecommerce and D2C (direct to consumer: a brand that sells its own products online to shoppers without a retailer in between.)
- Region
- United Kingdom
- Channels
- ecommerce
- Size indicators
two e-commerce websites running on Magento 1 and Magento 2 platforms
The client description and size indicators are quoted from the source; segment, region and channels are this site's classification. [ev-1187]
Systems in scope
- Ecommerce platform (Magento 1 and Magento 2): named in the Overview section
Where the source names each system is quoted or summarised from the case page. [ev-1187]
Problem as stated
A UK merchant ran two web stores, one on Magento 1 and one on Magento 2. It had its own development team but no in-house QA, and wanted a QA team to check the quality of both stores. The vendor was asked to plan and run testing across both sites. [ev-1187]
it lacked in-house web quality assurance(quoted from the source)
Approach
- Testing types
- functional (main scope: functional)
- Team shape
- not stated
- Duration
- Our cooperation started in 2017 and had been ongoing for the next two years.
- Engagement model
- not stated
- Tooling named in the source
- none named
Metrics claimed
| Metric | Value | Unit | Claimed by | Verifiable | Evidence |
|---|---|---|---|---|---|
| test cases planned for both stores | nearly 800 | test cases | vendor | no | ev-1188 |
| bugs detected over the engagement | nearly 1050 | bugs | vendor | no | ev-1189 |
| conversion rate increase | 4.7 | % | vendor | no | ev-1190 |
| user session time increase | nearly 20 | seconds | vendor | no | ev-1191 |
What the case does not say
- Baseline before the engagement.
- Measurement method.
- Duration over which the metric held.
- Team size on both sides.
- Cost.
- What was out of scope.
Reading notes for a buyer
This case speaks to the scenario “First outsourced QA team for a mid-size retailer” and feeds dimension D1 (Retail case study evidence) of the methodology. A question to put to QA Madness:
How was the conversion increase measured, from what baseline, and how was it tied to testing rather than other site changes? [ev-1187]
Source and verification
- Source
- QA Strategy for Multiple Web Stores - QA Madness
- Published
- not dated by the vendor
- Verified
- 2026-10-30
- Evidence grade
- secondary
- Corroborating sources
- none found
| Id | Claim | Source | Class | Grade | Published | Verified |
|---|---|---|---|---|---|---|
| ev-1187 | QA Strategy for Multiple Web Stores | QA Strategy for Multiple Web Stores - QA Madness | vendor-case-study | secondary | 2026-10-30 | |
| ev-1188 | First, we compiled testing documentation and checklists with nearly 800 test cases planned for both stores. | QA Strategy for Multiple Web Stores - QA Madness | vendor-case-study | secondary | 2026-10-30 | |
| ev-1189 | Over the 2-year cooperation, QA Madness team detected nearly 1050 bugs within both e-commerce platforms. | QA Strategy for Multiple Web Stores - QA Madness | vendor-case-study | secondary | 2026-10-30 | |
| ev-1190 | Website smooth performance led to a conversion rate increase by 4.7% | QA Strategy for Multiple Web Stores - QA Madness | vendor-case-study | secondary | 2026-10-30 | |
| ev-1191 | Our QA strategy helped to increase user session time in nearly 20 seconds. | QA Strategy for Multiple Web Stores - QA Madness | vendor-case-study | secondary | 2026-10-30 |
Vendor profile: QA Madness. System facet: Ecommerce platform testing vendors.
Positions on this page are computed from the methodology published at /methodology/ and from the evidence records linked on this page. Case study figures are the vendor's or the client's claims as published at the linked source; the "verifiable" column states what this site could check. The site does not recommend vendors, does not accept paid placement and does not publish rates. Last computed: 2026-11-03.
Published by the editor of retailqavendors.com. Last reviewed .