a1qa and an anonymised online retailer: Ecommerce platform test automation case analysed
Verdict: what is checkable
The evidence grade (this site's label for how a case is sourced: primary with client or press confirmation, secondary when vendor-hosted only.) is secondary: the only source is the vendor's own page; of 24 metrics, 0 are verifiable yes, 0 partial and 24 no. [ev-1055]
Client profile as published
- Client
A European online commerce platform
(the vendor's description; the vendor does not name the client)- Segment
- Ecommerce and D2C (direct to consumer: a brand that sells its own products online to shoppers without a retailer in between.)
- Region
- Europe
- Channels
- ecommerce
- Size indicators
about 1,100 automated tests
The client description and size indicators are quoted from the source; segment, region and channels are this site's classification. [ev-1055]
Systems in scope
- Ecommerce platform: named in the client description ("A European online commerce platform") and the scope list ("search, cart and checkout, order status and customer notifications"); platform not named
Where the source names each system is quoted or summarised from the case page. [ev-1055]
Problem as stated
The client had about 1,100 automated tests that its engineers had stopped trusting. In the critical subset 18% of tests were flaky, and 24% of reported failures were not reproducible product defects. Reruns and maintenance took about 45 engineering hours a week, and a regression decision took about 2 working days. The client wanted to know which tests to keep, modernize or replace. [ev-1055]
engineers no longer trusted the suite(quoted from the source)
Approach
- Testing types
- automation (main scope: automation)
- Team shape
- 3 automation engineers, 1 senior quality engineering architect, and part-time DevOps support
- Duration
- 12 weeks for assessment and the bounded rescue pilot
- Engagement model
- project
- Tooling named in the source
- Selenium, Java, TestNG, REST Assured, Jenkins, Docker, Allure, Git
Metrics claimed
| Metric | Value | Unit | Claimed by | Verifiable | Evidence |
|---|---|---|---|---|---|
| flaky test rate (page title) | 18% to 4.6% | % | vendor | no | ev-1056 |
| flaky test rate before the engagement | 18 | % | vendor | no | ev-1057 |
| failures that were not product defects before the engagement | 24 | % | vendor | no | ev-1058 |
| weekly maintenance and rerun effort before the engagement | about 45 | engineering hours per week | vendor | no | ev-1059 |
| CI runs within budget without rerun before the engagement | 52 | % | vendor | no | ev-1060 |
| regression decision time before the engagement | about 2 | working days | vendor | no | ev-1061 |
| planned releases per month before the engagement | 2 | releases per month | vendor | no | ev-1061 |
| tests stabilised | 170 | tests | vendor | no | ev-1062 |
| tests replaced | 150 | tests | vendor | no | ev-1063 |
| tests retired | one hundred | tests | vendor | no | ev-1064 |
| weeks for assessment and pilot (stat block) | 12 | weeks | vendor | no | ev-1065 |
| QA experts assigned (stat block) | 5 | QA experts | vendor | no | ev-1066 |
| tests in the inherited suite (stat block) | 1,100 | tests | vendor | no | ev-1067 |
| tests in the audited critical subset (stat block) | 420 | tests | vendor | no | ev-1068 |
| accepted automated tests in the stabilised suite (stat block) | 320 | tests | vendor | no | ev-1069 |
| flaky test rate reduction (stat block) | 13.4 | % | vendor | no | ev-1070 |
| false-failure rate decrease (stat block) | 17 | % | vendor | no | ev-1071 |
| working days saved per new accepted scenario (stat block) | 2.7 | working days | vendor | no | ev-1072 |
| CI runs within budget without rerun increase (stat block) | 39 | % | vendor | no | ev-1073 |
| hours saved per regression decision (stat block) | 17 | hours | vendor | no | ev-1074 |
| additional planned releases per month (stat block) | 1 | releases per month | vendor | no | ev-1075 |
| engineering hours freed per week (stat block) | 29 | engineering hours per week | vendor | no | ev-1076 |
| engineering hours returned to new work per week (stat block) | 29 | engineering hours per week | vendor | no | ev-1077 |
| regression and triage cost (stat block) | 32 | % | vendor | no | ev-1078 |
What the case does not say
- Duration over which the metric held.
- Cost.
- Whether the client is still a client.
- Date of the work.
Reading notes for a buyer
This case speaks to the scenario “Peak-season readiness in 90 days” and feeds dimension D4 (Test automation evidence) of the methodology. A question to put to a1qa:
How long was the agreed validation window behind the flaky-test rate, and has that rate held on the accepted tests since the pilot? [ev-1055]
Source and verification
- Source
- eCommerce · Flaky tests cut from 18% to 4.6%
- Published
- not dated by the vendor
- Verified
- 2026-10-30
- Evidence grade
- secondary
- Corroborating sources
- none found
| Id | Claim | Source | Class | Grade | Published | Verified |
|---|---|---|---|---|---|---|
| ev-1055 | eCommerce · Flaky tests cut from 18% to 4.6% | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1056 | Flaky tests cut from 18% to 4.6% | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1057 | The critical subset produced an 18% flaky test rate. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1058 | 24% of reported failures did not represent a reproducible product defect. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1059 | Maintenance and rerun work consumed about 45 engineering hours each week. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1060 | Only 52% of CI runs completed within the agreed 90-minute budget and without rerun. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1061 | A regression decision for the selected scope took about 2 working days, limiting the product area to 2 planned releases per month. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1062 | After stabilizing 170 tests whose business intent and architecture remained useful, | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1063 | Next, the team replaced 150 tests that could not be repaired economically. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1064 | Ultimately, one hundred obsolete or duplicate tests were retired from the pilot scope. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1065 | 12 weeks for assessment and the bounded rescue pilot | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1066 | 5 QA experts assigned to the project | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1067 | 1,100 tests in the inherited suite | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1068 | 420 tests in the audited critical subset | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1069 | 320 accepted automated tests in the stabilized critical suite | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1070 | 13.4% reduction in flaky test rate across the agreed validation window | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1071 | 17% decrease in false-failure rate for the pilot scope | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1072 | 2.7 working days saved for a new scenario to become an accepted test | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1073 | 39% increase in CI runs completed within the 90-minute budget and without rerun | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1074 | 17 hours saved per critical scope regression decision | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1075 | 1 additional planned release per month after the pilot scope became reliable | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1076 | 29 engineering hours freed per week for suite maintenance and reruns | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1077 | 29 engineering hours per week returned to new automated scenarios and exploratory work | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1078 | 32% lower regression and triage cost after ongoing maintenance was included | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 |
| Id | Claim | Source | Class | Grade | Published | Verified |
|---|---|---|---|---|---|---|
| ev-1055 | eCommerce · Flaky tests cut from 18% to 4.6% | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1056 | Flaky tests cut from 18% to 4.6% | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1057 | The critical subset produced an 18% flaky test rate. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1058 | 24% of reported failures did not represent a reproducible product defect. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1059 | Maintenance and rerun work consumed about 45 engineering hours each week. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1060 | Only 52% of CI runs completed within the agreed 90-minute budget and without rerun. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1061 | A regression decision for the selected scope took about 2 working days, limiting the product area to 2 planned releases per month. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1062 | After stabilizing 170 tests whose business intent and architecture remained useful, | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1063 | Next, the team replaced 150 tests that could not be repaired economically. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1064 | Ultimately, one hundred obsolete or duplicate tests were retired from the pilot scope. | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1065 | 12 weeks for assessment and the bounded rescue pilot | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1066 | 5 QA experts assigned to the project | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 |
Show 12 more evidence records
| Id | Claim | Source | Class | Grade | Published | Verified |
|---|---|---|---|---|---|---|
| ev-1067 | 1,100 tests in the inherited suite | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1068 | 420 tests in the audited critical subset | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1069 | 320 accepted automated tests in the stabilized critical suite | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1070 | 13.4% reduction in flaky test rate across the agreed validation window | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1071 | 17% decrease in false-failure rate for the pilot scope | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1072 | 2.7 working days saved for a new scenario to become an accepted test | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1073 | 39% increase in CI runs completed within the 90-minute budget and without rerun | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1074 | 17 hours saved per critical scope regression decision | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1075 | 1 additional planned release per month after the pilot scope became reliable | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1076 | 29 engineering hours freed per week for suite maintenance and reruns | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1077 | 29 engineering hours per week returned to new automated scenarios and exploratory work | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 | |
| ev-1078 | 32% lower regression and triage cost after ongoing maintenance was included | eCommerce · Flaky tests cut from 18% to 4.6% | vendor-case-study | secondary | 2026-10-30 |
Vendor profile: a1qa. System facet: Ecommerce platform testing vendors.
Positions on this page are computed from the methodology published at /methodology/ and from the evidence records linked on this page. Case study figures are the vendor's or the client's claims as published at the linked source; the "verifiable" column states what this site could check. The site does not recommend vendors, does not accept paid placement and does not publish rates. Last computed: 2026-11-03.
Published by the editor of retailqavendors.com. Last reviewed .