Momentic Mo QA agent just posted the kind of number that makes every QA lead stop scrolling: 269 real bugs found in 60 minutes, at $2.67 per confirmed bug. The catch, stated plainly by Momentic itself, is that Momentic built the benchmark, ran the agents, and graded the results.
That does not make the result useless. It makes it a vendor benchmark to read carefully — and, for freelancers who test other people’s apps, a pricing signal worth understanding.
What QA Bench measures
QA Bench points an agent at a running web application and counts how many real bugs it finds in a fixed time, and how many of its reports are real. The suite covers 1,396 human-labelled bugs across four open-source apps — Gitea, Metabase, Twenty, and Saleor — spanning 20 product areas.
Each configuration gets a fixed time budget, not a fixed token budget. Each run gives the agent one product area in a fresh sandbox and either 30 or 60 minutes. An LLM judge then matches the agent’s bug reports against the approved reference bugs for that area, plus claims reviewers already rejected.
Mo is Momentic’s testing agent, built on three years of test traces. It plans test cases, runs them in parallel across hosted browsers, simulators, or emulators, and a separate agent reproduces functional bugs before they are reported.
The evidence: the 60-minute leaderboard
| Detail | What the source says |
|---|---|
| Mo (Momentic) | 269.3 bugs matched, 19.3% recall, 84.2% precision, $2.67 per bug |
| Codex, GPT-6.1 Sol + Playwright MCP | 86.3 bugs, 6.2% recall, 86.0% precision, $0.80 per bug |
| Claude Code, Opus 5.5 + Playwright MCP | 59.0 bugs, 4.2% recall, about 80% precision, $4.09 per bug |
| Devin | 27 bugs, 1.9% recall, 84% precision, $3.89 per bug |
| Mo at 30 minutes | 176.7 bugs matched, rising 52% to 269.3 at 60 minutes |
| Best recall by app | Saleor at 31%, lowest on Metabase at 9% |
| Apps in the suite | Gitea, Metabase, Twenty, Saleor — together over 190,000 GitHub stars |
| Runs per area | Average of three runs per area, except Devin with a single pass |
| Grading | LLM judge (GPT-6.1 Sol) with a human reviewer; Momentic built, designed, ran, and graded the benchmark |
| Mo in production | Momentic reports 117,010 bugs caught before deploy on its product page |
Recall vs precision. Recall is the share of real bugs found; precision is the share of reports that are real. Mo leads on recall by a wide margin, while Codex leads slightly on precision. For shipping software, a missed bug costs more than a false alarm — but a flood of false alarms costs trust.
Run an AI bug bash before you ship
1. Point it at staging — Never at production data or real users.
2. Give it 60 minutes — Mo’s matches rose 52% from 30 to 60 minutes.
3. Reproduce before reporting — A bug without steps is a rumour.
4. Triage by severity — Fix the checkout bug before the typo.
5. Re-run on the fix — Confirm the bug is gone, not just closed.
How to profit from it
Manual QA is expensive, boring, and easy to skip — which is why clients pay for it after a bad release. An agent bug bash is a sellable service right now:
- Sell a pre-release bug bash. Point Mo, or a coding agent with browser automation, at a client’s staging app before launch and deliver a report with reproduction steps and evidence.
- Price per bug found, with a floor. Mo’s $2.67 per bug is a cost anchor, not your price. Charge for the bash, the triage, and the re-test — the report is the product.
- Use Codex for cheap precision checks. At $0.80 per bug and 86% precision, Codex with Playwright MCP is the budget option when the client wants fewer, cleaner reports.
- Do not quote recall as a guarantee. Mo found 19.3% of the benchmark’s bugs, not 19.3% of any app’s bugs. Saleor recall was 31%; Metabase was 9%. The app matters.
- Disclose the benchmark’s authorship. Tell clients Momentic graded its own test. That honesty is what separates a QA service from a sales pitch.
Sources
Momentic’s official QA Bench report, QA Bench: Evaluating QA Agents on Real Web Applications, and its launch coverage via AI Weekly. All benchmark figures are Momentic’s, from exports taken October 31, 2026, and have not been independently reproduced.
Frequently Asked Questions
What is Momentic Mo?
An autonomous QA agent that plans, explores, and bug-bashes web, iOS, and Android apps from a prompt, reviewing what passed, broke, or was blocked before shipping.
What is QA Bench?
Momentic’s benchmark of 1,396 human-labelled bugs across Gitea, Metabase, Twenty, and Saleor, measuring bugs found in a fixed time and how many reports are real.
How many bugs did Mo find?
269.3 on average in a 60-minute pass, or 19.3% recall, at 84.2% precision and $2.67 per confirmed bug.
How did Codex and Claude Code compare?
Codex with GPT-6.1 Sol found 86.3 bugs at 86% precision and $0.80 per bug; Claude Code with Opus 5.5 found 59.0 at about 80% precision and $4.09 per bug.
Is QA Bench independent?
No. Momentic built, designed, ran, and graded the benchmark, with an LLM judge and a human reviewer. Treat the ranking as a vendor result.
Does more time help Mo more than rivals?
Yes, in this benchmark: Mo’s matches rose 52% from 30 to 60 minutes, vs 16% for Codex.
Can Mo test mobile apps?
Yes. Web sessions use hosted browsers; iOS uses simulators and Android uses emulators, with agents running in parallel up to the plan limit.
Keep Reading
Want more tools that actually pay for themselves? Browse the AI Tools section for the latest launches, pricing breakdowns, and profit playbooks — and check the official source linked above before you buy or deploy anything.