Benchmark · run 2026-11-21
Round 1: Claude Code as a browser agent
We gave Claude Code — running headless, with nothing but a browser — 10 tasks that each have one checkable answer, and ran a hand-written Playwright script on the same tasks as a baseline.
Scope and disclosure. This round tests one agent setup (Claude Code + Playwright MCP) with three Claude models. It is not a cross-vendor comparison, and it does not yet include Browser Use, Stagehand or Skyvern. Claude Code was run on a Claude subscription; the cost figures below are the API-equivalent cost that Claude Code itself reports for each run (
total_cost_usd), not an invoice. BrowseBench has no commercial relationship with Anthropic.Short answer: Haiku 50/50, Sonnet 50/50, Opus 30/30 correct, against 50/50 for the script. On these simple tasks the agent's median time per task was 11.1 s / 12.7 s / 12.9 s versus 1.3 s for the script, and its reported cost per task ranged from $0.004 to $0.104. A perfect score on 10 easy tasks says the setup works; it doesn't say it will on harder sites — that is what the next rounds are for.
Results
| Setup | Correct answers | Median time per task | Median turns | Reported cost per task |
|---|---|---|---|---|
| Scripted Playwright (no model) | 50/50 | 1.3 s | — | $0 |
| Claude Code + Haiku claude-haiku-5-5 | 50/50 | 11.1 s | 6 | $0.004 |
| Claude Code + Sonnet claude-sonnet-5-5 | 50/50 | 12.7 s | 4 | $0.048 |
| Claude Code + Opus claude-opus-5-5 | 30/30 | 12.9 s | 5 | $0.104 |
"Correct" means the final ANSWER: line matched the ground truth. Time is wall-clock per run, including the agent starting its browser; turns are Claude Code's own num_turns. Total reported cost for all agent runs in this round: $5.69.
By task
| Task | Scripted Playwright (no model) | Haiku | Sonnet | Opus |
|---|---|---|---|---|
login-confirmGo to https://quotes.toscrape.com/login and log in (the demo accepts any username and password). After logging in, what is the text of the link shown at the top right of the page? Expected: Logout | 5/5 1.3 s | 5/5 11.6 s | 5/5 11.7 s | 3/3 12.8 s |
js-first-authorOpen https://quotes.toscrape.com/js/ (the quotes are rendered with JavaScript). Who is the author of the first quote on the page? Expected: Albert Einstein | 5/5 0.9 s | 5/5 9.6 s | 5/5 8.7 s | 3/3 9.1 s |
scroll-100th-authorOpen https://quotes.toscrape.com/scroll, which loads more quotes as you scroll. Who is the author of the 100th quote on that page? Expected: George R.R. Martin | 5/5 2.1 s | 5/5 16.8 s | 5/5 19.7 s | 3/3 22.3 s |
distinct-authorsquotes.toscrape.com spreads its quotes over several pages (follow the "Next" links starting at https://quotes.toscrape.com/). How many distinct authors appear across all pages? Answer with a number. Expected: 50 | 5/5 3.3 s | 5/5 10.5 s | 5/5 12.2 s | 3/3 13.4 s |
einstein-countAcross all pages of https://quotes.toscrape.com/ (follow "Next"), how many quotes are by Albert Einstein? Answer with a number. Expected: 10 | 5/5 3.5 s | 5/5 12.1 s | 5/5 12.8 s | 3/3 13.1 s |
travel-cheapestOn https://books.toscrape.com/ open the "Travel" category. What is the price of the cheapest book in that category? Answer with the number only, in pounds. Expected: 23.21 | 5/5 0.9 s | 5/5 10.9 s | 5/5 9.8 s | 3/3 11.0 s |
mystery-countOn https://books.toscrape.com/ open the "Mystery" category (it may have more than one page). How many books are in the Mystery category in total? Answer with a number. Expected: 32 | 5/5 1.3 s | 5/5 12.1 s | 5/5 10.4 s | 3/3 12.6 s |
book-upcOn https://books.toscrape.com/ find the book "A Light in the Attic" and open its product page. What is its UPC? Expected: a897fe39b1053632 | 5/5 1.3 s | 5/5 10.2 s | 5/5 11.3 s | 3/3 12.8 s |
table-max-dueOpen https://the-internet.herokuapp.com/tables. In "Example 1", which person has the largest "Due" amount? Answer with that person's email address. Expected: [email protected] | 5/5 1.0 s | 5/5 8.2 s | 5/5 9.7 s | 3/3 10.2 s |
dynamic-loadingOpen https://the-internet.herokuapp.com/dynamic_loading/2, click "Start" and wait for the result to appear. What text appears? Expected: Hello World! | 5/5 6.4 s | 5/5 15.5 s | 5/5 15.0 s | 3/3 17.4 s |
Setup
- Agent: Claude Code CLI 2.1.322 in headless mode (
claude -p), one fresh session per run,--no-session-persistence, no project settings or instruction files, and only the Playwright MCP tools allowed (no shell, no file access, no web fetch or search). Model chosen with--model haiku|sonnet|opus; the exact model IDs Claude Code reported are listed in the table. - Browser: @playwright/mcp 0.0.86, headless, isolated profile, the same Chromium binary as Round 0.
- Prompt: the task text shown above, plus an instruction to use only the browser and end with
ANSWER: <answer>. Nothing else. - Limits: 60 turns and $2 reported cost per run, 10 minutes wall-clock; up to 3 runs in parallel.
- Baseline: a hand-written Playwright script per task (same Chromium, sequential), to show what the agent costs in time compared with code written once.
- Sites: public practice sites (toscrape.com sandboxes, the-internet.herokuapp.com). Ground truth was computed by script on the run date.
What this does and doesn't tell you
- These are short, well-defined tasks on simple sites. They measure whether the agent can navigate, paginate, wait and read correctly — not how it handles 2FA, CAPTCHAs or messy real-world apps.
- Run counts are small. Treat one failure in five as a signal to look at, not a reliability figure.
- The scripted baseline is fast because a person already did the thinking; the agent's advantage is that nobody had to write the script.
Download and re-run
Tasks, runner, baseline, ground-truth script and every run's raw record are in the public repository github.com/RichardDu1/browsebench-harness (MIT).