design2code-rl: what the grader says about 114 Claude Code attempts, 20 animated and 30 framework trials

Agent: Claude Code (Opus 4.7), one attempt per trial, 10 attempts per task (12 on two tasks). Grader: render-based element matching, see docs/decisions/0001.

1. Headline

Every attempt lands strictly between 0 and 1: the floor is 0.483 and the ceiling 0.869, on a scale where the reference scores 1.0000 and a blank site scores 0.0000. There is room to improve in both directions, which is what an RL environment needs. The 41 infra failures below are sandbox and orchestration errors, not low scores; they carry no reward and are excluded from every statistic here.

Tiers: S, M, L. Agent spend as reported by the Claude Code CLI and recorded per trial by Harbor: $480 over all 155 attempts, $419 over the 114 graded ones ($3.67 per graded trial). Sandbox compute is a rounding error beside that: docs/decisions/0006 measures ~$0.20 per Modal trial against the credit balance, which this report cannot recompute (Modal bills the account, not the trial).

2. Per task

The spread within a task is small (sd 0.02-0.05) next to the spread between tasks (0.53 to 0.83). The agent is consistent; the tasks are what differ. No task is saturated and none is a floor, so every one of them still carries gradient.

3. Tier and archetype

Do the tiers separate? All rewards in this table are grader 1.0 (section 7 measures 1.1 against it). Means are S 0.665, M 0.635, L 0.551: monotone in the intended direction (S > M > L), but the gaps are small next to the within-tier range and the tier ranges overlap heavily. Tier is a page-count label, and page count is not the thing that makes a page hard. The S tier is also flattered by the hand-built atlas-cafe fixture, the easiest task in the set. Without it the means are S 0.614, M 0.635, L 0.551: S and M swap order and sit 0.021 apart, well inside the within-tier sd, while L is clearly lowest. So the ordering that survives is L below the rest; S vs M is noise, and all three sit inside 0.083. The prior-art risk (Design2Code-HARD: frontier models miss 30-40% of blocks on long pages, so the hard tier compresses) shows up here as compression: the L tier is separated, the two smaller tiers are not separated from each other.

Across the 11 tasks, Spearman rho between mean reward and mean reference elements per page is -0.48; between mean reward and page count it is -0.49. Both are moderate and negative, and at n=11 they are not distinguishable from each other: more page and more elements both cost reward, and this sample cannot say which costs more.

4. Backend

Backends should not change the score: the same image, the same verifier, the same Chromium. The gaps above are within the within-task sd, so nothing here suggests a backend effect; the trials differ because the agent's samples differ. Backend choice was made on throughput, not on score (docs/decisions/0006).

5. Where the reward is lost

Position is the whole story. These components are grader 1.0; section 7 shows colour moving 0.899 → 0.914 under the shipped 1.1, and nothing else moving. Size, colour, text and typography all sit high; position averages 0.743. The agent gets the right elements with roughly the right look and puts them in the wrong place, and vertical drift compounds down a long page. That is exactly the failure a screenshot-to-code policy should be trained to fix, and it is why the reward is dense rather than binary.

6. Reference, best and worst

Top 900 px of each index page (rendered at 360 px wide here, 420 px in the file). Left: the reference the agent never sees as HTML. Middle: the best of its attempts. Right: the worst.

atlas-cafe tier S, 5 pages

civic-ember-group-2 tier L, 8 pages

civic-juniper-collective-19 tier S, 5 pages

clinic-corvid-health-37 tier S, 5 pages

clinic-tessellate-supply-46 tier M, 6 pages

conference-anvil-kitchen-42 tier L, 8 pages

docs-wren-collective-33 tier M, 6 pages

fitness-harbor-partners-13 tier L, 8 pages

podcast-foxglove-25 tier S, 5 pages

portfolio-ember-44 tier M, 6 pages

travel-alder-works-35 tier M, 6 pages

7. Why a higher reward is a better replication

The grader is calibrated on a degradation suite: take the reference, break it in one controlled way, and check the reward moves the right way. fixtures/atlas-cafe, 5 pages, from docs/decisions/0001, all 26 rows, grader 1.1. Seventeen assertions in tests/test_validation.py hold this table in place.

Strictly monotone along five axes independently: deleted content, missing pages (1 of 5 costs exactly 20%), tint shift, desaturation, displacement.

Anti-gaming

Grader 1.0 vs 1.1 on the recorded trials

The rewards above were produced by grader 1.0; the repo ships 1.1, which composites every extracted colour over what is behind it and drops what is invisible. Harbor kept no candidate site, so scripts/regrade.py replays each trial's transcript and grades the reconstruction under both graders on one host. Reconstruction fidelity, checked by re-grading with 1.0 against the reward the trial was actually given: 99 exact, 14 within 0.01 (all Modal trials, perceptual term only - this host is arm64), 1 unrecoverable (portfolio-ember-44__tPa3EYh, which wrote 5 of its pages from subagents whose calls are not in the transcript) and excluded below.

Only colour moves: 1.1 reports the background a border-only or gradient box actually paints instead of black, so both sides agree more often. Pages scoring under 0.85 on colour: 34 of 700 under 1.0, 7 under 1.1 - the one figure in section 5 that the grader change moves. Everything else in this report is unaffected.

8. Does the grader agree with people?

The degradation suite says the reward falls when a page is broken in a known way. It does not say the reward orders two independent replications the way a viewer would. That needs a blind pairwise check: show the reference and two candidates with no scores, in randomised order, and ask which is closer. 40 pairs were put to a person: 30 spread across tasks at a reward gap of 0.03 or more, and 10 cut at a gap of 0.10 or more. 35 carry a verdict and 5 were left blank as genuinely uncallable. A Claude judge makes the same forced choice on all 30 of the first set and on 200 more random pairs, which is the only way to get an interval this side of a paid annotation run.

The page-scope confound is real and measurable: in 31 of 200 random pairs the index page alone ranks the two trials the opposite way from the whole-task reward. Scored against the page it was actually shown, the judge does better (112/200, 56%) than against the whole-task reward (97/200, 48%) - some of the disagreement is scope, not grader error. The same caveat applies to every human row in this table, which is why the human rows are labelled as index-page judgements.

Human protocol. 40 pairs, full-page contact sheets at 500 px per column (docs/report/pairs/, docs/report/pairs_cleargap/), no scores shown, A/B order randomised. The labeller is the author of the grader: blind to the scores on these pairs, but not an independent annotator, and there is only one of them, so there is no inter-annotator agreement to report. 35 of 40 rows carry a verdict, which puts the 95% interval on the pooled human row at about +/-13 pp.

Blanks. 5 of the 40 rows were left blank: on those the labeller reported the two replications as good and bad in different ways, with no honest preference to state. Their reward gaps run 0.033 to 0.070, inside the 0.033 to 0.130 range of the rows that were called. Blanks are excluded from every rate above rather than counted as a miss.

Human agreement is not one number, it is a function of how far apart the grader put the two replications. Both label sets pooled, split by that gap:

The first set is dominated by near-ties, so the second set of 10 pairs was cut at the other end of the scale: same task, reward gap 0.10 to 0.13 (docs/report/pairs_cleargap/). That is the claim that matters - that the ordering holds where the difference is large enough to see - and it is the band where the judge agrees most often too. The reading is that the reward is trustworthy for coarse orderings and noisy for fine ones, which is what a dense shaping signal is usually allowed to be.

The same random pairs, split by how far apart the grader put the two replications. If the reward tracks something a viewer can see, agreement should climb with the gap.

Pooling pairs hides the shape of the disagreement, so here is each trial's judge win rate against its grader reward, per task. Read the rho column as a direction, not a measurement: a win rate built from a handful of comparisons is a noisy quantity.

The honest reading. On the widest gaps the two agree (the last bucket above). Below that they are indistinguishable from a coin flip - the pooled 97/200 is not anti-correlation, it is no signal. The mean per-task rho is -0.06, i.e. about zero; individual tasks run from -0.89 to +0.64, but each of those is fitted on a median of 4 comparisons per trial, so the spread is what noise looks like at this sample size and no single task's sign should be believed. Three readings fit the data and it cannot separate them: near-ties dominate the pair set and two replications that close are hard for a viewer to rank either (the human band table above shows the same slope, and every blank row sits in this band); or the judge is ranking the index page while the reward ranks 5-8 pages, which is measured above; or the reward genuinely weights fine layout geometry a viewer does not. What the data does not show is that the reward is arbitrary: the degradation suite, which compares a page against a controlled damage of itself, is monotone on every axis.

Judge: claude-opus-5, 230 forced-choice vision calls (750,642 input / 48,577 output tokens), estimated $4.97. A/B order randomised per call; the judge never sees a score.

9. Site complexity against the Design2Code baseline

Design2Code criticises WebSight for generating pages simpler than real ones. The same criticism applies to us until we measure it. Source complexity of the reference sites against their 484 real scraped pages (Appendix Table 4). Depth is counted inside <body>: add 2 for the html/body wrapper to compare. Rendered elements are the verifier's own reference-element count.

Read honestly, and over the 10 generated sites only - the hand-built fixture is a different animal and is kept out of the mean. On tag count they are 2.7x the real-page baseline (2.2x if every inline SVG tag is thrown away), and ahead on variety (43 unique tags vs 22) - though inline SVG is the only graphics these sites may use, so part of both numbers is drawing, not structure. The gap that remains is depth: 8.2 inside <body>, so about 10 against their 13. That follows from the generator's own header + 4-8 sections + footer constraint and is the one axis on which these pages are simpler than the web. The fixture, at 111 tags and depth 4.4, is the small S-tier page the note in docs/decisions/0005 measured; the generated sites are several times denser. Nothing gates on any of this.

10. Infrastructure failures

41 of 155 trials (26%) died before the verifier ran and wrote no reward and no breakdown (the manifest collector asserts that reward present <=> breakdown present). Per docs/decisions/0006, none is a task, grader or agent defect: 28 are Modal dropping the exec stdio stream or hanging silently 20+ minutes into a run, 4 are our own auto_stop_interval_mins=15 flag stopping a Daytona sandbox mid-agent, and 9 are local Docker jobs killed as duplicates or lost to a bootstrap curl. They are orchestration losses, not zero scores: scoring them as zeros would move the mean by a third and would be measuring Modal, not the agent. The fix is a retry around a trial rather than around a job.

11. Part 2: animated tasks

These tasks carry animated = true and are graded by 1.2 with --animated, which renders both sides at 8 fixed instants (0, 150, 300, 500, 750, 1000, 1500, 2000 ms) with reduced-motion: no-preference and scores the matched elements' delta trajectories. A page with reference motion scores page = 0.7 × static + 0.3 × motion; a page with none scores exactly what 1.1 gave it. A static misplacement is the static term's business and cancels out of the delta comparison, so the two structural halves do not double count; the motion perceptual term compares whole viewport frames, so it does see static misplacement, at 20 percent of the motion term, as 0007 states. See docs/decisions/0007 and docs/design-part2.md. Sections 1 to 10 above are the Part 1 set and exclude these trials entirely, cost and infrastructure failures included, so every Part 1 number there is the number it was before Part 2 existed.

Agent spend on the Part 2 attempts, as recorded per trial by Harbor and counted separately from section 1: $86 over all 22 attempts, $85 over the 20 graded ones ($4.24 per graded trial).

The two terms are worth reading apart. The static half scores 0.661 and the motion half 0.089; mixed at 0.3 on the pages that carry reference motion, that lands the mean reward at 0.489. Motion is the harder of the two here, which is where the headroom is. Section 1's Part 1 mean is 0.622, and the two figures are not comparable: different tasks, grader 1.0 there against 1.2 here, and 20 graded animated trials. Both are printed; no delta is quoted.

The index page at each of the 8 instants, whole viewport scaled to 240x150 per cell: reference row over candidate row. These are the frames the motion term is computed from.

agency-bramble-139-anim tier S, 5 pages

ecommerce-wren-studio-101-anim tier S, 5 pages

Part 2 attempts that were cancelled or failed, so they carry no reward: 2 of 22.

12. Part 3: frameworks

These tasks carry framework in task.toml. The agent is given the same screenshots as the HTML task of the same site and a ready Vite project; the verifier runs npm run build, opens each built page with JavaScript on, freezes the rendered DOM and grades it with the unchanged grader (docs/design-part3.md). A compliance violation returns 0 and keeps the grader's number as graded_reward; both are shown. 1 site: one site cannot separate framework effects from site effects, so every comparison below is within a site.

podcast-foxglove-25

One dot per trial at the reward the grader scored; a hollow dot is a trial the compliance check zeroed, drawn where the grader scored it. A build failure has no dot.

HTML + CSS column: 10 Part 1 trials, rewards from docs/report/regrade-summary.json (grader 1.1, identical to 1.2 on static pages). Framework columns: grader 1.2 on the snapshot. mean graded reward ignores the compliance zero.

Per page, paired against the HTML column

Compliance violations

React + CSS

React + Tailwind

Solid + Tailwind

Generated by scripts/report.py. Pairwise material: docs/report/human_pairs.csv, docs/report/pairs/, docs/report/judge_pairs.csv.