This page reports the full experimental results, including sample counts, scoring rules, and additional analyses.
Profile-steered generation
Coding agents repeatedly grow the question collection with PixelProof
A discovery profile is a set of generation instructions
that defines a family of questions by the computation they require, with
five text-only examples. The nine profiles
cover tracing, topology, correspondence, search, state tracking,
measurement, prior conflict, global structure, and declared transforms.
Five coding-agent configurations were run on all nine profiles: 45 campaigns, each with six hours of cumulative agent-session time and a 20-minute session cap. The coding agents are GPT-5.6 Sol at high and maximum reasoning effort (Sol (high), Sol (max)) and GPT-5.6 Luna, all run through Codex; Claude Opus 5 in Claude Code; and DeepSeek V4 Flash (DS-V4-Flash) in OpenCode. An
episode-hour counts agent-session time, including time spent waiting for model responses, but not verification. A
200-scene replay is a later agreement check on fresh
sampled scenes.
45 / 45campaigns produced accepted worlds
1,331accepted worlds from 1,426 episodes
1,301worlds passed later replay
Overall throughput was 4.95 accepted worlds per episode-hour. Throughput did not drop over the six hours: in the second half it rose for three configurations, held steady for Sol (high) (7.37 vs. 7.33 per hour), and fell only for Opus 5 (4.30 to 3.83). Of the accepted worlds, 19 failed replay and 11 were not replayed.
Scroll horizontally to view all columns →
Generation outcomes: profile-steered generation
Coding agent
Episodes
Accepted worlds
Passed replay
Accepted / hour
Sol (high)
396
395
392
7.35
Sol (max)
263
255
250
4.75
Luna
230
204
196
3.79
Opus 5
248
219
211
4.06
DS-V4-Flash
289
258
252
4.78
Overall
1,426
1,331
1,301
4.95
No normalized question text was reused across worlds; exact decoded
images were shared by two pairs of worlds. These checks address exact
repetition, not semantic novelty.
The inverse program catches errors that other checks miss
For 29% of the worlds, the inverse program failed at least once during generation and the agent fixed its code. In one case, a renderer bug drew five black keys when the scene called for four; the inverse program counted five.
Agents can control which parts of an image a question depends on
Some questions depend on a small object; others require a long path or
several separated regions. This experiment requests one of nine
size–shape combinations: focal, regional, or distributed extent,
combined with compact, pathlike, or multipart shape.
The measured spatial pattern is given by the regions whose
masking changes the inverse answer or causes an error. It measures the
program’s sensitivity to masking, not human attention. The measured
pattern does not determine acceptance.
835 / 874replay-verified worlds with complete measurements matched the
request
7.09 vs. 5.20spatial patterns represented per campaign, with and without steering
3.35 / houraccepted worlds in steered campaigns
One of 875 replay-verified worlds had an incomplete pattern measurement. In random samples of ten worlds per campaign, steered campaigns covered 6.38 of the nine patterns on average, compared with 4.26 for unsteered campaigns with the same coding agent and profile (35 matched campaign pairs). The two sets of campaigns were run separately, so this comparison is descriptive.
Generation counts and the broader request-matching population
Scroll horizontally to view all columns →
Generation outcomes: spatially-steered generation
Coding agent
Episodes
Accepted worlds
Passed replay
Accepted / hour
Sol (high)
355
348
341
6.47
Sol (max)
254
235
231
4.36
Luna
199
140
129
2.60
Opus 5
203
155
149
2.88
DS-V4-Flash
168
25
25
0.46
Overall
1,179
903
875
3.35
Across all 1,179 episodes, 981 returned a measured pattern and 918
matched the request. This broader population can include rejected
submissions and differs from the replay-verified population above.
Steered campaigns produced 903 accepted worlds; 875 passed replay.
Model evaluation
Generated questions challenge open models more than frontier models
We test six vision–language models (VLMs), which answer questions about images; we call them VLM evaluators. Three are frontier models (Sol (high), Opus 5, and Gemini 3.7 Flash) and three are open-weight models (Gemma 4 12B, Gemma 4 31B, and Qwen 3.8 27B).
Each VLM evaluator answers five fixed instances per world. The evaluation
includes 1,298 profile-steered worlds (6,490 instances) and 874
spatially-steered worlds (4,370 instances). Accuracy in these tables counts scored responses only; missing, refused, and unparseable responses are left out.
Declared-transform questions were the most difficult profile for each
frontier evaluator. The collections differ in composition; the score
differences do not establish that steering causes greater difficulty.
Model-feedback-steered generation
PixelProof can incorporate model feedback to generate harder questions
A world is hard for frontier models if at least two of
the three frontier models get two or more of its five instances wrong,
with all 15 responses present. Of 1,547 worlds with complete frontier
evaluations from the earlier experiments, 70 met this criterion.
Two disjoint sets of ten such worlds supplied example implementations
and evaluator feedback. Each coding-agent configuration ran on both sets
for six hours of cumulative agent-session time per campaign. Later episodes received errors on new
verified worlds. Verification stayed independent of model predictions.
Open-weight models provided no feedback.
266 → 264worlds passed final checks → later replay
21 / 252worlds with complete frontier evaluations were hard for frontier models (8.3%)
5worlds challenged all three frontier evaluators
The other 16 worlds hard for frontier models challenged exactly two
evaluators. All 21 passed replay. This demonstrates generation under
model feedback; without
a matched no-feedback control, it does not isolate the effect of
feedback itself.
Model-feedback-steered generation: accuracy (%) on worlds with complete frontier evaluations
Coding agent
Sol (high)
Opus 5
Gemini 3.7 Flash
Gemma 4 12B
Gemma 4 31B
Qwen 3.8 27B
Sol (high)
96.5
88.4
91.6
28.1
28.1
30.6
Sol (max)
97.4
93.4
97.0
29.2
26.2
23.0
Luna
97.8
93.8
90.7
42.2
40.4
37.3
Opus 5
90.9
87.3
88.6
20.9
25.5
21.4
DS-V4-Flash
100.0
98.3
100.0
51.3
42.6
46.1
Overall
96.3
91.3
93.0
31.7
30.7
29.8
The frontier models supply in-loop feedback. Open-weight scores are
measured after generation on 1,260 instances from the 252 worlds
with complete frontier evaluations. Unparseable open-weight responses count as
incorrect.
Human review
Most generated questions are clear and answerable to human reviewers
Two reviewers independently reviewed the same 800 instances from all three generation experiments. They answered the
questions and recorded concerns about instructions, images, or recorded
answers.
Scroll horizontally to view all columns →
Human-review outcomes
Experiment
Instances
Flagged by neither
Flagged by one
Flagged by both
Profile-steered
400
370
25
5
Spatially-steered
200
183
13
4
Model-feedback-steered
200
180
16
4
Total
800
733
54
13
A reviewer flags an instance when they judge it invalid because of its instructions, image, or recorded answer. Neither reviewer flagged 733 instances (91.6%), one flagged 54, and both flagged 13.
Review agreement
The two reviewers agree on whether an instance is valid for 746 of 800 instances (93.25%). This is agreement on validity, not on the answer.
Training on generated questions
Fine-tuning VLMs on PixelProof images improves accuracy on held-out benchmarks
Supervised fine-tuning (SFT) trains a model on
image–question pairs and their short answer labels.
Low-rank adaptation (LoRA) updates small adapter
matrices while keeping the base model weights fixed.
Training source748 profile-steered worlds
27,762 instances in the pool 555 reserved for validation
→
AdaptationShort answer labels
One epoch · LoRA rank 16 No chain-of-thought targets
→
Separate evaluation252 model-feedback-steered worlds
1,260 instances Base and SFT checkpoints compared
The main training set contains only profile-steered worlds. It does
not include the model-feedback-steered evaluation worlds or spatially-steered
worlds. The training and evaluation sets contain different worlds, but may still contain similar questions.
A percentage point (pp) is the difference between two percentages: 30.7% → 36.0% is +5.3 pp. The table compares each unchanged base checkpoint with the mean and standard deviation
across four independently fine-tuned checkpoints.
+ Increase in accuracy− Decrease in accuracy
Scroll horizontally to view all columns →
Transfer to 252 model-feedback-steered worlds · four fine-tuned checkpoints
Model
Base (%)
SFT (%)
Change (pp)
Gemma 4 12B
31.7
34.5±1.5
+2.8
Gemma 4 31B
30.7
37.8±0.5
+7.1
Qwen 3.8 27B
29.8
35.8±0.3
+6.0
Average
30.7
36.0
+5.3
All three models improve on the model-feedback-steered worlds.
The mean change is +5.3 pp.
Training recipe
The main runs adapt attention and feed-forward projections in the
language model; the vision tower is not directly adapted. LoRA uses rank
16, scaling 32, and dropout 0.05. Training uses one epoch, learning rate
0.0001, cosine decay, 3% warm-up, batch size two, and eight accumulation
steps. The loss covers the answer and turn-end token. Qwen’s thinking
mode is disabled.
Additional training studies
Held-out worlds from profile-steered generation
Scroll horizontally to view all columns →
Profile-steered test sets · four splits, 45 held-out worlds per split
Model
Base (%)
SFT (%)
Change (pp)
Gemma 4 12B
57.8±1.0
69.2±3.7
+11.5
Gemma 4 31B
64.4±1.7
74.6±3.8
+10.2
Qwen 3.8 27B
59.8±2.2
70.2±4.7
+10.4
Average
60.7
71.3
+10.7
Each of four splits holds out 45 worlds (1,669–1,700 instances) from the same generation experiment used for training. The mean improvement is 10.7 percentage points. This is a different test from transfer to model-feedback-steered
worlds. In a separate answer-position check, balancing correct-answer positions changes SFT accuracy from 71.0% to 71.5% for Gemma 4 12B, from 74.4% to 75.7% for Gemma 4 31B, and from 71.8% to 72.5% for Qwen 3.8 27B.
Train directly on model-feedback-steered worlds
Scroll horizontally to view all columns →
Direct training on model-feedback-steered worlds · 50 held-out worlds, 250 instances
Model
Base (%)
SFT (%)
Change (pp)
Gemma 4 12B
28.8
29.6
+0.8
Gemma 4 31B
27.2
46.4
+19.2
Qwen 3.8 27B
25.6
38.4
+12.8
Average
27.2
38.1
+10.9
This separate experiment trains on 214 model-feedback-steered worlds, with
32,560 training instances selected from 63,857 available instances.
Evaluation holds out 50 entire worlds. It does not use the same training
or evaluation population as the main transfer result.
Exclude an entire discovery profile from training
Scroll horizontally to view all columns →
Exclude one profile from training: change in accuracy (pp)
Excluded profile
Gemma 4 12B
Gemma 4 31B
Qwen 3.8 27B
Mean change
Correspondence
+8.7
+13.0
+12.6
+11.4
Measurement
-1.5
+6.0
+13.8
+6.1
Prior conflict
+6.4
+1.7
+7.1
+5.1
Declared transform
+3.1
+2.7
+4.0
+3.3
Search
+8.8
+9.7
+10.5
+9.7
Global structure
+4.1
+1.2
+8.6
+4.6
State tracking
+9.3
+9.1
+14.4
+10.9
Topology
+11.8
+10.6
+14.6
+12.3
Tracing
+10.7
+11.7
+16.7
+13.0
Average
+6.8
+7.3
+11.4
+8.5
Scroll horizontally to view all columns →
Exclude one profile from training · Gemma 4 12B
Excluded profile
Base (%)
SFT (%)
Change (pp)
Correspondence
51.6
60.3
+8.7
Measurement
68.2
66.7
-1.5
Prior conflict
77.9
84.3
+6.4
Declared transform
51.8
54.9
+3.1
Search
60.5
69.3
+8.8
Global structure
64.5
68.6
+4.1
State tracking
36.2
45.5
+9.3
Topology
63.0
74.8
+11.8
Tracing
56.3
67.0
+10.7
Average
58.9
65.7
+6.8
Scroll horizontally to view all columns →
Exclude one profile from training · Gemma 4 31B
Excluded profile
Base (%)
SFT (%)
Change (pp)
Correspondence
54.9
67.9
+13.0
Measurement
74.7
80.7
+6.0
Prior conflict
84.9
86.6
+1.7
Declared transform
58.6
61.3
+2.7
Search
65.1
74.8
+9.7
Global structure
69.7
70.9
+1.2
State tracking
40.7
49.8
+9.1
Topology
66.7
77.3
+10.6
Tracing
62.9
74.6
+11.7
Average
64.2
71.5
+7.3
Scroll horizontally to view all columns →
Exclude one profile from training · Qwen 3.8 27B
Excluded profile
Base (%)
SFT (%)
Change (pp)
Correspondence
54.3
66.9
+12.6
Measurement
65.2
79.0
+13.8
Prior conflict
80.7
87.8
+7.1
Declared transform
51.2
55.2
+4.0
Search
61.6
72.0
+10.5
Global structure
63.6
72.3
+8.6
State tracking
33.5
47.9
+14.4
Topology
60.3
74.9
+14.6
Tracing
53.9
70.5
+16.7
Average
58.3
69.6
+11.4
For each model, nine separate adapters are each trained without one
profile and evaluated on that profile. Accuracy improves in 26 of the 27
model–profile combinations; the exception is Gemma 4 12B on measurement,
which declines by 1.5 points. Averaged over profiles, the gains are
+6.8 points for Gemma 4 12B, +7.3 for Gemma 4 31B, and +11.4 for
Qwen 3.8 27B. Each cell is a single run; Gemma 4 12B uses sixteen
accumulation steps.
External benchmark transfer
Transfer to external benchmarks
Three base models were each fine-tuned on four training splits. The
table reports each model’s mean change and standard deviation across
those four splits. The average across models and benchmarks is +1.9 points.
Positive changeNegative changeColors supplement the numerical values.
Scroll horizontally to view all columns →
External benchmarks: changes across four training splits
Benchmark
Gemma 4 12B
Gemma 4 31B
Qwen 3.8 27B
Mean change
VLMsAreBlind
+30.8 ± 0.7
+8.3 ± 1.2
+2.9 ± 1.1
+14.0
PuzzleVQA
+5.5 ± 2.3
+3.7 ± 2.9
+10.0 ± 2.2
+6.4
VLMsAreBiased
+4.5 ± 1.7
+5.0 ± 3.1
+4.4 ± 2.4
+4.6
AlgoPuzzleVQA
+1.6 ± 0.7
+4.5 ± 1.4
+3.2 ± 1.7
+3.1
PGM
+0.8 ± 1.4
+1.5 ± 0.6
+1.9 ± 1.0
+1.4
V*
-0.6 ± 0.9
-3.1 ± 1.3
+7.8 ± 1.2
+1.4
NaturalBench
+1.1 ± 0.5
+0.6 ± 0.2
+1.8 ± 0.5
+1.2
VSR
-0.2 ± 1.6
+0.6 ± 0.3
+1.7 ± 0.5
+0.7
CV-Bench
-0.2 ± 0.9
+0.7 ± 0.5
+0.6 ± 0.3
+0.4
GQA
+0.1 ± 1.0
-1.2 ± 1.1
+1.5 ± 0.2
+0.1
BLINK
-1.7 ± 1.2
-0.3 ± 1.3
+2.3 ± 1.3
+0.1
A-OKVQA
-2.7 ± 0.9
+0.4 ± 0.3
+1.1 ± 0.1
-0.4
DocVQA
-1.7 ± 0.6
+0.3 ± 0.6
+0.2 ± 0.4
-0.4
CLEVR
-5.3 ± 1.0
+1.0 ± 3.7
+2.5 ± 0.3
-0.6
MathVista
-2.9 ± 2.4
-3.8 ± 6.3
+4.6 ± 1.2
-0.7
RAVEN
-0.8 ± 5.6
-1.8 ± 3.1
-1.2 ± 4.4
-1.2
Average
+1.8
+1.0
+2.8
+1.9
Entries are mean change ± standard deviation in percentage points. BLINK covers IQ_Test; RAVEN covers center_single.
The average is taken over benchmarks, not over pooled examples.
VLMsAreBlind, PuzzleVQA, VLMsAreBiased, AlgoPuzzleVQA, PGM, and
NaturalBench have positive mean changes for all three models. On the other ten, results differ by model; RAVEN declines for all three models, but by less than its standard deviation across splits.
Base and fine-tuned scores across four splits
These are the base scores and four-checkpoint SFT means underlying the changes above. SFT entries show mean ± standard deviation; changes are in percentage points. DocVQA instead uses
Average Normalized Levenshtein Similarity (ANLS), a
text-similarity score, multiplied by 100.
Scroll horizontally to view all columns →
External scores across four training splits · Gemma 4 12B
Benchmark
Base
SFT (mean ± SD)
Change
VLMsAreBlind
46.4
77.3±0.7
+30.8
PuzzleVQA
58.0
63.5±2.3
+5.5
VLMsAreBiased
26.7
31.2±1.7
+4.5
AlgoPuzzleVQA
35.7
37.3±0.7
+1.6
PGM
13.2
14.0±1.4
+0.8
V*
56.2
55.6±0.9
-0.6
NaturalBench
78.6
79.8±0.5
+1.1
VSR
80.0
79.8±1.6
-0.2
CV-Bench
83.2
83.0±0.9
-0.2
GQA
56.3
56.4±1.0
+0.1
BLINK
29.3
27.7±1.2
-1.7
A-OKVQA
84.1
81.4±0.9
-2.7
DocVQA
81.8
80.1±0.6
-1.7
CLEVR
60.0
54.7±1.0
-5.3
MathVista
54.6
51.7±2.4
-2.9
RAVEN
14.8
14.1±5.6
-0.8
Average
53.7
55.5
+1.8
Scroll horizontally to view all columns →
External scores across four training splits · Gemma 4 31B
Benchmark
Base
SFT (mean ± SD)
Change
VLMsAreBlind
74.6
82.9±1.2
+8.3
PuzzleVQA
68.1
71.8±2.9
+3.7
VLMsAreBiased
23.6
28.6±3.1
+5.0
AlgoPuzzleVQA
38.8
43.3±1.4
+4.5
PGM
14.3
15.8±0.6
+1.5
V*
75.1
72.1±1.3
-3.1
NaturalBench
82.9
83.5±0.2
+0.6
VSR
84.0
84.6±0.3
+0.6
CV-Bench
85.6
86.3±0.5
+0.7
GQA
58.0
56.8±1.1
-1.2
BLINK
27.3
27.0±1.3
-0.3
A-OKVQA
89.3
89.8±0.3
+0.4
DocVQA
92.3
92.7±0.6
+0.3
CLEVR
68.4
69.4±3.7
+1.0
MathVista
55.8
52.0±6.3
-3.8
RAVEN
22.6
20.8±3.1
-1.8
Average
60.0
61.1
+1.0
Scroll horizontally to view all columns →
External scores across four training splits · Qwen 3.8 27B
Benchmark
Base
SFT (mean ± SD)
Change
VLMsAreBlind
91.3
94.2±1.1
+2.9
PuzzleVQA
58.6
68.7±2.2
+10.0
VLMsAreBiased
43.1
47.5±2.4
+4.4
AlgoPuzzleVQA
36.9
40.1±1.7
+3.2
PGM
15.8
17.7±1.0
+1.9
V*
81.7
89.6±1.2
+7.8
NaturalBench
81.4
83.1±0.5
+1.8
VSR
85.8
87.5±0.5
+1.7
CV-Bench
87.1
87.7±0.3
+0.6
GQA
62.8
64.3±0.2
+1.5
BLINK
26.0
28.3±1.3
+2.3
A-OKVQA
87.6
88.7±0.1
+1.1
DocVQA
96.0
96.2±0.4
+0.2
CLEVR
95.0
97.5±0.3
+2.5
MathVista
53.3
57.9±1.2
+4.6
RAVEN
48.5
47.3±4.4
-1.2
Average
65.7
68.5
+2.8
BLINK covers the IQ_Test subset and RAVEN covers center_single. Changes are computed from unrounded scores, so displayed values can differ by 0.1 point when subtracted.