Search and filter the full collection on its own page.
2,439
Abstract
From pre-training on human-written tokens to reinforcement learning on verifiable rewards (RLVR), obtaining cheaply verifiable data is the fuel for current large language models. RLVR often works in mathematics and similar domains because the labels are verifiable by construction and establish a strong self-supervision loop that bootstraps model capabilities. Constructing this loop outside math domains, especially in vision, is an interesting open challenge. We introduce PixelProof, a method that takes human-input prompts and automatically synthesizes novel visual-question images via Python where labels are verifiable by the generation loop itself (without human supervision). That is, PixelProof parameterizes a set of questions by 3 Python programs: (1) a Forward program that translates scene specifications into an expected answer; (2) a Renderer that renders an image; and (3) an Inverse program that progressively transforms the input image using image processing tools to derive its own verification answer independently. Comparing the answers between the Forward (what is designed) vs. Inverse programs (what is observed) self-supervises the generation. Generation can be steered toward (a) a human-defined topic e.g., music; (b) where in the image the answer's evidence lies; and (c) harder questions. As an initial demonstration of PixelProof, five coding agents working across nine visual reasoning directions produced over 2,400 verified questions. Fine-tuning three open-weight models on the newly generated questions improves their performance on held-out questions by an average of +10.7 points, and by +1.9 points on average across 16 external benchmarks. PixelProof shows a promising loop for automatically synthesizing visual questions for evaluating and training vision–language models.
Introduction
Generating visual questions requires checking that their answers can be recovered from the images. The program that draws an image knows the answer from the scene it constructed, even when the rendered image does not contain enough information to answer the question. Occlusion, resolution, and other rendering effects can remove information needed to answer the question.
Using a vision-language model to judge whether a question is answerable makes the check depend on that model’s capabilities. PixelProof instead asks a coding agent to write two programs: one computes the answer from the scene specification, and the other answers from the image alone.
The verifier checks that the two answers agree on 72 scenes. We assess whether the questions are clear to people and useful for training in separate experiments.
Method
PixelProof uses coding agents to generate visual questions and test whether their answers can be recovered from the images. For each question, the agent writes a sampler, a renderer, a forward program that answers from the scene specification, and an inverse program that answers from the rendered image. Before a world is accepted, the verifier runs the final checks, including forward–inverse agreement on 72 scenes.
Question worlds and instances
A question world (world) is a visual question with a
sampler, a renderer, a forward program, and an inverse program. The
sampler draws a scene specification; the renderer turns it into an
image. An instance is one scene specification, its
rendered image, and the world’s question.
Forward and inverse programs
The scene specification describes the objects and
their relationships. The forward program answers from
this specification. The inverse program answers from
the image alone.
Choose a question world
FORWARD AND INVERSE PROGRAMS
How many strings are visible?
The forward program reads the specified string count. The inverse program counts strings in the image.
Forward program
Scene → answer
READ THE SCENE SPECIFICATION
string_count:6
string_span:36
offset_x:−23
…and other scene properties
↓ compute the decision
The count is specified directly
return scene["string_count"]
For this question, the forward computation is a single lookup.
Forward answer6
Rendered image
The renderer draws the scene
String count, spacing, and position become the visible guitar.
scene→renderer→pixels
↳
The inverse program receives only the image. The inverse
does not receive the scene specification.
Inverse program
Image → answer
READ ONLY THE IMAGE PIXELS
1. Locate the bridge and inspect its pixels.
Original pixels · magnified detail
2. Count dark runs. Each dip is a string.
Check 3 neighboring rows6 · 6 · 6
Inverse answer6
=
VERIFIER Compare the answers
Forward 6=Inverse 6The verifier repeats this check on 72 scenes from three fixed seeds.
Brightness traces use the original image pixels. The enlarged regions and rows are chosen for display; the inverse program finds the bridge from the pixels on its own.
Read the original question and full source.
Which panel contains interlocked rings?
Forward program
Scene → answer
READ THE SCENE SPECIFICATION
panels[].linked
Read each panel’s linked flag and select the unique linked pair.
Original forward code
def analytic_gold(scene: dict) -> str:
linked = [i for i, panel in enumerate(scene["panels"]) if panel["linked"]]
if len(linked) != 1:
raise ValueError(f"scene has {len(linked)} interlocked panels, expected exactly 1")
return f"panel-{linked[0] + 1}"
This image shows five framed square panels of identical size standing side by side in one row, numbered 1 to 5 from left to right. Every panel holds exactly two rings of the same thickness, drawn in two different flat colours, and in every panel the two rings cross each other at exactly two places, so ring count, ring size, colour, position and the amount of ink carry no information. The rings are opaque, so at each crossing one ring passes in front and hides a short piece of the other one, leaving a plain break in the ring behind. Two rings are interlocked when neither ring stays in front the whole way round: one ring passes in front at one crossing and behind at the other, so each of the two rings carries exactly one break and each is still a single connected curve. Two rings are merely stacked when the same ring passes in front at both crossings, so that front ring carries no break at all while the ring behind it is broken twice and falls into two separate pieces. Exactly one panel holds a pair of interlocked rings. In each of the other four panels the pair is merely stacked, and the piece of the ring behind that lies between its two breaks is at least 26 pixels long, so both of its breaks are plainly visible. Which ring is drawn in which colour, which ring is in front at a given crossing, and where the pair sits inside its panel are all free to vary. Which numbered panel holds the two interlocked rings, that is the panel in which each of the two rings carries exactly one break? Answer with exactly one of: panel-1, panel-2, panel-3, panel-4, panel-5.
The diagrams apply the inverse program’s thresholds and operations to these images, for illustration.
How many checkpoints are inside the band?
Forward program
Scene → answer
READ THE SCENE SPECIFICATION
trace_valueslower_valuesupper_values
Compare each trace value with the lower and upper band values; count strict inclusions.
Original forward code
def analytic_gold(scene: dict[str, Any]) -> str:
"""Return the visible count, using only the three rendered value arrays."""
flags = [
lower < value < upper
for value, lower, upper in zip(
scene["trace_values"], scene["lower_values"], scene["upper_values"]
)
]
if len(flags) != 6:
raise ValueError("a scene must contain six checkpoints")
return str(sum(flags))
All four dots are candidates. The disconnected dark arcs are pieces of one circle. Which colored dot is at that circle's center? Answer red, blue, green, or orange.
The diagrams apply the inverse program’s thresholds and operations to these images, for illustration.
Reading the diagram
Components of a Question World
The sampler and renderer create an instance. The forward and inverse programs answer the same question from different inputs. The verifier checks their agreement.
01
Sampler
Choose what to draw. The sampler uses a random seed to produce a scene specification: the objects in one instance and their properties.
02
Renderer
Draw the chosen scene. The renderer turns the scene specification into an image. This makes the chosen properties visible as pixels.
03
Forward program
Answer using the scene data. The forward program computes the decision from the scene specification. Here that is a lookup; other questions may require geometry, connectivity, or simulation.
04
Inverse program
Answer using only the image. The inverse program recovers the decision from visible pixels. It can know the world’s visual conventions, but does not receive this instance’s scene specification.
05
Verifier
Check that the two answers agree. After submission, the verifier runs the final checks: both answers must be valid and equal on 72 scenes from three fixed seeds, except scenes excluded by a declared ambiguity rule.
What verification establishes. Passing shows agreement on the tested instances. It does not by itself establish that people will find the question clear or interpret it as intended. See the image-replacement test →
Generation Loop
A controller prepares each attempt and carries results
forward. An episode is one fresh agent session followed
by verification. A campaign is a sequence of episodes
with the same coding-agent configuration and generation instructions.
1
Controller prepares inputs
Instructions, three implementation demonstrations, accepted code, and
the latest rejection report.
2
Agent writes and tests
The agent runs available self-checks, repairs failures, and finalizes
one submission within its time limit.
3
Verifier runs the final checks
The final checks run after submission in a clean, offline container.
They are the same checks the agent could run during its session, but
the agent can no longer change its code.
4
Controller updates the record
A world is kept only if it passes the final checks, the agent finalized its submission, and its question text and images do not exactly copy an earlier world. The next agent receives accepted code and the latest rejection report.
↶ Repeat with a fresh coding-agent session
What does verification require?
Both programs must produce valid, matching answers on each tested scene
that is not excluded. Checks also require reproducible generation and at least two different answers across scenes.
An agent may declare an ambiguity rule: a deterministic
rule that identifies scenes near a decision threshold. The verifier
recomputes it and excludes those scenes from agreement testing. The
current protocol does not cap the excluded fraction. Passing these
checks establishes answer recovery on tested instances; it does not
establish human clarity, novelty, or difficulty.
The three implementation demonstrations cover circle contact, angle
acuteness, and counting with distractors. They are distinct from the
generated examples selected below.
Experiments and Results
We study collection growth, requests about spatial patterns, model performance, model feedback, human review, and fine-tuning. The summaries below follow the paper’s order; full tables and additional analyses are on the results page.
1 · Profile-steered generation
Coding agents repeatedly grow the question collection with PixelProof
A discovery profile is a set of generation instructions that defines a family of questions by the computation they require, with five text-only examples. Five coding-agent configurations ran on nine profiles, for 45 campaigns with six hours of cumulative agent-session time each. Individual sessions were capped at 20 minutes. The coding agents are GPT-5.6 Sol at high and maximum reasoning effort (Sol (high), Sol (max)) and GPT-5.6 Luna, all run through Codex; Claude Opus 5 in Claude Code; and DeepSeek V4 Flash (DS-V4-Flash) in OpenCode.
45 / 45campaigns produced accepted worlds
1,331accepted worlds
1,301worlds passed later 200-scene replay
Throughput was 4.95 accepted worlds per episode-hour. Episode-hours count agent-session time, including time spent waiting for model responses, but not verification. Throughput did not drop over the six hours: in the second half it rose for three configurations, held steady for Sol (high) (7.37 vs. 7.33 per hour), and fell only for Opus 5 (4.30 to 3.83). Of the accepted worlds, 19 failed replay and 11 were not replayed.
The inverse program catches errors that other checks miss
“How many separate raised black keys are visible above the seven white keys?”
Rendered image
1Keep black pixels
2Group connected regions
3Count separate keys
Forward 4≠Inverse 5Self-check fails
Agent repairs the renderer
Repaired image
Forward 4=Inverse 4Checks pass
Given a rendered piano octave, the inverse program processes the image deterministically and arrives at the answer 5. This does not match the forward program’s answer of 4, revealing a bug in the renderer. The mismatch is feedback that helps the agent repair the renderer (right). Without the inverse program, this world would have recorded the wrong answer 4 for an image with five black keys. Steps 1–3 are the inverse program’s own outputs on the faulty image.
Why it matters. For 29% of the worlds, the inverse program failed at least once during generation, and the agent fixed its code before submitting. Nearly half of these fixes changed how the image was drawn or sampled.
In a later 200-scene replay, the forward and inverse programs disagreed on some of the new scenes for 1.6% of previously verified worlds; these worlds are removed before evaluation and training. In a controlled test, the checks detected all 300 image replacements that changed the answer.
Agents can control which parts of an image a question depends on
Each request asks for one of nine spatial patterns for where the answer’s evidence should lie: three scales—focal, regional, distributed—combined with three shapes—compact, pathlike, multipart. The measured pattern comes from masking regions and checking whether the inverse answer changes or the program raises an error.
835 / 874measurable replay-verified worlds matched the request
6.38 vs. 4.26spatial patterns represented in equal-count samples, with versus without steering
In random samples of ten worlds per campaign, steered campaigns covered 6.38 of the nine patterns on average, compared with 4.26 for unsteered campaigns with the same coding agent and profile (35 matched campaign pairs).
Most generated questions are clear and answerable to human reviewers
Two reviewers independently answered and reviewed the same 800 instances across the three generation experiments.
Scroll horizontally to view all columns →
Human-review outcomes
Experiment
Instances
Flagged by neither
Flagged by one
Flagged by both
Profile-steered
400
370
25
5
Spatially-steered
200
183
13
4
Model-feedback-steered
200
180
16
4
Total
800
733
54
13
Neither reviewer flagged 733 instances (91.6%), one flagged 54, and both flagged 13. The two reviewers agreed on whether an instance was valid for 746 of 800 instances (93.25%).
Generated questions challenge open models more than frontier models
We test six vision–language models (VLMs), which answer questions about images; we call them VLM evaluators. Three are frontier models (Sol (high), Opus 5, and Gemini 3.7 Flash) and three are open-weight models (Gemma 4 12B, Gemma 4 31B, and Qwen 3.8 27B).
Each VLM evaluator answers five fixed instances per world. Profile-steered evaluation covers 1,298 worlds; spatially-steered evaluation covers 874. The table reports accuracy over scored responses. Errors, refusals, and unparseable responses are accounted for separately.
Scroll horizontally to view all columns →
Model accuracy (%) over scored responses
Model
Model group
Profile-steered
Spatially-steered
Sol (high)
Frontier
96.1
93.7
Opus 5
Frontier
93.7
86.5
Gemini 3.7 Flash
Frontier
95.9
93.6
Gemma 4 12B
Open-weight
56.9
49.1
Gemma 4 31B
Open-weight
61.0
52.5
Qwen 3.8 27B
Open-weight
55.4
51.1
Declared-transform questions were the hardest profile for each frontier evaluator. Different collection compositions prevent a causal comparison of difficulty.
PixelProof can incorporate model feedback to generate harder questions
Agents receive example worlds and frontier-model errors, then generate new worlds. A world is hard for frontier models when at least two of the three frontier models get two or more of its five instances wrong, with all 15 responses present.
264worlds passed later replay
21 / 252worlds with complete frontier evaluations were hard for frontier models
5worlds challenged all three frontier evaluators
All 21 of these worlds passed replay. Verification remained independent of model predictions. Without a matched no-feedback control, the experiment demonstrates generation under feedback but does not isolate its causal effect.
Fine-tuning VLMs on PixelProof images improves accuracy on held-out benchmarks
Supervised fine-tuning (SFT) trains models on image–question pairs and short answer labels. Training uses 748 profile-steered worlds. Four fine-tuned checkpoints per model are evaluated on 1,260 instances from 252 separate model-feedback-steered worlds. The table reports their mean accuracy and standard deviation.
Scroll horizontally to view all columns →
Transfer to 252 model-feedback-steered worlds · four fine-tuned checkpoints
Model
Base (%)
SFT (%)
Change (pp)
Gemma 4 12B
31.7
34.5±1.5
+2.8
Gemma 4 31B
30.7
37.8±0.5
+7.1
Qwen 3.8 27B
29.8
35.8±0.3
+6.0
Average
30.7
36.0
+5.3
All three models improve on these worlds, by an average of +5.3 percentage points (pp): mean accuracy rises from 30.7% to 36.0%. The training and evaluation sets contain different worlds, but may still contain similar questions.
Transfer to external benchmarks
Averaged over four training splits, three models, and 16 external benchmarks, accuracy rises by +1.9 points. Six benchmarks improve for all three models: VLMsAreBlind, PuzzleVQA, VLMsAreBiased, AlgoPuzzleVQA, PGM, and NaturalBench. The others are mixed, and RAVEN declines slightly for all three, by less than its variation across splits.
Agreement between two generated programs can preserve a shared mistake.
Verification tests selected scenes and does not cap ambiguity-rule
exclusions. Some accepted worlds fail later replay. Human review therefore
remains a separate check of meaning and clarity.
Checking for exact copies does not show that a world is novel. Steering and feedback
experiments support descriptive findings without matched causal controls.
Training gains vary by model and benchmark. The campaigns lasted hours, so
long-term diversity remains untested. Prior exposure to similar questions
in closed models’ training data cannot be ruled out.
Rendered image
1Keep black pixels
2Group connected regions
3Count separate keys
Repaired image
