← Project overview

Short questions a frontier model missed

Each question below is one sentence, and each image is one that at least one frontier vision-language model answered wrongly. Try them before opening the answers.

These are development samples generated after submission, chosen because the model missed them; they are not a benchmark and we report no accuracy. Each world passed the same automatic forward–inverse check as the worlds in the paper; none has been through human review.

Question image: Reach a dot through the maze

1 · Tracing

Which colored dot can be reached from the red dot without crossing a wall?

bluegreenorangepurple

Show the answers

Recorded answer: orange

GPT-6.1 Sol (medium reasoning) answered: blue

Watch the programs solve it, step by step
Question image: Trace a route, then turn it

2 · Tracing and rotation

After tracing from green, which letter matches its endpoint turned a quarter-turn clockwise about the cross?

ABCD

Show the answers

Recorded answer: C

GPT-5.6 Sol (high reasoning) answered: A

Opus 5 (high reasoning) answered: D

Question image: Fit a piece into the hole

3 · Shape matching

Which piece fits the hole without being flipped?

ABCD

Show the answers

Recorded answer: A

GPT-6.1 Sol (medium reasoning) answered: D

Watch the programs solve it, step by step