← Project overview

3D scenes and photographs

The worlds in the paper are 2D images drawn with Python code. After submission we kept the forward–inverse check unchanged and gave the coding agent two other ways to make an image: Blender for 3D scenes, and an image-generation model that turns a program rendering into a photograph. In both, one program computes the answer and a second program must recover it from the pixels alone.

Both are first runs. The samples below are exploratory, none has been through formal human review, and we report no model accuracy on them.

3D scenes rendered in Blender

The same forward–inverse check, with a 3D renderer in place of 2D drawing code. One program builds a scene, renders it in Blender and computes the answer; a second program receives only the rendered image and the question, and must reach the same answer from the pixels.

Exploratory samples made after submission; they are not part of the paper's dataset and we report no model accuracy on them. One coding agent ran for two hours on each of three themes and attempted 23 worlds: 22 passed the check during development, and 20 passed it again on six new scenes each (120 of 120 answers agree). This page shows 9 of those 20, chosen by us, with three scenes each. No world has been through human review, and one agent wrote both programs of each world.

1 · 3D spatial relationships

Are the two closed metal rings linked together?

yesno

Rendered scene: Linked metal rings

Answer from both programs: yes

  • Crossing 1: the brass ring is on top.
  • Crossing 2: the steel ring is on top.
  • Different rings on top at the two crossings: linked.
Rendered scene: Linked metal rings

Answer from both programs: no

  • Crossing 1: the brass ring is on top.
  • Crossing 2: the brass ring is on top.
  • The same ring on top at both crossings: not linked.
Rendered scene: Linked metal rings

Answer from both programs: yes

  • Crossing 1: the brass ring is on top.
  • Crossing 2: the steel ring is on top.
  • Different rings on top at the two crossings: linked.
Watch the inverse program work, step by step
Inverse program as generated (81 lines)
import math
import numpy as np
import cv2


def _ellipse(mask):
    n, labels, stats, _ = cv2.connectedComponentsWithStats(mask.astype(np.uint8),8)
    if n < 2:
        raise ValueError('No ring material recovered')
    keep=[max(range(1,n),key=lambda i: stats[i,cv2.CC_STAT_AREA])]
    clean=np.isin(labels,keep)
    y,x=np.where(clean)
    if len(x)<500:
        raise ValueError('Insufficient exposed ring surface')
    pts=np.column_stack([x,y]).astype(np.float32)
    fit=cv2.fitEllipse(pts.reshape(-1,1,2))
    # Robustly trim feet and specular stray pixels using radial residual.
    for _ in range(3):
        c,axes,ang=fit
        u=pts-np.array(c)
        t=math.radians(ang); co=math.cos(t); si=math.sin(t)
        xx=u[:,0]*co+u[:,1]*si
        yy=-u[:,0]*si+u[:,1]*co
        rr=np.sqrt((xx/(axes[0]/2))**2+(yy/(axes[1]/2))**2)
        trimmed=pts[np.abs(rr-1)<0.15]
        if len(trimmed)<500: raise ValueError('Incomplete ellipse')
        fit=cv2.fitEllipse(trimmed.reshape(-1,1,2))
    return fit


def _point(e,t):
    c,a,ang=e
    co=math.cos(math.radians(ang)); si=math.sin(math.radians(ang))
    x=a[0]/2*math.cos(t); y=a[1]/2*math.sin(t)
    return np.array([c[0]+co*x-si*y,c[1]+si*x+co*y])


def _implicit(e,p):
    c,a,ang=e
    u=p-np.array(c); co=math.cos(math.radians(ang)); si=math.sin(math.radians(ang))
    return ((u[0]*co+u[1]*si)/(a[0]/2))**2+((-u[0]*si+u[1]*co)/(a[1]/2))**2-1


def measure(image):
    rgb=np.asarray(image.convert('RGB')).astype(np.float32)
    r,g,b=rgb[:,:,0],rgb[:,:,1],rgb[:,:,2]
    steel=(b>r*1.075)&(b>g*1.015)&(r<180)&(b<220)
    brass=(r>g*1.14)&(g>b*1.20)&(r>55)
    e1=_ellipse(steel); e2=_ellipse(brass)
    crossings=[]
    for i in range(720):
        lo=2*math.pi*i/720; hi=2*math.pi*(i+1)/720
        fl=_implicit(e2,_point(e1,lo)); fh=_implicit(e2,_point(e1,hi))
        if fl*fh>=0: continue
        for _ in range(20):
            mid=(lo+hi)/2; fm=_implicit(e2,_point(e1,mid))
            if fl*fm<=0: hi=mid
            else: lo=mid; fl=fm
        p=_point(e1,(lo+hi)/2)
        x,y=[int(round(v)) for v in p]
        st=int(steel[y-4:y+5,x-4:x+5].sum())
        br=int(brass[y-4:y+5,x-4:x+5].sum())
        if max(st,br)<30 or abs(st-br)<25:
            raise ValueError('Crossing order not clearly observable')
        crossings.append({'xy':p.tolist(),'steel_pixels':st,'brass_pixels':br,'over':'steel' if st>br else 'brass'})
    if len(crossings)!=2:
        raise ValueError('Supported scope requires two transverse projected crossings')
    linked=crossings[0]['over']!=crossings[1]['over']
    return {'ellipses':[{'center':list(e[0]),'axes':list(e[1]),'angle':e[2]} for e in (e1,e2)],
            'crossings':crossings,'trace':'Opposite crossing orders imply interlock of two simple closed rings; equal orders imply separation.',
            'decision':'linked' if linked else 'separate','answer':'yes' if linked else 'no'}


def answer_from_image(image,question,choices):
    result=measure(image)['answer']
    if result not in choices: raise ValueError('Missing public answer choice')
    return result


def explain_from_image(image,question,choices):
    return measure(image)

2 · Mechanisms

With ideal pin joints and the two lower pivots fixed, which four-sided frame can change shape without bending its bars?

leftright

Rendered scene: Pinned-frame mobility

Answer from both programs: right

  • Left frame: 5 bars between 4 pins; the bars lock its shape.
  • Right frame: 4 bars between 4 pins; it can still change shape.
  • Blue pins are the two fixed lower pivots.
Rendered scene: Pinned-frame mobility

Answer from both programs: left

  • Left frame: 4 bars between 4 pins; it can still change shape.
  • Right frame: 5 bars between 4 pins; the bars lock its shape.
  • Blue pins are the two fixed lower pivots.
Rendered scene: Pinned-frame mobility

Answer from both programs: right

  • Left frame: 5 bars between 4 pins; the bars lock its shape.
  • Right frame: 4 bars between 4 pins; it can still change shape.
  • Blue pins are the two fixed lower pivots.
Watch the inverse program work, step by step
Inverse program as generated (69 lines)
"""Pixel-only pin detection, bar recovery, and pinned rigidity computation."""
import itertools
import cv2
import numpy as np


def _measure(image):
    rgb=np.asarray(image.convert('RGB'))
    hsv=cv2.cvtColor(rgb,cv2.COLOR_RGB2HSV)
    # Brass hue is a material cue. It does not encode linkage state.
    mask=((hsv[:,:,0]>8)&(hsv[:,:,0]<39)&(hsv[:,:,1]>75)&(hsv[:,:,2]>65)).astype(np.uint8)*255
    mask=cv2.morphologyEx(mask,cv2.MORPH_CLOSE,np.ones((3,3),np.uint8))
    n,labels,stats,centers=cv2.connectedComponentsWithStats(mask)
    pins=[]
    for k in range(1,n):
        x,y,w,h,area=stats[k]
        if 80<area<2200 and .55<w/max(h,1)<1.8 and 10<w<55 and 10<h<55:
            pins.append([float(centers[k][0]),float(centers[k][1])])
    if len(pins)!=8: raise ValueError('ambiguous pivot segmentation: '+str(len(pins)))
    pins.sort(key=lambda p:p[0])
    groups=[pins[:4],pins[4:]]
    # Neutral dark steel bars contrast with the pale mounting plates.
    gray=cv2.cvtColor(rgb,cv2.COLOR_RGB2GRAY)
    dark=gray<175
    records=[]
    for ps in groups:
        p=np.array(ps,float)
        fixed=list(np.argsort(p[:,1])[-2:])
        free=[i for i in range(4) if i not in fixed]
        edges=[]; scores=[]
        for i,j in itertools.combinations(range(4),2):
            d=p[j]-p[i]; length=np.linalg.norm(d); normal=np.array([-d[1],d[0]])/length
            # End exclusion removes heads; a small transverse band tolerates
            # the visible height offset between the pin cap and the steel bar.
            ts=np.linspace(.19,.81,90)
            profiles=[]
            for offset in range(-10,11):
                q=p[i][None,:]+ts[:,None]*d+normal[None,:]*offset
                xy=np.rint(q).astype(int)
                profiles.append(dark[xy[:,1],xy[:,0]])
            support=np.max(np.array(profiles),axis=0)
            score=float(support.mean())
            if .40<score<.80: raise ValueError('ambiguous bar support')
            if score>=.80: edges.append([i,j])
            scores.append({'pair':[i,j],'support':score})
        rows=[]
        for i,j in edges:
            d=p[i]-p[j]; d=d/np.linalg.norm(d)
            row=np.zeros(4)
            if i in free: row[2*free.index(i):2*free.index(i)+2]=d
            if j in free: row[2*free.index(j):2*free.index(j)+2]=-d
            rows.append(row)
        if len(edges) not in (4,5): raise ValueError('unsupported recovered linkage')
        singular=np.linalg.svd(np.array(rows),compute_uv=False)
        rank=int(np.sum(singular>.015))
        records.append({'pivots':ps,'fixed':list(map(int,fixed)),'bars':edges,'bar_scores':scores,'singular_values':singular.tolist(),'constraint_rank':rank,'internal_dof':4-rank})
    mobile=[i for i,r in enumerate(records) if r['internal_dof']>0]
    if len(mobile)!=1: raise ValueError('ambiguous mobility comparison')
    return {'objects':records,'decision':'left' if mobile[0]==0 else 'right','trace':'Recover brass pivot centers; measure continuous steel bars; fix lower two centers; rank bar-length constraints on upper pivots.'}


def explain_from_image(image,question,choices):
    result=_measure(image)
    if result['decision'] not in choices: raise ValueError('unsupported choices')
    return result


def answer_from_image(image,question,choices):
    return explain_from_image(image,question,choices)['decision']

3 · Mechanisms

Which linkage lets its left grounded crank complete a full turn? Assume ideal pin joints and ignore collisions between bars in separate layers.

Left linkageRight linkage

Rendered scene: Full crank rotation

Answer from both programs: Right linkage

  • Left linkage: crank 124, coupler 160, output 208, ground 292 px; the crank cannot turn fully.
  • Right linkage: crank 95, coupler 246, output 165, ground 258 px; the crank can turn fully.
  • Pink is the left grounded crank.
Rendered scene: Full crank rotation

Answer from both programs: Left linkage

  • Left linkage: crank 129, coupler 243, output 209, ground 263 px; the crank can turn fully.
  • Right linkage: crank 122, coupler 158, output 176, ground 277 px; the crank cannot turn fully.
  • Pink is the left grounded crank.
Rendered scene: Full crank rotation

Answer from both programs: Right linkage

  • Left linkage: crank 129, coupler 169, output 187, ground 276 px; the crank cannot turn fully.
  • Right linkage: crank 111, coupler 241, output 205, ground 292 px; the crank can turn fully.
  • Pink is the left grounded crank.
Inverse program as generated (61 lines)
"""Pixel-only pivot recovery followed by full-cycle four-bar closure."""
import math
import numpy as np
import cv2


def measure(image):
    rgb=np.asarray(image.convert('RGB'))
    # Dark circular pin heads are ordinary visible mechanical fasteners.
    gray=cv2.cvtColor(rgb,cv2.COLOR_RGB2GRAY)
    mask=(gray<90).astype(np.uint8)*255
    # Remove narrow cast-shadow bridges/tails while retaining round pin cores.
    mask=cv2.morphologyEx(mask,cv2.MORPH_OPEN,
                          cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(7,7)))
    count,labels,stats,centers=cv2.connectedComponentsWithStats(mask)
    pins=[]
    for k in range(1,count):
        x,y,w,h,area=map(int,stats[k])
        if not (350<area<1100 and 18<w<36 and 18<h<36): continue
        if max(w,h)/min(w,h)>1.45: continue
        if area/(w*h)<.60: continue
        pins.append([float(centers[k][0]),float(centers[k][1])])
    if len(pins)!=8:
        raise ValueError('Expected eight unoccluded round pin heads; recovered '+str(len(pins)))
    pins.sort(key=lambda p:p[0])
    groups=[pins[:4],pins[4:]]
    measurements=[]
    for group in groups:
        # In this supported upright configuration both fixed pivots lie below
        # both moving pivots. Infer the grounded pair from visible image order.
        by_y=sorted(group,key=lambda p:p[1],reverse=True)
        ground=sorted(by_y[:2],key=lambda p:p[0])
        moving=sorted(by_y[2:],key=lambda p:p[0])
        A,D=ground; B,C=moving
        a=math.dist(A,B); b=math.dist(B,C); c=math.dist(C,D); g=math.dist(D,A)
        lower=abs(g-a)-abs(b-c)
        upper=b+c-g-a
        margin=min(lower,upper)
        if abs(margin)<12: raise ValueError('Full-cycle closure too close to boundary')
        measurements.append({'pivots_px':[A,B,C,D], 'lengths_px':{'input':a,'coupler':b,'output':c,'ground':g},
                             'crank_distance_range_px':[abs(g-a),g+a],
                             'closure_annulus_px':[abs(b-c),b+c],
                             'endpoint_clearances_px':[lower,upper],
                             'margin_px':margin,'full_turn':margin>0})
    valid=[m['full_turn'] for m in measurements]
    if sum(valid)!=1: raise ValueError('Expected one uniquely feasible full-turn linkage')
    return {'objects':measurements,'decision':'left' if valid[0] else 'right',
            'trace':'The complete crank distance interval must lie strictly inside the two-link closure annulus.'}


def answer_from_image(image,question,choices):
    m=measure(image)
    wanted='Left linkage' if m['decision']=='left' else 'Right linkage'
    if wanted not in choices: raise ValueError('Unsupported choices')
    return wanted


def explain_from_image(image,question,choices):
    result=measure(image)
    result['answer']='Left linkage' if result['decision']=='left' else 'Right linkage'
    return result

4 · Mechanisms

For equal added loads, which assembly extends farther? Assume identical linear springs and rigid crossbars.

Left assemblyRight assembly

Rendered scene: Spring extension

Answer from both programs: Right assembly

  • Left assembly: 2 then 3 springs side by side; stretch is proportional to 1/2 + 1/3 = 0.83.
  • Right assembly: 1 then 4 springs side by side; stretch is proportional to 1/1 + 1/4 = 1.25.
  • More springs side by side are stiffer; stages in a chain add their stretch.
Rendered scene: Spring extension

Answer from both programs: Left assembly

  • Left assembly: 1 then 4 springs side by side; stretch is proportional to 1/1 + 1/4 = 1.25.
  • Right assembly: 3 then 2 springs side by side; stretch is proportional to 1/3 + 1/2 = 0.83.
  • More springs side by side are stiffer; stages in a chain add their stretch.
Rendered scene: Spring extension

Answer from both programs: Right assembly

  • Left assembly: 2 then 3 then 3 springs side by side; stretch is proportional to 1/2 + 1/3 + 1/3 = 1.17.
  • Right assembly: 1 then 3 then 4 springs side by side; stretch is proportional to 1/1 + 1/3 + 1/4 = 1.58.
  • More springs side by side are stiffer; stages in a chain add their stretch.
Watch the inverse program work, step by step
Inverse program as generated (73 lines)
"""Pixel-only crossbar localization, spring grouping, and compliance calculation."""
import numpy as np
import cv2


def measure(image):
    rgb = np.asarray(image.convert('RGB'))
    hsv = cv2.cvtColor(rgb, cv2.COLOR_RGB2HSV)
    # Natural brass crossbars are chromatic; steel coils and frames are neutral.
    mask = ((hsv[:,:,0] >= 8) & (hsv[:,:,0] <= 38) &
            (hsv[:,:,1] > 65) & (hsv[:,:,2] > 65)).astype(np.uint8)
    mask = cv2.morphologyEx(mask,cv2.MORPH_CLOSE,np.ones((3,5),np.uint8))
    mask = cv2.morphologyEx(mask,cv2.MORPH_OPEN,np.ones((1,55),np.uint8))
    num, labels, stats, centroids = cv2.connectedComponentsWithStats(mask)
    bars = []
    for x,y,w,h,area in stats[1:]:
        if w > 150 and 6 < h < 60 and w/h > 5 and area > 1200:
            bars.append([int(x),int(y),int(w),int(h)])
    if len(bars) not in [6,8]:
        raise ValueError('Cannot recover six or eight distinct brass crossbars: '+str(bars))
    centers = sorted([b[0]+b[2]/2 for b in bars])
    split = (centers[len(centers)//2-1]+centers[len(centers)//2])/2
    groups = [[b for b in bars if b[0]+b[2]/2 < split],
              [b for b in bars if b[0]+b[2]/2 > split]]
    gray = cv2.cvtColor(rgb,cv2.COLOR_RGB2GRAY)
    result = []
    for group in groups:
        group.sort(key=lambda b:b[1])
        stages = []
        for upper,lower in zip(group,group[1:]):
            # Exclude bar faces and eyelets; each coil crosses this central band.
            y1 = upper[1]+upper[3]
            y2 = lower[1]
            ya = int(y1+0.26*(y2-y1))
            yb = int(y1+0.74*(y2-y1))
            xa = int(max(upper[0],lower[0])+7)
            xb = int(min(upper[0]+upper[2],lower[0]+lower[2])-7)
            crop = gray[ya:yb,xa:xb]
            threshold = min(112.0, float(np.median(crop)) * 0.70)
            dark = (crop < threshold).astype(np.uint8)
            # A coil is a persistent vertical band of dark wire. Integrating
            # across its turns bridges horizontal holes without joining coils.
            profile = dark.mean(axis=0)
            active = (profile > 0.13).astype(np.uint8)
            active = cv2.morphologyEx(active.reshape(1,-1), cv2.MORPH_CLOSE,
                                     np.ones((1,17),np.uint8)).reshape(-1)
            transitions = np.diff(np.r_[0,active,0].astype(int))
            starts = np.where(transitions == 1)[0]
            ends = np.where(transitions == -1)[0]
            spans = [[int(a+xa),int(b+xa)] for a,b in zip(starts,ends) if 10 <= b-a <= 48]
            if not 1 <= len(spans) <= 4:
                raise ValueError('Ambiguous spring count: '+str(spans))
            stages.append({'band':[xa,ya,xb,yb], 'spring_spans':spans, 'parallel_count':len(spans)})
        result.append({'crossbars':group,'stages':stages,
                       'compliance':float(sum(1/s['parallel_count'] for s in stages))})
    a,b = [r['compliance'] for r in result]
    separation = abs(a-b)/max(a,b)
    if separation < 0.12:
        raise ValueError('Indistinguishable assembly compliances')
    return {'assemblies':result,'relative_separation':separation,
            'decision':'Left assembly' if a>b else 'Right assembly',
            'trace':'parallel stage stiffness = n*k; series compliance = sum(1/n)/k'}


def explain_from_image(image,question,choices):
    result = measure(image)
    if result['decision'] not in choices:
        raise ValueError('Unsupported choices')
    return result


def answer_from_image(image,question,choices):
    return explain_from_image(image,question,choices)['decision']

5 · Functional inspection

Does the strap pass alternately over and under successive bars?

yesno

Rendered scene: Strap-weave alternation

Answer from both programs: no

  • 5 bars found. Strap at each bar, in order: over, under, under, under, over.
  • Two neighbouring bars have the same state, so it does not alternate.
Rendered scene: Strap-weave alternation

Answer from both programs: yes

  • 7 bars found. Strap at each bar, in order: under, over, under, over, under, over, under.
  • It alternates all the way.
Rendered scene: Strap-weave alternation

Answer from both programs: no

  • 6 bars found. Strap at each bar, in order: under, over, under, under, under, over.
  • Two neighbouring bars have the same state, so it does not alternate.
Inverse program as generated (65 lines)
"""Pixel-only occlusion measurement followed by sequence inspection."""
import numpy as np
import cv2


def measure(image):
    rgb=np.asarray(image.convert('RGB')).astype(np.float32)
    R,G,B=rgb[:,:,0],rgb[:,:,1],rgb[:,:,2]
    blue=(B>R*1.30)&(B>G*1.12)&(B-R>24)&(B>55)
    wood=(R>G*1.12)&(G>B*1.16)&(R-B>25)
    yy,xx=np.nonzero(blue)
    if len(xx)<1500: raise ValueError('Insufficient visible strap')
    pts=np.column_stack([xx,yy]).astype(float)
    origin=np.mean(pts,axis=0)
    cov=np.cov(pts.T); val,vec=np.linalg.eigh(cov); axis=vec[:,-1]
    if axis[1]<0:axis=-axis
    normal=np.array([axis[1],-axis[0]])
    H,W=blue.shape
    gy,gx=np.indices((H,W))
    px=(gx-origin[0])*axis[0]+(gy-origin[1])*axis[1]
    qx=(gx-origin[0])*normal[0]+(gy-origin[1])*normal[1]
    # Width is estimated from the strap pixels, independent of bar count.
    qblue=(pts-origin)@normal
    width=float(np.percentile(qblue,97)-np.percentile(qblue,3))
    if width<20:raise ValueError('Strap too narrow')
    side=(np.abs(qx)>width*.85)&(np.abs(qx)<width*1.5)
    bins=np.floor(px+H).astype(np.int32)
    sw=np.bincount(bins[side&wood],minlength=3*H)
    st=np.bincount(bins[side],minlength=3*H)
    profile=sw/np.maximum(st,1)
    active=(profile>.58).astype(np.uint8)
    # Separate rung bands, rejecting isolated pixels and end-clamp supports.
    runs=[];beg=None
    for i,v in enumerate(active):
        if v and beg is None:beg=i
        if beg is not None and (not v or i==len(active)-1):
            stop=i
            if stop-beg>=8:runs.append((beg-H,stop-H))
            beg=None
    # Frame rails lie outside the side strips; actual rungs span both sides.
    states=[]; measured=[]
    for lo,hi in runs:
        mid=(lo+hi)/2
        region=(np.abs(px-mid)<max(2,(hi-lo)*.20))&(np.abs(qx)<width*.27)
        score=float(np.mean(blue[region])) if np.any(region) else .5
        if .15<score<.85:raise ValueError('Ambiguous crossing')
        state='over' if score>=.85 else 'under'
        states.append(state)
        measured.append(dict(position=float(mid),band_width=float(hi-lo),blue_fraction=score,state=state))
    if not 5<=len(states)<=7:raise ValueError('Expected five to seven exposed bars; found '+str(len(states)))
    alternating=all(a!=b for a,b in zip(states,states[1:]))
    return dict(strap_width_px=width,axis=axis.tolist(),crossings=measured,
                states=states,violations=[i for i in range(len(states)-1) if states[i]==states[i+1]],
                decision='yes' if alternating else 'no')


def answer_from_image(image,question,choices):
    d=measure(image)['decision']
    for choice in choices:
        if choice.lower()==d:return choice
    raise ValueError('Unsupported choices')


def explain_from_image(image,question,choices):
    return measure(image)

6 · Functional inspection

In this overhead view, is the lamp shade's center inside the triangle formed by the centers of the three rubber feet?

yesno

Rendered scene: Lamp support projection

Answer from both programs: no

  • Triangle through the three feet (blue); shade centre (pink).
  • Signed distance from the centre to each side: +182, +446, -114 px.
  • One is negative: the centre is outside.
Rendered scene: Lamp support projection

Answer from both programs: yes

  • Triangle through the three feet (blue); shade centre (pink).
  • Signed distance from the centre to each side: +275, +73, +169 px.
  • All positive: the centre is inside.
Rendered scene: Lamp support projection

Answer from both programs: no

  • Triangle through the three feet (blue); shade centre (pink).
  • Signed distance from the centre to each side: +343, +212, -62 px.
  • One is negative: the centre is outside.
Inverse program as generated (64 lines)
"""Pixels only: locate rubber pads and enamel shade, then test support triangle."""
import math
import cv2
import numpy as np


def _components(mask):
    n,l,stats,centroids=cv2.connectedComponentsWithStats(mask.astype(np.uint8),8)
    return [(stats[i],centroids[i],l==i) for i in range(1,n)]


def measure(image):
    a=np.asarray(image.convert('RGB'))
    hsv=cv2.cvtColor(a,cv2.COLOR_RGB2HSV)
    # Yellow enamel is an ordinary material, common to both decisions.
    shade=((hsv[:,:,0]>10)&(hsv[:,:,0]<42)&(hsv[:,:,1]>105)&(hsv[:,:,2]>65))
    shade=cv2.morphologyEx(shade.astype(np.uint8),cv2.MORPH_CLOSE,np.ones((7,7),np.uint8))
    shapes=[v for v in _components(shade) if v[0][4]>1500]
    if len(shapes)!=1: raise ValueError('ambiguous shade segmentation')
    st,center,mask=shapes[0]
    contours,_=cv2.findContours(mask.astype(np.uint8),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)
    c=max(contours,key=cv2.contourArea)
    (sx,sy),sr=cv2.minEnclosingCircle(c)
    if sr<35 or sr>110: raise ValueError('shade scale unsupported')
    if cv2.contourArea(c)/(math.pi*sr*sr)<.82: raise ValueError('shade outline not circular')
    # Neutral, very dark rubber. Thin metal legs and small fittings disappear
    # under a disk opening; retain only large compact round support pads.
    gray=cv2.cvtColor(a,cv2.COLOR_RGB2GRAY)
    dark=((gray<95)&(hsv[:,:,1]<110)).astype(np.uint8)
    dark=cv2.morphologyEx(dark,cv2.MORPH_OPEN,cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(17,17)))
    pads=[]
    for st,cen,m in _components(dark):
        if st[4]<1800 or st[4]>10000: continue
        cs,_=cv2.findContours(m.astype(np.uint8),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)
        cc=max(cs,key=cv2.contourArea)
        (x,y),r=cv2.minEnclosingCircle(cc)
        fill=cv2.contourArea(cv2.convexHull(cc))/(math.pi*r*r)
        if fill>.85 and 25<r<60:
            pads.append([float(x),float(y)])
    if len(pads)!=3: raise ValueError('expected three unoccluded rubber feet, measured %s'%len(pads))
    hub=np.mean(pads,axis=0)
    pads.sort(key=lambda p:math.atan2(p[1]-hub[1],p[0]-hub[0]))
    ds=[]
    for p,q in zip(pads,pads[1:]+pads[:1]):
        dx,dy=q[0]-p[0],q[1]-p[1]
        ds.append((dx*(sy-p[1])-dy*(sx-p[0]))/math.hypot(dx,dy))
    d=min(ds)
    if abs(d)<15: raise ValueError('shade center too near support boundary')
    return {'shade':{'center':[float(sx),float(sy)],'radius':float(sr)},
            'feet':pads,'signed_edge_distances_px':[float(x) for x in ds],
            'margin_px':float(abs(d)), 'decision':'yes' if d>0 else 'no',
            'trace':['segment circular enamel shade','recover three rubber foot centers',
                     'sort feet around centroid','test shade center against all three oriented edges']}


def explain_from_image(image,question,choices):
    return measure(image)


def answer_from_image(image,question,choices):
    d=measure(image)['decision']
    for choice in choices:
        if choice.lower()==d: return choice
    raise ValueError('public choices must contain yes/no')

7 · Functional inspection

Which power strip has two neighboring empty sockets?

Left power stripRight power strip

Rendered scene: Neighboring empty sockets

Answer from both programs: Right power strip

  • Left strip, top to bottom: plug, plug, empty, plug, empty.
  • Right strip, top to bottom: plug, plug, empty, empty, plug. Two empty sockets are neighbours.
Rendered scene: Neighboring empty sockets

Answer from both programs: Left power strip

  • Left strip, top to bottom: plug, plug, empty, empty, plug. Two empty sockets are neighbours.
  • Right strip, top to bottom: plug, plug, empty, plug, empty.
Rendered scene: Neighboring empty sockets

Answer from both programs: Right power strip

  • Left strip, top to bottom: plug, empty, plug, empty, plug.
  • Right strip, top to bottom: plug, plug, plug, empty, empty. Two empty sockets are neighbours.
Watch the inverse program work, step by step
Inverse program as generated (74 lines)
"""Pixel-only socket occupancy and ordered run detection."""
import cv2
import numpy as np


def measure(image):
    rgb=np.asarray(image.convert('RGB'))
    # Pale plastic housings on a darker neutral bench. No unique part colors.
    light=((rgb.min(axis=2)>176)&((rgb.max(axis=2)-rgb.min(axis=2))<50)).astype(np.uint8)*255
    light=cv2.morphologyEx(light,cv2.MORPH_CLOSE,np.ones((5,5),np.uint8))
    contours,_=cv2.findContours(light,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
    bodies=[]
    for c in contours:
        rect=cv2.minAreaRect(c); a,b=rect[1]
        if min(a,b)>65 and max(a,b)/min(a,b)>3.4 and cv2.contourArea(c)>20000:
            bodies.append((float(rect[0][0]),rect))
    bodies=sorted(bodies,key=lambda b:b[0])
    if len(bodies)!=2:
        raise ValueError('Ambiguous housing detection: '+str(len(bodies)))
    result=[]
    for x,rect in bodies:
        box=cv2.boxPoints(rect).astype(np.float32)
        top=box[np.argsort(box[:,1])[:2]]; bottom=box[np.argsort(box[:,1])[2:]]
        tl,tr=top[np.argsort(top[:,0])]; bl,br=bottom[np.argsort(bottom[:,0])]
        w=int(round((np.linalg.norm(tr-tl)+np.linalg.norm(br-bl))/2))
        h=int(round((np.linalg.norm(bl-tl)+np.linalg.norm(br-tr))/2))
        transform=cv2.getPerspectiveTransform(np.array([tl,tr,br,bl],np.float32),np.array([[0,0],[w-1,0],[w-1,h-1],[0,h-1]],np.float32))
        patch=cv2.warpPerspective(rgb,transform,(w,h))
        dark=(patch.max(axis=2)<150).astype(np.uint8)
        dark[:int(h*.055)]=0; dark[int(h*.945):]=0
        dark[:,:int(w*.16)]=0; dark[:,int(w*.84):]=0
        n,labels,stats,centers=cv2.connectedComponentsWithStats(dark,8)
        parts=[]
        for i in range(1,n):
            xx,yy,ww,hh,area=stats[i]
            if area>=10:
                parts.append(dict(y=float(centers[i][1]),area=int(area),bbox=[int(xx),int(yy),int(ww),int(hh)]))
        parts.sort(key=lambda p:p['y'])
        if len(parts)<5:
            raise ValueError('Not enough visible stations')
        gaps=[parts[i+1]['y']-parts[i]['y'] for i in range(len(parts)-1)]
        cuts=sorted(sorted(range(len(gaps)),key=lambda i:gaps[i],reverse=True)[:4])
        groups=[]; start=0
        for end in cuts+[len(parts)-1]:
            groups.append(parts[start:end+1]); start=end+1
        stations=[]
        for g in groups:
            area=sum(p['area'] for p in g)
            ratio=area/(w*w)
            if .06<ratio<.16:
                raise ValueError('Ambiguous occupancy area')
            stations.append(dict(y=float(np.mean([p['y'] for p in g])),dark_area=area,normalized_area=round(ratio,4),occupied=bool(ratio>.11)))
        empty=[not s['occupied'] for s in stations]
        pairs=[i for i in range(4) if empty[i] and empty[i+1]]
        result.append(dict(center_x=x,housing_size=[w,h],stations=stations,empty_neighbor_pairs=pairs,qualifies=bool(pairs)))
    good=[i for i,s in enumerate(result) if s['qualifies']]
    if len(good)!=1:
        raise ValueError('Expected one unambiguous free pair')
    return dict(strips=result,decision='left' if good[0]==0 else 'right',trace='Detect housings; rectify rows; group dark bores or plug bodies into five ordered stations; classify occupancy by area; find consecutive empty stations.')


def answer_from_image(image,question,choices):
    m=measure(image)
    word='Left' if m['decision']=='left' else 'Right'
    matches=[c for c in choices if c.lower().startswith(word.lower())]
    if len(matches)!=1:
        raise ValueError('Unrecognized choices')
    return matches[0]


def explain_from_image(image,question,choices):
    m=measure(image)
    m['answer']=answer_from_image(image,question,choices)
    return m

8 · Mechanisms

For ideal pulleys, how many times the rope tension is the upward force on the lower moving block?

2 times3 times4 times5 times

Rendered scene: Pulley supporting strands

Answer from both programs: 4 times

  • 5 vertical rope runs between the blocks.
  • The rightmost run continues down to the free end, so it is the pulled end (blue).
  • The other 4 runs hold up the lower block (pink).
Rendered scene: Pulley supporting strands

Answer from both programs: 3 times

  • 4 vertical rope runs between the blocks.
  • The rightmost run continues down to the free end, so it is the pulled end (blue).
  • The other 3 runs hold up the lower block (pink).
Rendered scene: Pulley supporting strands

Answer from both programs: 5 times

  • 6 vertical rope runs between the blocks.
  • The rightmost run continues down to the free end, so it is the pulled end (blue).
  • The other 5 runs hold up the lower block (pink).
Inverse program as generated (63 lines)
"""Pixel-only rope incidence measurement for exposed planar tackles."""
import numpy as np
import cv2


def _runs(row):
    ids=np.flatnonzero(row)
    if len(ids)==0:return []
    groups=np.split(ids,np.where(np.diff(ids)>1)[0]+1)
    return [(int(g[0]),int(g[-1])) for g in groups if len(g)>=3]


def measure(image):
    rgb=np.asarray(image.convert('RGB'))
    hsv=cv2.cvtColor(rgb,cv2.COLOR_RGB2HSV)
    # Natural tan rope is the sole warm material, not an answer code.
    mask=((hsv[:,:,0]>=8)&(hsv[:,:,0]<=35)&(hsv[:,:,1]>85)&(hsv[:,:,2]>65)).astype(np.uint8)
    mask=cv2.morphologyEx(mask,cv2.MORPH_CLOSE,np.ones((3,3),np.uint8))
    yy,xx=np.nonzero(mask)
    if len(xx)<300:raise ValueError('No sufficiently visible rope')
    ymin,ymax=int(yy.min()),int(yy.max())
    # Long repeated vertical runs define the clear region between fixed and
    # moving sheaves; curved wrap bands are short and fail this persistence test.
    lo=ymin+int(.18*(ymax-ymin)); hi=ymin+int(.70*(ymax-ymin))
    records=[]
    for y in range(lo,hi):
        groups=_runs(mask[y]>0)
        groups=[g for g in groups if g[1]-g[0]<22]
        if 3<=len(groups)<=6:
            records.append((y,tuple(round((a+b)/2) for a,b in groups)))
    best=[]; current=[]
    for rec in records:
        if current and (rec[0]!=current[-1][0]+1 or len(rec[1])!=len(current[-1][1]) or max(abs(a-b) for a,b in zip(rec[1],current[-1][1]))>3):
            if len(current)>len(best):best=current
            current=[]
        current.append(rec)
    if len(current)>len(best):best=current
    if len(best)<35:raise ValueError('Ambiguous rope runs in inter-block gap')
    centers=np.median(np.asarray([p[1] for p in best]),axis=0)
    if min(np.diff(centers))<25:raise ValueError('Unresolved adjacent rope runs')
    # The rightmost rope is followed below the moving sheave band to its free
    # grip; all other vertical strands terminate on or wrap the moving block.
    x=int(centers[-1]); other=[int(c) for c in centers[:-1]]
    bottom=[]
    for cx in [*other,x]:
        rows=np.flatnonzero(np.any(mask[:,max(0,cx-3):cx+4]>0,axis=1))
        bottom.append(int(rows.max()) if len(rows) else -1)
    if bottom[-1]<max(bottom[:-1])+25:raise ValueError('Hauling tail not visibly separate below moving block')
    supports=len(centers)-1
    return {'vertical_rope_x':centers.tolist(),'gap_rows':[best[0][0],best[-1][0]],
       'rope_bottom_rows':bottom,'free_tail_x':x,'supporting_runs':supports,
       'decision':str(supports)+' times',
       'trace':'Identify separate downward hauling tail; exclude it. Each remaining taut vertical run applies one upward tension to the moving assembly.'}


def answer_from_image(image,question,choices):
    result=measure(image)['decision']
    if result not in choices:raise ValueError('Unsupported answer choices')
    return result


def explain_from_image(image,question,choices):
    return measure(image)

9 · 3D spatial relationships

Is the central roof seam a ridge or a valley?

RidgeValley

Rendered scene: Roof ridge or valley

Answer from both programs: Ridge

  • Fold found at x = 486 (pink).
  • Along the roof's far edge, the image height changes by -0.31 per pixel left of the fold and +0.26 right of it.
  • The edge peaks at the fold: a ridge.
Rendered scene: Roof ridge or valley

Answer from both programs: Valley

  • Fold found at x = 522 (pink).
  • Along the roof's far edge, the image height changes by +0.25 per pixel left of the fold and -0.34 right of it.
  • The edge dips at the fold: a valley.
Rendered scene: Roof ridge or valley

Answer from both programs: Ridge

  • Fold found at x = 508 (pink).
  • Along the roof's far edge, the image height changes by -0.30 per pixel left of the fold and +0.30 right of it.
  • The edge peaks at the fold: a ridge.
Inverse program as generated (61 lines)
"""Image-only roof silhouette recovery and joined-slope computation."""
from __future__ import annotations
import numpy as np
import cv2


def measure(image):
    rgb=np.asarray(image.convert('RGB'),dtype=np.float32)
    # Natural dark blue-gray roof material versus warm timber and neutral ground.
    mask=((rgb[:,:,2]-rgb[:,:,0]>12)&(rgb[:,:,1]-rgb[:,:,0]>5)&
          (rgb[:,:,0]<165)).astype(np.uint8)
    mask=cv2.morphologyEx(mask,cv2.MORPH_CLOSE,np.ones((3,3),np.uint8))
    n,labels,stats,_=cv2.connectedComponentsWithStats(mask,8)
    if n<2:
        raise ValueError('No roof surface recovered')
    idx=1+int(np.argmax(stats[1:,cv2.CC_STAT_AREA]))
    mask=labels==idx
    yy,xx=np.nonzero(mask)
    if len(xx)<15000:
        raise ValueError('Roof surface too small')
    lo,hi=int(xx.min()),int(xx.max())
    xs=np.arange(lo+0.08*(hi-lo),hi-0.08*(hi-lo),1).astype(int)
    ys=np.array([np.flatnonzero(mask[:,x])[0] if mask[:,x].any() else np.nan for x in xs])
    valid=np.isfinite(ys);xs=xs[valid].astype(float);ys=ys[valid].astype(float)
    # Fit continuous two-line roof edge with an unknown seam position.
    # y = intercept + tilt*x + fold*|x-seam|. This removes common camera tilt.
    best=None
    for seam in np.linspace(lo+0.32*(hi-lo),lo+0.68*(hi-lo),81):
        mat=np.column_stack([np.ones(len(xs)),xs-seam,np.abs(xs-seam)])
        coef=np.linalg.lstsq(mat,ys,rcond=None)[0]
        residual=ys-mat@coef
        error=float(np.mean(np.minimum(residual**2,100)))
        if best is None or error<best[0]:
            best=(error,float(seam),coef)
    error,seam,coef=best
    left_slope=float(coef[1]-coef[2]);right_slope=float(coef[1]+coef[2])
    # Image y points downward. A ridge is /\, so left slope <0, right >0.
    if error>20 or min(abs(left_slope),abs(right_slope))<0.08:
        raise ValueError('Ambiguous or unsupported roof boundary')
    if left_slope<0 and right_slope>0:
        decision='ridge'
    elif left_slope>0 and right_slope<0:
        decision='valley'
    else:
        raise ValueError('Roof has no supported central fold')
    return {'roof_bounds':[lo,int(yy.min()),hi,int(yy.max())],
            'roof_pixels':int(mask.sum()),'seam_x':seam,
            'left_image_slope':left_slope,'right_image_slope':right_slope,
            'fit_mse_clipped':error,'decision':decision,
            'trace':['Segment roof surface from pixels','Extract exposed rear roof edge',
                     'Fit adjoining edge slopes after common tilt','Classify their convex or concave fold']}


def explain_from_image(image,question,choices):
    report=measure(image)
    report['answer']=next(c for c in choices if c.lower()==report['decision'])
    return report


def answer_from_image(image,question,choices):
    return explain_from_image(image,question,choices)['answer']

From program renderings to photographs

Each pair below shows a rendering drawn by a world’s program and a photograph that an image-generation model produced from that rendering. A world passes only if the same pixel-only inverse program recovers the expected answer from all five renderings and all five photographs. In a two-hour run, 20 of 40 worlds passed; 11 of them are shown here, with 54 image pairs.

This is a first feasibility experiment run after submission, not a benchmark. These are development samples: the check is automatic, there was no formal human review, and a pass does not certify that a photograph is free of other artifacts. The worlds shown are a selection of the passing ones, and the pairs were screened by eye before inclusion, so a world may show fewer than five pairs.

Unusual objects

An everyday object is shown either in its usual form or in an unusual but physically plausible one, and the question asks which.

Wall outlet, ground hole up or down

Is the ground hole below the two slots?

yesno

Answer: no · recovered by the inverse program from both images
Answer: yes · recovered by the inverse program from both images
Answer: no · recovered by the inverse program from both images
Answer: yes · recovered by the inverse program from both images
Answer: no · recovered by the inverse program from both images

Peanut pod with two or three bulges

Does the peanut pod have two bulges?

yesno

Answer: no · recovered by the inverse program from both images
Answer: yes · recovered by the inverse program from both images
Answer: no · recovered by the inverse program from both images
Answer: yes · recovered by the inverse program from both images
Answer: no · recovered by the inverse program from both images

Matching

One of three candidates has to be matched to a reference object by its pattern or shape.

Matching domino

Which domino has the same pip layout as the domino above, allowing a half-turn?

leftcenterright

Answer: center · recovered by the inverse program from both images
Answer: left · recovered by the inverse program from both images
Answer: right · recovered by the inverse program from both images
Answer: center · recovered by the inverse program from both images
Answer: left · recovered by the inverse program from both images

Puzzle piece and gap

Which loose puzzle piece fits the empty space, allowing rotation?

leftcenterright

Answer: center · recovered by the inverse program from both images
Answer: left · recovered by the inverse program from both images
Answer: right · recovered by the inverse program from both images
Answer: center · recovered by the inverse program from both images
Answer: left · recovered by the inverse program from both images

Pair of knitted gloves

Which glove has the same knitted stripe pattern as the single glove above?

leftcenterright

Answer: center · recovered by the inverse program from both images
Answer: left · recovered by the inverse program from both images
Answer: right · recovered by the inverse program from both images
Answer: center · recovered by the inverse program from both images
Answer: left · recovered by the inverse program from both images

Shaker and loose lid

Which shaker has the same hole arrangement as the loose lid, allowing rotation or reflection?

leftcenterright

Answer: left · recovered by the inverse program from both images
Answer: right · recovered by the inverse program from both images
Answer: center · recovered by the inverse program from both images
Answer: left · recovered by the inverse program from both images

Tracing

Two cables cross or touch, and the question asks which object a cable is connected to.

Headphones on a desk

Which headphones are plugged into the laptop?

red headphonesblue headphones

Answer: blue headphones · recovered by the inverse program from both images
Answer: red headphones · recovered by the inverse program from both images
Answer: blue headphones · recovered by the inverse program from both images
Answer: red headphones · recovered by the inverse program from both images
Answer: blue headphones · recovered by the inverse program from both images

Guitar, pedals and amplifier

Which pedal is connected to the guitar?

orange pedalgreen pedal

Answer: green pedal · recovered by the inverse program from both images
Answer: orange pedal · recovered by the inverse program from both images
Answer: green pedal · recovered by the inverse program from both images
Answer: orange pedal · recovered by the inverse program from both images
Answer: green pedal · recovered by the inverse program from both images