Execution Before Memory: Diagnosing Memory Failures in Video World Models

Code Paper
Oil study of an open tin, an orange marble, and a purple disk.
Weiyang Jin1, Jiwen Yu1, Sihan Xu2, Yuwei Niu3, Xihui Liu1
1University of Hong Kong    2University of Michigan    3Peking University

Overview

A memory score should separate three questions: was the scene set, was the change carried out, and does the last frame show it. Suites fold them into one total, so a miss reads as forgetting. MIST hides the target, then asks for it later. M3-Agent rewrites the failed move. The kept clip is an oracle: the same judge both scores and selects. That does not show the model stored the target.

Overview of MIST, WRBench, MBench-T, and the M3-Agent loop.
Figure 1. Three experiments. Do not add a number from one into another.

What one clip is asked

SetupWas the scene set before the reveal?
LeakDid the target show up while it was supposed to stay hidden?
RevealWas the asked change actually carried out?
InsideDoes the last frame match the target state?

Baseline, 43 cases: setup 0.852, leak 0/43, reveal 0.147, inside 0.047, and 41 clips never open. One kept rewrite: setup 0.851, reveal 0.492, inside 0.442. This 0.492 is MIST reveal, not an MBench state score. Best-of-five inside is 0.721, a different cell.

M3-Agent

At test time the agent reads the last frame, names the visible failure, and rewrites only that move. Actor, objects, and order stay. On MIST it regenerates the last shot five times and keeps the original eligible. Ties keep the earlier clip.

MIST's three shots and the M3-Agent loop on R0-03.
Figure 2. R0-03 is a wrong binding: the raw take opens onto another object.

One kept clip, then five tries

One kept clip is not best of five. Labels move from 41 never-open and 1 correct to 20 never-open, 10 wrong inside, and 13 correct.

Closed lids fall from 41 to 20. Best of five is 0.763 only when the target is named.
Figure 3. One label per clip, then best of five. Dashed marks withhold the target. Keep-best is an oracle.
What we doAnswer in promptRevealInsideClosedOk
Look, then change the moveyes0.7630.721529
Write the target, same moveyes0.6840.581324
Same sentence, new seed onlyyes0.6350.512421
Open and holdno0.2680.09363
Look, then change the moveno0.1740.047282

Same judge, K = 5. Naming the target already reaches 0.684. Rewriting the move reaches 0.763, but both arms may name the target. Withheld, the rewrite is 0.174 and 28 lids stay closed.

Official suites

These 10-second official suites use their own scorers. They are not the internal three-shot probe, and they do not share one fix.

WRBench gain is mostly the leave-return path. MBench-T keeps 23 baseline clips.
Figure 4. WRBench gains on leave–return. MBench-T keeps 23 of 30 baseline clips.
SuiteBaselineKeep-bestnWhat the gain is
WRBench unique T0.4520.57040Completing a missing leave–return. Caption stays task-context none.
MBench-T unique U0.2920.44630Mostly refusal. The baseline clip is kept on 23 of 30 items.

Published U 0.446 averages two oracles; one clip per item is 0.443. A last-frame selector that cannot see the scorer stays near baseline (T 0.466, U 0.248).

Selected comparisons

Drag the handle. Filtered examples, not the 43-case mean.

Raw Revised
Figure 5. R0-03. Raw opens onto the wrong object. Revised puts the orange marble in the tin.
Raw Revised
Figure 6. R0-10. Raw stays shut. Revised opens onto the blue disk and yellow cube.
Raw Revised
Figure 7. S04. Raw is empty. Revised shows the token inside.

Conclusion

The scene is set and the gap does not leak, yet 41 of 43 clips never show the change. Revising the failed move opens more lids than naming the target, and the result can still be wrong. Reveal goes from 0.147 to 0.492. Official T goes from 0.452 to 0.570; U goes from 0.292 to 0.446, with 23 of 30 baseline clips kept.

The score is lost before persistence is tested. These cells do not show that the model stored the target. Report setup, the change, and the last frame separately.

Related

Paper · Code. Wan2.2 code · paper. Qwen code. WRBench page · paper. MBench page · paper.

BibTeX

@article{jin2026execution,
  title={Execution Before Memory: Diagnosing Memory Failures in Video World Models},
  author={Jin, Weiyang and Yu, Jiwen and Xu, Sihan and Niu, Yuwei and Liu, Xihui},
  journal={arXiv preprint arXiv:2610.23542},
  year={2026}
}