DiM-WAM: World Action Modeling with Diverse Historical Event Memory

Kai Wang1,2, Zhaopeng Gu1,2, Yixiang Chen1, Yuan Xu1, Qisen Ma1
Jiabing Yang1, Zhaowen Li2,*, Yan Huang1,3,*, Liang Wang1, Peng Su2

1 CASIA    2 Yinwang Intelligent Technology    3 FiveAges
* Corresponding authors

arXiv Code Coming Soon Simulation Videos Real-World Videos
Four real-world Franka Panda manipulation tasks.

Abstract

Abstract

DiM-WAM augments a world action model with Diverse Historical Event Memory (DHEM) for temporally dependent robot manipulation.

World action models (WAMs) jointly predict future visual states and actions, but short local context limits temporally dependent tasks. We introduce DiM-WAM, which augments a WAM with Diverse Historical Event Memory (DHEM). DHEM uses bank-conditioned candidate features, novelty-aware selection, and accumulated-mass-weighted fusion to retain complementary event tokens in bounded memory; these tokens condition video and action denoising, while auxiliary supervision encourages coarse progress cues. In a training-matched comparison with LingBot-VA on RMBench, DiM-WAM improves the average full-task success rate from 34.8% to 69.8%. Under the same demonstration and evaluation protocol on four real-world tasks, it improves the average stage success ratio from 70.6% to 93.5% and the average full-task success rate from 52.5% to 90.0%.

Framework

Framework

DiM-WAM overview with the model framework, training supervision, and redundancy-based memory merging.
Overview of DiM-WAM: (a) the DHEM-conditioned WAM framework, (b) task-progress supervision, and (c) redundancy-based memory merging.

(a) DHEM-conditioned WAM

Bank-specific queries extract complementary event candidates from real observations. Retained memory tokens receive bank-identity and slot-relative time embeddings, then condition both video and action denoising alongside the sliding-window KV cache.

(b) Training supervision

Historical memory tokens support an auxiliary progress prediction objective. Clean video and action latents are used only as conditioning during training, while the model denoises future video and action latents.

(c) Redundancy-based merging

Each bounded bank preserves an initial-state anchor and the latest event. When full, novelty-aware selection discards a redundant arrival or merges the most redundant adjacent history pair using accumulated-mass-weighted token and timestamp fusion.

Simulation Experiments

Simulation Experiments

RMBench evaluates long-horizon non-Markovian manipulation tasks where local observations can become ambiguous after key historical evidence leaves the context window.

69.8% RMBench total average success

Controlled LingBot-VA baseline: 34.8%; Mem-0 benchmark baseline: 42.0%.

80.6% Average success on M(1) tasks

Improves over controlled LingBot-VA at 34.2% and Mem-0 at 52.8%.

56.3% Average success on M(n) tasks

Improves over LingBot-VA at 35.5% and Mem-0 at 28.5%.

Task TMC DP ACT pi0.5 X-VLA Mem-0 Fast-WAM† LingBot-VA† DiM-WAM†
Observe and Pick Up M(1) 1.0 1.0 9.0 9.0 4.0 2.0 4.0 13.0
Rearrange Blocks M(1) 0.0 29.0 13.0 13.0 89.0 0.0 63.0 99.0
Put Back Block M(1) 0.0 0.0 11.0 18.0 90.0 0.0 41.0 98.0
Swap Blocks M(1) 11.0 2.0 24.0 16.0 67.0 3.0 38.0 96.0
Swap T M(1) 20.0 2.0 15.0 3.0 14.0 9.0 25.0 97.0
Average M(1) 6.4 6.8 14.4 11.8 52.8 2.8 34.2 80.6
Battery Try M(n) 10.0 19.0 16.0 26.0 28.0 5.0 33.0 48.0
Blocks Ranking Try M(n) 10.0 0.0 6.0 1.0 18.0 17.0 48.0 87.0
Cover Blocks M(n) 0.0 0.0 0.0 2.0 68.0 0.0 42.0 56.0
Press Button M(n) 0.0 0.0 0.0 0.0 0.0 0.0 19.0 34.0
Average M(n) 5.0 4.8 5.5 7.3 28.5 5.5 35.5 56.3
Total Average - 5.8 5.9 10.4 9.8 42.0 4.0 34.8 69.8

Simulation Case Videos

DiM-WAM success cases for the nine RMBench long-horizon tasks.

Real-World Experiments

Real-World Experiments

Four long-horizon Franka Panda tasks evaluate whether memory can preserve target identity, spatial state, and event order across multiple real-world stages.

93.5% Real-world average stage success

Improves over LingBot-VA at 70.6% across four Franka tasks.

90.0% Real-world average full-task success

Preserves target identity, spatial state, and event order across stages.

Method Find Blue Block Line Swap Triangle Swap Press Twice Avg. (%)
SSR SR SSR SR SSR SR SSR SR SSR SR
pi0.5 0/40 0/10 3/60 0/10 0/60 0/10 0/30 0/10 1.3 0.0
Fast-WAM 0/40 0/10 0/60 0/10 0/60 0/10 0/30 0/10 0.0 0.0
LingBot-VA 21/40 1/10 47/60 6/10 31/60 4/10 30/30 10/10 70.6 52.5
DiM-WAM 39/40 9/10 55/60 9/10 51/60 8/10 30/30 10/10 93.5 90.0

Real-World Experiment Videos

Rows are the four Franka tasks and columns are the evaluated methods.

Task / Method
pi0.5
Fast-WAM
LingBot-VA
DiM-WAM

Memory Analysis

Memory Behavior Analysis

The retained events and token-space structure show how different banks preserve complementary historical evidence during one complete episode.

Timeline of retained memory events across memory banks.
Retained memory events over normalized task progress. Each row corresponds to one memory bank, and each dot denotes a retained event token.
PCA visualization of memory tokens across banks.
PCA of memory tokens from the same episode. Colors and marker shapes indicate different memory banks, while ellipses summarize bank-wise token distributions.