MULTI-GAME OFFLINE REINFORCEMENT LEARNING

MHWA.
Multi-timescale
Hierarchical
World-Action Model

Learn the world.
Act with structure.

A strided high-level context channel meets a locally reactive mixture of experts. One world-action model, fourteen Atari games, and less training computation.

01 / THE MODEL IN ACTION AGENT GAMEPLAY
One model / multiple worldsFour games at a time · real RGB recordings

Pengcheng Pan1,2,4,*Guoqing Ma1,3,4,*Yuhan Zhang1,2,4Yang Chen1,4Yichen Liu1,3,4Ziheng Li1,2,4Shan Yu1,4,†

1 Laboratory of Brain Atlas and Brain-inspired Intelligence, Institute of Automation, Chinese Academy of Sciences

2 School of Artificial Intelligence, University of Chinese Academy of Sciences

3 School of Future Technology, University of Chinese Academy of Sciences

4 State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences

Beijing, China · * Equal contribution · † Corresponding author: Shan Yu

AGGREGATE PERFORMANCE0.884IQM HNS

RLiable seed-by-game aggregation
Three training seeds · 14 games

MODEL SIZE77million parameters

Numerically comparable to the
published JOWA-150M result

TRAINING COMPUTATION70.88% fewer FLOPs

Forward + backward per sequence
Compared with JOWA-150M

THE QUESTION

Can better structure
reduce the need
for greater scale?

Small prediction errors become bigger decisions.

Autoregressive world models can drift as they roll out their own predictions. Scaling model capacity helps, but increases training cost. MHWA explores a complementary direction: organize context and computation across timescales.

The high-level Manager processes input embeddings sampled every four tokens and learns to predict the next sampled token, which may represent an observation or an action. Within the Worker, a gating network combines high-level context with the current token representation to select and weight experts.

Explore the full formulation

02 / ARCHITECTURE

Two levels.
One world-action model.

A high-level context channel guides sparse computation at the token level.

MHWA architecture: the strided Manager predicts the next sampled observation or action token, and the Worker's gating network uses context and the current token representation to select and weight experts.
MHWA architecture. The Manager predicts the next sampled observation or action token. High-level context and the current token representation guide expert selection and weighting in the Worker.
01

Learn coarse context

The Manager processes input embeddings sampled every four tokens. Its prediction head learns to predict the next sampled observation or action token.

02

Separate learning signals

Stop-gradient isolates the Manager from low-level losses, while the projector, router, and Worker remain trainable.

03

Route local computation

The Worker's gating network combines high-level context with the current token representation to select and weight two of four experts in each MoE layer.

03 / PAPER RESULTS

Strong performance.
Less training computation.

14 Atari games · 10% subsampled offline data
Matched MHWA / JOWA-70M comparison across three seeds.

Aggregate Atari performance

IQM HNS ↑
MHWA and JOWA aggregate performanceMatched JOWA-70M has IQM 0.529, confidence interval 0.459 to 0.602. MHWA-77M has IQM 0.884, confidence interval 0.736 to 1.031. The published JOWA-150M reference is 0.833.0.00.30.60.9JOWA-70MMHWA-77MJOWA-150M*0.5290.8840.833*

Whiskers: 95% stratified-bootstrap confidence intervals. * Published reference; separate from the matched three-seed comparison.

FORWARD + BACKWARD / SEQUENCE

JOWA-150M165.524 GFLOPs
JOWA-70M84.238 GFLOPs
MHWA-77M48.199 GFLOPs
70.88%less training computation
than JOWA-150M

Identical input shapes: 8 frames / 296 tokens per sequence. These figures measure training computation, not inference speed.

Matched performance and training computation, from the paper.
ModelIQM HNS95% confidence intervalGFLOPs / sequence
JOWA-70M0.529[0.459, 0.602]84.238
MHWA-77M0.884[0.736, 1.031]48.199
JOWA-150M · published reference0.833—165.524

IQM uses RLiable’s seed-by-game aggregation. The paper also reports a mean of seed-level IQMs of 0.867 ± 0.047; these are different aggregation procedures.

BEYOND THE TRAINING GAMES

On five held-out Atari games, MHWA reaches 0.643 ± 0.007 IQM DNS under the JOWA fine-tuning protocol, compared with the published JOWA-150M result of 0.647.

04 / WORLD-MODEL QUALITY

Look beyond
the next prediction.

Open-loop reconstruction measures how predictions evolve after conditioning on four real frames.

Paper figure comparing open-loop reconstruction MSE, PSNR, and SSIM across rollout steps five to eight for MHWA and JOWA baselines.
Open-loop reconstruction within the context window, from the paper. MHWA shows more stable reconstruction than the smaller monolithic baseline.

CITE THIS WORK

Build on MHWA.

The paper is available now. The public code release is in preparation.

Get in touch ↗
@misc{pan2026mhwa,
  title = {MHWA: Multi-timescale Hierarchical World-Action Model},
  author = {Pan, Pengcheng and Ma, Guoqing and Zhang, Yuhan
            and Chen, Yang and Liu, Yichen and Li, Ziheng and Yu, Shan},
  year = {2026},
  eprint = {2610.14727},
  archivePrefix = {arXiv}
}