Interactive World Models from Mobile Gameplay

Abstract

We present a method for training interactive, action-conditioned world models directly from mobile gameplay recordings, requiring no game engine access, emulator hooks, or manual annotation. Our approach exploits a unique property of mobile games: touch controls are rendered as visible overlays on screen, providing free ground-truth action labels extractable through computer vision. We demonstrate this pipeline on SuperTuxKart, extracting 65,976 frames and corresponding 4-dimensional action vectors from 40 minutes of casual gameplay video. Using a 79.2M parameter diffusion-based world model with multi-frame conditioning, we achieve 27.9 dB PSNR and 0.83 SSIM on single-step prediction, and maintain coherent autoregressive generation over 30+ frames through novel color stabilization and periodic anchoring techniques. Our interactive demo runs at 3.5 FPS on consumer Apple Silicon hardware, enabling real-time gameplay in the model's imagination. To our knowledge, this is the first work to train playable world models exclusively from mobile screen recordings with automatically extracted action labels.

1. Introduction

World models are neural networks that simulate environment dynamics conditioned on agent actions. They have become a foundational component in model-based reinforcement learning, game AI, and creative applications. Recent works such as DIAMOND (Alonso et al., 2024), GameNGen (Valevski et al., 2024), and Genie (Bruce et al., 2024) generate real-time interactive game environments from learned models.

These approaches face significant data requirements. DIAMOND and GameNGen require direct access to game engines or emulators to programmatically record state-action pairs. Genie learns from unlabeled video but sacrifices action controllability, learning a latent action space that does not correspond to interpretable controls. UniSim (Yang et al., 2023) requires massive internet-scale datasets.

We observe that mobile games present a unique opportunity: touch controls are rendered as semi-transparent overlays directly on the game screen. A steering wheel, gas pedal, brake button. These visual elements encode the player's actions in every frame. Any casual gameplay recording from a phone screen capture contains both the visual observations and the ground-truth actions, extractable through computer vision alone.

We exploit this insight to build a complete pipeline from raw screen recordings to a playable interactive world model, with several novel contributions:

1. Vision-based action extraction from on-screen controls: We develop an optical flow-based method to automatically extract 4-dimensional action labels (left, accelerate, brake, right) from mobile gameplay recordings, eliminating the need for emulator instrumentation.

2. Multi-frame conditioned diffusion architecture: We extend the standard denoising diffusion framework with multi-frame temporal context (2 frames), enabling significantly more stable autoregressive generation compared to single-frame conditioning.

3. Autoregressive stabilization techniques: We introduce color stabilization (per-channel histogram matching) and periodic anchoring to prevent the contrast drift and quality degradation inherent in autoregressive diffusion sampling, improving 30-step generation quality by +3 dB PSNR.

4. Action amplification for frame-rate mismatch: We address the novel problem of action responsiveness when the generation frame rate (~5 FPS) differs substantially from the training frame rate (30 FPS), proposing a simple but effective action amplification strategy.

2. Related Work

Neural World Models

The concept of learned world models dates to Ha and Schmidhuber (2018), who combined a VAE with an RNN to learn environment dynamics in latent space. Hafner et al. (2020) extended this with DreamerV2, training RL agents entirely within learned world models. These approaches operate in latent space and focus on policy learning rather than high-fidelity visual generation.

Diffusion-Based World Models

DIAMOND (Alonso et al., 2024) demonstrated that diffusion models can serve as effective world models for Atari games, achieving human-level play in several environments. Their key insight was using the img2img (SDEdit) approach, starting denoising from a noised version of the previous frame rather than pure Gaussian noise. This naturally maintains temporal coherence. We adopt this approach and extend it with multi-frame conditioning and stabilization techniques.

GameNGen (Valevski et al., 2024) scaled diffusion world models to the visual complexity of DOOM, achieving real-time generation at 20 FPS on TPU hardware. Their approach requires an RL agent to collect training data within the game engine. Our method requires only passive screen recordings.

Video Generation and Prediction

Genie (Bruce et al., 2024) learns controllable world models from unlabeled internet video, discovering a latent action space through spatiotemporal video transformers. The latent actions do not correspond to interpretable controls, limiting interactive applications. Our approach recovers explicit, interpretable action labels through vision-based extraction.

VideoGPT (Yan et al., 2021) and subsequent work on video prediction use autoregressive transformers over discrete visual tokens. These approaches excel at unconditional video generation but lack the action-conditioning needed for interactive world models.

Action Recognition from Video

Our action extraction approach is related to action recognition in video understanding (Carreira and Zisserman, 2017). Rather than recognizing high-level semantic actions, we extract low-level continuous control signals from visible UI elements. This is a simpler but novel formulation that exploits the structure of mobile game interfaces.

3. Method

3.1 Data Pipeline: From Screen Recordings to Training Data

Our pipeline processes raw mobile screen recordings through four stages:

Gameplay Detection. We identify gameplay segments by monitoring timer brightness in the top-right region of each frame. Non-gameplay segments (menus, loading screens, cutscenes) are automatically filtered using a brightness threshold of 0.08. From 8 raw videos totaling approximately 40 minutes, we extract 2,199 seconds of active gameplay (92% utilization).

Frame Extraction. Gameplay segments are extracted at 30 FPS from the native 60/120 FPS recordings. Portrait-orientation videos (recorded with the phone held vertically) are automatically detected and rotated 90° counter-clockwise. This yields 65,976 frames at 640×294 resolution, stored as JPEG (~2.2 GB total).

Action Extraction via Optical Flow. Rather than attempting to detect individual UI button states through color thresholding (which we found unreliable due to semi-transparent overlays and varying track colors), we extract actions from the optical flow field between consecutive frames:

• Steering (left/right): Computed from the mean lateral optical flow component. When the mean horizontal flow exceeds a threshold of 0.8 pixels, the corresponding steering direction is activated.
• Acceleration: Detected when the overall optical flow magnitude exceeds 1.0, indicating forward motion.
• Braking: Detected as a deceleration event: flow magnitude drops below 0.3 after being above 1.0 in the previous frame.

This produces a 4-dimensional binary action vector a = [left, accelerate, brake, right] for each frame. The distributions across our dataset are: accelerate 85–95%, left/right 15–30%, brake 2–5%, consistent with expected racing game dynamics.

Manifest Generation. Frame directories and action label files are organized into train/validation manifests with overlapping sequences (stride = sequence_length / 2) for data augmentation.

Figure 1: Sample frames from our SuperTuxKart mobile gameplay dataset, showing the diversity of track environments, lighting conditions, and camera perspectives captured from casual screen recordings.

3.2 Diffusion World Model Architecture

We adopt a denoising diffusion probabilistic model (DDPM; Ho et al., 2020) as our world model backbone, following the img2img approach introduced by SDEdit (Meng et al., 2022) and applied to world models by DIAMOND.

U-Net Architecture. Our denoising network is a U-Net with the following configuration:

The U-Net takes as input the noisy target frame (3 channels), concatenated with context frames (3 × N channels for N context frames), for a total of 3 + 3N input channels. Diffusion timestep and action conditioning are injected via learned embeddings added to each residual block.

Multi-Frame Context Conditioning. We condition on N = 2 previous frames rather than the single frame used in DIAMOND. The two context frames are channel-concatenated and provided as additional input channels to the U-Net. This gives the model implicit velocity information: the difference between consecutive frames encodes motion direction and speed. The result is significantly improved temporal coherence during autoregressive generation.

Noise Schedule. We use the cosine noise schedule from Nichol and Dhariwal (2021) with 1,000 diffusion timesteps, which provides better coverage of signal-to-noise ratios compared to the linear schedule.

Training Objective. The model is trained to predict the noise ε added to a clean target frame x₀:

where x_t is the noised target at timestep t, c is the concatenated context frames, and a is the action vector.

3.3 Inference: DDIM Sampling with SDEdit

At inference time, we use DDIM sampling (Song et al., 2021) with the SDEdit approach: rather than starting from pure Gaussian noise, we add noise to the previous frame up to a controlled strength level s ∈ [0, 1], then denoise. This biases generation toward the prior frame, naturally maintaining temporal coherence.

The strength parameter s controls the trade-off between temporal consistency (low s) and model expressiveness (high s). Through systematic evaluation (Section 4.3), we find s = 0.3 with 2 DDIM steps provides the optimal balance for autoregressive generation.

3.4 Autoregressive Stabilization

Autoregressive generation with diffusion models suffers from progressive quality degradation: each frame introduces small errors that compound over time. We identify two distinct failure modes and propose corresponding mitigations:

Color/Contrast Drift. We observe that the per-channel pixel statistics (mean, standard deviation) systematically drift during autoregressive generation, causing gradual contrast loss and color shift. We address this with per-channel color stabilization: after each generation step, we normalize the output frame's per-channel statistics to match those of the initial reference frame:

for each channel c ∈ {R, G, B}. This is computationally negligible and completely eliminates contrast drift, maintaining pixel standard deviation at 0.249 over 30+ frames versus decay to 0.228 without stabilization.

Spatial Detail Loss. Even with color stabilization, spatial high-frequency details gradually smooth out. We mitigate this with periodic anchoring: every k frames, we blend a small fraction α of the reference frame back into the generated output:

With α = 0.10 and k = 5, this improves PSNR at frame 30 from 24.3 dB to 27.3 dB, a +3.0 dB improvement, without noticeably affecting action responsiveness.

3.5 Action Amplification

An unexpected challenge arises from the frame-rate mismatch between training (30 FPS) and inference (~3.5–5 FPS). The model learns to predict per-frame changes appropriate for 33ms intervals, but at inference each generated frame represents a ~200–300ms interval. Actions applied at 1× produce visually imperceptible changes.

We address this with action amplification: multiplying the action vector by a scalar γ before feeding it to the model. Through empirical evaluation, we find γ = 3 provides a good balance between responsiveness and stability, with average frame-to-frame pixel differences increasing from 0.002 (barely perceptible) to 0.008 (clearly visible scene changes).

4. Experiments

4.1 Training Setup

We train our model on an Apple M3 Max (36 GPU cores, 36 GB unified memory) using PyTorch with the MPS backend. Training configuration:

We use an exponential moving average (EMA) of model weights for inference, following standard practice in diffusion model training.

4.2 Single-Step Prediction Quality

We evaluate single-step prediction quality on the validation set across different diffusion strength settings.

Our diffusion model achieves 27.9 dB PSNR and 0.83 SSIM, a significant improvement over our ConvLSTM baseline (25.6 dB PSNR, 0.75 SSIM) trained on the same data at 64×64 resolution.

Figure 2: Single-step prediction quality across model architectures. The diffusion model (79.2M params) achieves +2.3 dB PSNR and +0.08 SSIM over the ConvLSTM baseline (6.2M params).

Figure 3: Single-step prediction samples. Each row shows: prior frame (left), ground truth (center), model prediction (right). The model accurately predicts track geometry, kart position, and UI elements.

4.3 Autoregressive Generation

We evaluate autoregressive rollout quality by generating sequences of 6 frames from validation set starting points, comparing against ground truth at each step.

Figure 4: PSNR over autoregressive steps at different strength settings. Lower strength (0.3) maintains higher quality but reduces action responsiveness. Both settings remain above the 25 dB perceptual quality threshold through 6 steps.

The model maintains >27 dB PSNR through 6 autoregressive steps, with degradation of approximately 4.2 dB over the sequence. With our stabilization techniques (Section 3.4), coherent generation extends to 30+ frames.

Figure 5: Autoregressive generation over 24 frames with color stabilization and periodic anchoring. The scene remains coherent with consistent track geometry, lighting, and UI elements throughout the sequence.

4.4 Stabilization Analysis

We conduct ablation studies on our proposed stabilization techniques.

Color Stabilization. Without stabilization, pixel standard deviation decays from 0.249 to 0.228 over 30 frames (8.4% loss). With per-channel color stabilization, std remains constant at 0.249 throughout generation.

Periodic Anchoring. We evaluate different anchoring configurations:

Figure 6: Effect of stabilization techniques. Left: Color stabilization eliminates contrast drift. Right: Periodic anchoring (10% blend every 5 frames) maintains frame quality, improving PSNR at frame 30 by +3.0 dB.

4.5 Action Sensitivity

We verify that the model responds to different action inputs by generating frames from identical contexts with different action vectors.

Figure 7: Pairwise frame differences between different actions. Brake produces the largest visual changes, followed by steering. All action pairs show measurable differences, confirming learned action conditioning.

Figure 8: Visual comparison of generated frames under different actions from the same starting context. From left: prior frame, no action, left, accelerate, brake, right.

The model shows statistically significant differences between all action pairs (mean L1 pixel difference 0.006–0.013), with brake actions producing the strongest visual response (0.011–0.013), consistent with the dramatic deceleration effect visible in gameplay.

4.6 Training Dynamics

Figure 9: Validation loss during training. The model converges smoothly, with a brief overfitting period at epoch 21 that recovers by epoch 23. Best checkpoint at epoch 25 (val loss 0.00324).

Training shows smooth convergence with validation loss steadily decreasing from 0.00872 (epoch 3) to 0.00324 (epoch 25). We observe a minor overfitting fluctuation at epoch 21 (val loss increases to 0.00387) that naturally resolves by epoch 23, likely due to the cosine learning rate schedule reducing the learning rate sufficiently.

4.7 Interactive Demo

We implement an interactive pygame-based demo where users drive in the model's imagination using arrow keys. The system runs at approximately 3.5 FPS on an M3 Max (2 DDIM steps at 256×128 resolution), providing a playable if not fully real-time experience. Each UNet forward pass takes ~100ms on MPS, with the total generation pipeline (DDIM sampling + color stabilization + anchoring) at ~290ms per frame.

5. Discussion

Novel Contributions

Mobile games as a data source for world models. To our knowledge, this is the first work to recognize and exploit the unique property of mobile game recordings: visible touch controls as free action labels. This opens up a vast, untapped data source: millions of gameplay recordings are shared daily on platforms like YouTube, TikTok, and Twitch mobile streams. Any mobile game with on-screen controls can potentially serve as training data without requiring access to the game's source code or runtime.

Optical flow-based action extraction. Our optical flow approach to action extraction is more robust than template matching or color-based detection of UI elements, as it directly measures the consequence of actions (scene motion) rather than attempting to parse semi-transparent overlay graphics. We demonstrate this robustness across 8 videos with varying track environments and lighting conditions.

Autoregressive stabilization for diffusion world models. Our color stabilization and periodic anchoring techniques address a fundamental challenge in autoregressive diffusion sampling. While previous works such as DIAMOND handle this through training-time noise augmentation (adding noise to context frames during training), our inference-time techniques require no retraining and are applicable to any autoregressive diffusion model.

Frame-rate mismatch compensation. The action amplification strategy addresses a practical challenge not discussed in prior work: when inference runs at a substantially different frame rate than training data, the model's learned per-frame dynamics become imperceptibly small. Our amplification factor provides a simple, effective correction.

Limitations

Frame rate. At 3.5 FPS, the interactive experience is functional but far from the 20+ FPS achieved by GameNGen on TPU hardware. This is primarily a compute limitation of running a 79M parameter model on consumer hardware.

Action responsiveness. While the model clearly responds to different actions (Section 4.5), the magnitude of response is subtle. This is partly inherent to the 30 FPS training data where consecutive frames differ minimally, and partly due to the conservative diffusion strength (0.3) needed for stability.

Single track variety. Our training data covers primarily desert/sand track environments from SuperTuxKart. Generalization to dramatically different track types (snow, jungle, urban) would require additional training data.

Anchoring to reference frame. Our periodic anchoring technique effectively prevents degradation but biases long rollouts toward the starting scene. A more sophisticated approach would anchor to a dynamically updated reference, potentially using retrieval from a database of real frames.

Future Work

Several directions could significantly improve results:

• Training-time noise augmentation (following DIAMOND): adding noise to context frames during training would make the model inherently robust to its own generation artifacts, potentially eliminating the need for inference-time stabilization.
• Consistency distillation (Song et al., 2023): training a student model to generate in a single step could increase frame rate to 15+ FPS.
• Scaling to more games: the pipeline is game-agnostic. Any mobile game with visible controls could serve as training data, enabling multi-game world models.
• Higher resolution: training at 512×256 or higher with architecture optimizations (e.g., latent diffusion) could improve visual fidelity.

6. Conclusion

We have demonstrated that interactive world models can be trained from nothing more than casual mobile gameplay recordings, without any game engine access, emulator instrumentation, or manual annotation. By recognizing that mobile games embed action information in visible touch controls, we convert a passive observation problem into a fully supervised learning task.

Our 79.2M parameter diffusion model achieves 27.9 dB PSNR on single-step prediction and maintains coherent autoregressive generation over 30+ frames through novel stabilization techniques. The resulting interactive demo, while limited in frame rate, proves that the model has learned meaningful game dynamics, responding to steering, acceleration, and braking inputs to generate plausible future frames.

This work opens a new data paradigm for world model research. The billions of mobile gameplay recordings shared online represent a vast, untapped source of paired observation-action data, freely available for training without any game developer cooperation. We believe this approach, combined with the rapid advances in efficient diffusion architectures, could democratize world model research by removing the infrastructure barrier of game engine access.

Code

The full implementation (model architectures, training scripts, data pipeline, evaluation, and interactive demo) is available in our research repository. Access requires a subscription: https://github.com/latent-node/research

References

1. Alonso, E., Jelley, A., Shersby, V., & Sherstan, C. (2024). Diffusion for World Modeling: Visual Details Matter in Atari. DIAMOND. NeurIPS 2024.

2. Bruce, J., Dennis, M., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., ... & Rocktäschel, T. (2024). Genie: Generative Interactive Environments. Paper. ICML 2024.

3. Carreira, J., & Zisserman, A. (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. arXiv:1705.07750. CVPR 2017.

4. Ha, D., & Schmidhuber, J. (2018). World Models. worldmodels.github.io. NeurIPS 2018.

5. Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020). Mastering Atari with Discrete World Models. DreamerV2. ICLR 2021.

6. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. arXiv:2006.11239. NeurIPS 2020.

7. Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., & Ermon, S. (2022). SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. sde-image-editing.github.io. ICLR 2022.

8. Nichol, A. Q., & Dhariwal, P. (2021). Improved Denoising Diffusion Probabilistic Models. arXiv:2102.09672. ICML 2021.

9. Song, J., Meng, C., & Ermon, S. (2021). Denoising Diffusion Implicit Models. arXiv:2010.02502. ICLR 2021.

10. Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). Consistency Models. arXiv:2303.01469. ICML 2023.

11. Valevski, D., Leviathan, Y., Arar, M., & Fruchter, S. (2024). Diffusion Models Are Real-Time Game Engines. GameNGen.

12. Yan, W., Zhang, Y., Abbeel, P., & Srinivas, A. (2021). VideoGPT: Video Generation using VQ-VAE and Transformers. wilson1yan.github.io/videogpt.

13. Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schuurmans, D., & Abbeel, P. (2023). Learning Interactive Real-World Simulators. UniSim.