Skip to main content

World Models & Spatial Intelligence

From Representation to Prediction, Planning and Physical Intelligence

CUHK(SZ) ยท SAI ยท BL&SP Research Group

Start Learning โ†’Explore Roadmap

Choose Your Path

๐Ÿงฉ

Foundations

Fill the gaps โ€” linear algebra, probability, PyTorch, DL, Transformers, CV, RL.

Target learner
Anyone arriving from an adjacent field; dip in as needed.
Modules
8 modules
Estimated time
~2 weeks part-time
Prerequisites
None โ€” this is the on-ramp

Optional & non-linear: refer back as needed.

Start Track โ†’
๐Ÿง‘โ€๐Ÿ”ฌ

World Model Scientist

Representation โ†’ Dynamics โ†’ Prediction โ†’ Planning โ†’ Evaluation.

Target learner
Researchers & engineers who want to build world models.
Modules
14 modules
Estimated time
~8โ€“10 weeks part-time
Prerequisites
Grad-level ML or equivalent self-study

Exit: reproduce Dreamer / TD-MPC-class systems.

Start Track โ†’
๐Ÿ‘ท

Spatial & Embodied Engineer

Geometry โ†’ 3D โ†’ SLAM โ†’ Spatial Memory โ†’ Navigation โ†’ Robot.

Target learner
Robotics, autonomous driving, AR/VR and spatial computing builders.
Modules
13 modules
Estimated time
~8โ€“10 weeks part-time
Prerequisites
Grad-level ML + basic geometry

Exit: build a perception โ†’ state estimation โ†’ planning system.

Start Track โ†’

Course Matrix

Every module pairs one lecture source, one canonical paper and one hands-on lab.

ModuleLearnWatchReadBuild
State Space ModelsKalman filter & LGSSM, belief-state inferenceUPenn CIS6280 L04Kalman (1960)Lab 1Lab 1 โ€” Open in ColabLab 1 โ€” Open in Colab: lab01_kalman_filter
Latent DynamicsVAE + latent prediction, observation โ†’ latent stateUPenn CIS6280 L07โ€“08PlaNet (Hafner et al. 2019)Lab 2Lab 2 โ€” Open in ColabLab 2 โ€” Open in Colab: lab02_latent_dynamics
RSSMRecurrent + stochastic latent state, prior/posterior KLUPenn CIS6280 L08DreamerV1 (Hafner et al. 2020)Lab 3Lab 3 โ€” Open in ColabLab 3 โ€” Open in Colab: lab03_tiny_rssm
Planning with World ModelsMPC / CEM / MPPI in latent spaceUPenn CIS6280 L09TD-MPC (Hansen et al. 2022)Lab 4Lab 4 โ€” Open in ColabLab 4 โ€” Open in Colab: lab04_mpc_cem_planning
Spatial RepresentationVolume rendering, NeRF / 3D Gaussian SplattingCMU 16-825 (NeRF & Differentiable Rendering)NeRF (Mildenhall et al. 2020)Lab 5Lab 5 โ€” Open in ColabLab 5 โ€” Open in Colab: lab05_nerf_gaussian_splatting
Dynamic 3D / 4D WorldsScene flow, deformation fields, 4D GaussiansCMU 16-825 (Dynamic 3D Representations)4D Gaussian Splatting (Wu et al. 2024)Lab 6Lab 6 โ€” Open in ColabLab 6 โ€” Open in Colab: lab06_dynamic_4d_worlds
Video World ModelsDiffusion / flow matching, action-conditioned predictionUPenn CIS6280 L11โ€“13Genie (Bruce et al. 2024)Lab 7Lab 7 โ€” Open in ColabLab 7 โ€” Open in Colab: lab07_video_world_model
SLAM / VIOFactor graphs, visual-inertial odometry, loop closureMIT 16.485 VNAV (VIO & SLAM)ORB-SLAM (Mur-Artal et al. 2015)Lab 8Lab 8 โ€” Open in ColabLab 8 โ€” Open in Colab: lab08_navigation_world_model
World Model + PolicyActor-critic in imagination, MBRL closed loopUPenn CIS6280 L10DreamerV3 (Hafner et al. 2023)Lab 9Lab 9 โ€” Open in ColabLab 9 โ€” Open in Colab: lab09_world_model_policy
OOD, Drift & EvaluationCalibration, rollout error, closed-loop driftUPenn CIS6280 L23MBPO (Janner et al. 2019)Lab 10Lab 10 โ€” Open in ColabLab 10 โ€” Open in Colab: lab10_ood_drift_evaluation

Hands-on Labs

All labs run one-click on free Colab GPUs. Eleven labs plus a capstone โ€” here are the first three.

Lab 0 ยท Tiny World

Build a Gymnasium-compatible environment from scratch โ€” observation, action, state, transition.

CPU onlyOpen in ColabOpen in Colab: lab00_tiny_world

Lab 1 ยท Kalman Filter

Hand-write the predictโ€“update loop, track a noisy 2D target, then compare against dynamax.

CPU onlyOpen in ColabOpen in Colab: lab01_kalman_filter

Lab 2 ยท Latent Dynamics

Learn a latent state from pixels and train a predictor โ€” the minimal world-model loop.

Colab T4Open in ColabOpen in Colab: lab02_latent_dynamics
Browse all labs โ†’