🛡️ WorldGuard
JEPA-Inspired Video World Model for Unsupervised CCTV Anomaly Detection
Inspired by Yann LeCun's AMI Labs world model thesis — implemented locally on a single RTX A4000
📊 Results
Benchmark Performance (Frame-level AUROC)
| Dataset | Run 1 (ShanghaiTech only) | Run 2 (Balanced) | Change |
|---|---|---|---|
| UCSD Ped2 | 0.545 | 0.788 | +24% |
| ShanghaiTech | 0.639 | 0.614 | -2.5% |
Run 1 trained on ShanghaiTech data only. Run 2 retrained on balanced UCSD Ped2 + ShanghaiTech clips — UCSD Ped2 jumped dramatically once the model saw that scene during training. ShanghaiTech dropped slightly, a normal trade-off when splitting training capacity between two datasets.
Context vs Published Work
| Method | UCSD Ped2 AUROC |
|---|---|
| Supervised SOTA | ~0.96–0.99 |
| Unsupervised / reconstruction-based | ~0.82–0.92 |
| WorldGuard (this model, 50 epochs) | 0.788 |
WorldGuard achieves competitive unsupervised performance with zero anomaly labels during training, on a single consumer GPU in 50 epochs.
To Push Further
- Train 100+ epochs — loss was still decreasing at epoch 50
- Add more ShanghaiTech training data (currently 2077 clips; full set = 330 videos)
🧠 Why This Exists — The World Model Thesis
In March 2026, Yann LeCun's AMI Labs raised $1.03 billion to build AI that goes beyond LLMs. The core thesis: real intelligence predicts abstract representations of future states, not pixels or tokens.
WorldGuard is a direct local implementation of that thesis applied to CCTV surveillance:
Train a model to predict what should happen next. When reality deviates from prediction — that's an anomaly.
Why Not Just Train a Classifier?
| Supervised Classifier | WorldGuard (World Model) | |
|---|---|---|
| Labels needed | Hundreds per class | Zero |
| Detects novel anomalies | No — known classes only | Yes — any deviation |
| Generalizes to new cameras | Poor | Better — learns scene structure |
| Failure mode | Unknown unknowns invisible | All deviations flagged |
⚙️ How It Works
CCTV Video Stream
│
▼
┌─────────────────────┐
│ Frame Extractor │ 16 frames @ 224×224, stride 2 (~1s @ 30fps)
└─────────────────────┘
│
▼
┌──────────────────────────────────────────┐
│ JEPA WORLD MODEL │
│ │
│ Context Encoder (ViT-S/16) │
│ → z_ctx [B, T_ctx, D] │
│ │
│ Predictor (Transformer, 4 layers) │
│ → z_pred [B, T_future, D] │
│ │
│ Target Encoder (EMA — no gradients) │
│ → z_tgt [B, T_future, D] │
│ │
│ Loss = L2(z_pred, z_tgt) │
│ ↑ latent space only — never pixels │
└──────────────────────────────────────────┘
│
▼
┌─────────────────────┐
│ Anomaly Scorer │ mean prediction error per clip
│ + Spatial Heatmap │ per-patch error → 224×224 overlay
└─────────────────────┘
The model never trains on anomalies. It learns the statistical structure of normality. At inference, anything that breaks that structure produces a spike in latent prediction error.
🏗️ Architecture
| Component | Design | VRAM |
|---|---|---|
| Context Encoder | VideoMAE-pretrained ViT-S/16 (21M params) | ~1.8 GB |
| Target Encoder | EMA copy — frozen, no gradients | ~1.8 GB |
| Predictor | 4-layer Transformer (D=384, heads=6) | ~0.4 GB |
| Activations | Batch=16, 16 frames @ 224×224 | ~8–10 GB |
| Total | ~13–14 GB |
🚀 Quickstart
1. Install dependencies
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
2. Download checkpoint
from huggingface_hub import hf_hub_download
path = hf_hub_download(
repo_id="irfanalii/worldguard",
filename="checkpoints/train_default_epoch050_val0.0191.pt"
)
3. Score a video
python inference/score_video.py \
--video /path/to/footage.mp4 \
--checkpoint checkpoints/train_default_epoch050_val0.0191.pt \
--camera-id cam01
4. Evaluate on UCSD Ped2
python eval/eval_roc.py \
--checkpoint checkpoints/train_default_epoch050_val0.0191.pt \
--test-dir data/ucsd_ped2/ \
--output outputs/eval/
5. Retrain on your own footage
# Extract clips from your normal CCTV recordings
python data/extract_clips.py \
--video /path/to/cam01.mp4 \
--output-dir data/train \
--camera-id cam01
# Train
python training/train.py --config configs/train_default.yaml
# Calibrate threshold
python training/calibrate.py \
--checkpoint checkpoints/best.pt \
--val-dir data/val \
--camera-id cam01
📚 References
- Assran et al. (2025). V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Bardes et al. (2024). V-JEPA: Revisiting Feature Prediction for Learning Visual Representations from Video
- LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence
📄 License
MIT
Built by @Irfanalee · Source Code · Inspired by LeCun's world model thesis · Runs entirely on local hardware