Zero2FSD
Repo← Portfolio

Zero2FSD

Zero2FSD is a free, self-study course from zero to research level in autonomous driving. From Week 1 you drive a complete but bad car in a simulator, and every week you replace one of its boxes with code you wrote. The course moves from the whole game to the classical car, the learned car and research. It runs on Colab, and your progress stays in this browser. Labs and source are in the course repo.

The course

Build your own self-driving car, one box at a time

Every week is a short lecture, a Colab lab where you write one box of the car, a quiz, and a teach-back.

Start here: Week 1 · Meet your car
CameraMovePerceptionSteerInstruments
Filled: every exercise for that box is passed. Half: some are. Outline: none yet.

Part 1 · The whole game

Play a junior version of the whole game. You get a complete car that drives badly, then open its boxes one at a time and swap in your own.

Part 2 · Classical car · 9 modules · ~48 weeksEvery classical box of the car becomes yours - transforms, control, camera geometry, lane perception, state estimation, tracking and planning.
Part 3 · Learned car · 11 modules · ~52 weeksLearned boxes must beat your own classical boxes.

  • Autograd from scratch — 5 weeks

  • ML fundamentals — 5 weeks

  • Neural nets and CNNs in PyTorch — 5 weeks

  • Learned lane perception — 5 weeks

  • Object detection on real frames — 6 weeks

  • Multi-task nets — 3 weeks

  • Attention and transformers from scratch — 5 weeks

  • BEV perception and point clouds — 6 weeks

  • Occupancy and temporal fusion — 4 weeks

  • Motion forecasting — 4 weeks

  • Imitation-learned planner — 4 weeks

Part 4 · Research car · 8 modules · ~40 weeksRead, reproduce, extend and write up research. The exact papers are chosen when this part is written.

  • Reading and reproducing papers — 3 weeks

  • RL for driving in the course sim — 6 weeks

  • End-to-end driving — 5 weeks

  • World models and generative simulation — 5 weeks

  • Vision-language-action models — 4 weeks

  • Data engines and the long tail — 4 weeks

  • Safety, verification and scenario-based evaluation — 5 weeks

  • Thesis — 8 weeks

v1 chapter library (reference)

11 original chapter-by-chapter modules. They stay online and their links keep working.

How to use a module

  1. Read the module page on this site.
  2. Open the notebook in Colab (or run the module folder locally).
  3. Do the exercises and run pytest in the course repo.
Module 00
Hands-onFoundations

Teaching a computer to recognize road scenes

Small CNN on road patches, per-class recall, and an error gallery

Train a tiny classifier on 64×64 driving crops, compare cross-entropy and focal loss on rare classes, and inspect wrong predictions—not just overall accuracy.

Module guideOpen in Colab (run the code)
Module 01
Hands-onFoundations

From camera pixels to a top-down map

Pinhole math, ground homography, BEV warp, and three-camera stitch

Build camera intrinsics and a ground homography, warp images to a metric bird's-eye view, blend three cameras, and see how pitch error grows with distance.

Module guideOpen in Colab (run the code)
Module 02
Reading notesVision-to-BEV

Multi-Task HydraNet Perception

One shared backbone with separate heads for several driving tasks

Run one vision trunk once, then branch to lanes, objects, and related outputs while balancing how much each task contributes to training.

Module guideOpen in Colab (run the code)
Module 03
Reading notesVision-to-BEV

Bird's-Eye View (BEV) Transform

Turning several camera views into one top-down map with a learned depth guess

Estimate depth along each pixel ray, splat features into a metric grid, and refine them with convolutions—Lift-Splat-Shoot style.

Module guideOpen in Colab (run the code)
Module 04
Reading notesVision-to-BEV

3D Occupancy & Temporal Memory

A 3D grid of what space is occupied, updated over time

Discretize space into voxels, predict occupancy and motion, and carry information forward when objects are briefly hidden.

Module guideOpen in Colab (run the code)
Module 05
Reading notesPlanning & Control

Vector Space Tracking & HD Maps

Track movers and represent lanes as curves in a map-style view

Follow dynamic objects across frames with a motion model and data association, and fit lane geometry into a vector scene description.

Module guideOpen in Colab (run the code)
Module 06
Reading notesPlanning & Control

Lattice Trajectory Planning

Sample smooth path options and score them for safety and comfort

Generate candidate trajectories in Frenet coordinates and rank them with costs for collision risk, jerk, and progress toward the goal.

Module guideOpen in Colab (run the code)
Module 07
Reading notesPlanning & Control

Closed-Loop Control & Kinematics

Steer a simple car model to follow a planned path

Use Stanley lateral control on a kinematic bicycle model to turn trajectory waypoints into steering commands in simulation.

Module guideOpen in Colab (run the code)
Module 08
Reading notesCapstone

Capstone: Full Mini-FSD Integration

Wire perception, BEV, planning, and control into one runnable demo

Connect the course modules into an end-to-end loop from multi-camera input through mapping and planning to actuation.

Module guideOpen in Colab (run the code)
Module 09
Research readingCapstone

Cutting-Edge Research & Frontier

Open problems and pointers for deeper study

Survey where vision-first driving research is heading—end-to-end models, occupancy trends, and questions the mini stack leaves open.

Module guideOpen in Colab (run the code)
Appendix 00b
OptionalFoundations

Neural Networks & Autograd from Scratch

Optional: build autograd and a tiny MLP before leaning on PyTorch

Implement scalar autograd and train a small network in pure Python if you want the mechanics under the hood before Module 00’s PyTorch workflow.

Module guideOpen in Colab (run the code)