CV Lab 21 · Module 9 — Motion & Video

Optical Flow

Two consecutive frames, one question at every pixel: which way did the brightness under me move? Lucas-Kanade answers that question for a handful of chosen points. The same equation, run on every pixel at once, paints the whole frame with motion.

Brightness constancy Assume a point's brightness stays the same between two frames, just moved. That single assumption, linearised, is the entire mathematics of optical flow — sparse or dense, it is the same equation underneath.
Tracked point / normal velocity Sudden acceleration flash Segmented moving object
Ready.
Live Explanation
Loading…

Click the moving scene to drop a tracking point (up to 8). Watch its neon tail — a sudden colour flip to red means the point's velocity changed sharply between frames.

Window size
15×15

One equation, two use cases

The Lucas-Kanade equation

Ix·u + Iy·v = -It, solved over a small window using the same 2×2 structure tensor from Harris corners — which is exactly why flow only tracks reliably at corners, not on flat or edge-only patches.

Why pyramids matter

The linear approximation only holds for small motions. A coarse, shrunken image turns a big real displacement into a small pixel displacement the equation can handle — solve there first, then refine at full resolution.

Sparse vs dense trade-off

Sparse tracks a handful of well-chosen points cheaply and precisely. Dense answers "what is moving, everywhere" at once — far more information, far more computation, which is why real-time dense flow historically needed dedicated hardware.

Predict, then verify

You drop a Lucas-Kanade tracking point in the middle of a large, perfectly flat, textureless wall in the scene. What happens to its tracking?