Explainer: the 'predictive closure' view in JEPA research

JEPA learns what to keep from what it sees, and uses that to predict what comes next. But what about the details it can’t predict? Wouldn’t it be easier for the model to drop them? Look at this example Imagine a traffic light that changes color randomly. Its next color is unpredictable, but its current color helps predict whether a cyclist will stop or move. If we drop the light, the cyclist becomes harder to predict. ▪️ A recent research call this set-level view "predictive closure:" A feature’s value depends on its relationships with the other features the model represents. Keeping the light makes the cyclist easier to predict, but adds an unpredictable color to the model’s targets. Dropping it removes that target, but loses a useful clue. The researchers separate these two costs: 1. Aleatoric uncertainty: what remains uncertain even with the full current observation, like the light’s next random color. 2. Predictive leakage: useful information available in the observation bu…