The Annotated JEPA

By Elon Litman | January 27, 2026 | Filed under: Deep Learning

This guide serves as a comprehensive, from-scratch, annotated walkthrough of Joint Embedding Predictive Architectures (JEPAs).

The objective here is to mirror the impact of The Annotated Transformer by meticulously constructing the architecture, clarifying every internal mechanism, and concluding with a functional training loop.

The Core Philosophy

JEPA represents Yann LeCun's strategic response to a pivotal challenge in self-supervised learning (SSL): How can a model acquire a deep understanding of the world without human-provided labels, while avoiding the "collapse" into trivial solutions and ignoring useless noise?

The solution is as elegant as it is nuanced: perform prediction within the representation (latent) space rather than the input space.

Why Visuals over Text?

While many models focus on LLMs, JEPA emphasizes images and video. The reasoning is as follows:

• Language is already a discrete, highly compressed abstraction of knowledge. Predicting the next token is more about modeling human communication than physical reality.
• Visuals require a model to grasp concepts like persistence, occlusion, and physical dynamics.

Pixel-level reconstruction is viewed as a waste of computational capacity because it forces the model to obsess over irrelevant high-frequency details.

---

Roadmap of the Implementation

• Build the foundational I-JEPA architecture.
• Explore extensions to video via V-JEPA and V-JEPA 2.
• Analyze LeJEPA's distributional regularizers.
• Implement the final training loop.

Evolution of the Architecture

Note on Implementation: To keep the mathematics transparent, this walkthrough omits production-level engineering such as FlashAttention, mixed-precision training, gradient checkpointing, and complex batching strategies.

---

The Fundamental Problem

The central struggle of self-supervised representation learning is finding an objective function that captures meaningful structure without labels. JEPA solves this by predicting representations.

The Intuition

Imagine you are presented with a partial image (the context $x$) and must predict a hidden region (the target $y$).

1. A Predictor takes the encoding of the context and guesses the representation $\hat{s}_y$.
2. The system only succeeds if the context encoding $s_x$ contains enough semantic information to constrain the possibilities of $s_y$.

Examples of Semantic Necessity:

• If you see a car's hood, you can only predict the wheels if your encoder understands the concept: "This is a car."
• If you see a face, predicting the hair requires an understanding of identity, pose, and lighting.

Because the predictor cannot "hallucinate" details missing from the context, the encoder is forced to ignore pixel-level noise and focus on structural features (object identity, spatial constraints, etc.).

Theoretical Framework

LeCun frames this within an Energy-Based Model (EBM) context, viewing JEPA as the bedrock for "World Models" that can plan within a latent space.

---

The JEPA Template

A JEPA operates on paired, semantically linked views of the world. Generally, these are triples $(x, y, z)$:

• $x$: The observed context.
• $y$: The target to be predicted.
• $z$: An optional latent variable used to handle multimodality (when one $x$ could lead to multiple valid $y$ outcomes).

Application Domains

• Images: $x$ = visible patches $\rightarrow$ $y$ = masked patches.
• Video: $x$ = clip prefix $\rightarrow$ $y$ = future continuation.
• Cross-Modal: $x$ = audio $\rightarrow$ $y$ = corresponding video.

The Mathematical Formulation

Both observations are mapped into a shared latent space:

$s_x = f_\theta(x), \quad s_y = f_{\bar\theta}(y)$

Where $f_\theta$ and $f_{\bar\theta}$ are the encoders. These may share parameters or utilize different architectures depending on the specific implementation.

Conceptual Code Structure

def jepa_loss(context, target):
    # Encode both views
    s_x = context_encoder(context)
    s_y = target_encoder(target)
    
    # Predict target representation from context
    s_y_hat = predictor(s_x)
    
    # Calculate distance (Energy) in latent space
    loss = distance_function(s_y_hat, s_y)
    return loss