• Date: 2026-06-30
• Time: 09:00AM (PT)
• Location: Teams Meeting

Agenda

Presentation from Ashena Gorgan Mohammadi and Manu Srinath Halvagal of a recent study from Friedemann Zenke group.

Understanding neural circuit principles for representation learning through joint-embedding predictive architectures

Meeting Recording

Meeting Notes

Introduction of Recurrent Predictive Learning (RPL) Model: Ashena and Manu from the Friedman Zenke Group presented their work on the Recurrent Predictive Learning (RPL) model, explaining its architecture, motivation, and how it addresses limitations in classical predictive coding frameworks, with Jerome, Sarah, and other participants engaging in clarifying questions.

Motivation and Background: Ashena described the challenge of how the brain learns latent variables from sensory experience in a self-supervised manner, highlighting the limitations of classical predictive coding frameworks, such as the need for back-propagation to the sensory periphery and issues with noise sensitivity and representational die-out.

RPL Model Architecture: Ashena detailed the RPL model, which uses a feedforward encoder, a recurrent integrator, and a feedforward predictor to learn latent representations by predicting future embeddings rather than raw sensory input, optimizing only through the prediction stream.

Comparison to Joint Embedding Predictive Architectures: Ashena explained that, unlike typical JEPA models that use transformer backbones and random masking, their approach uses simpler feedforward and recurrent networks, making it more biologically plausible and robust to noise.

Clarification on Model Inputs and Loss: In response to Jerome and Sarah's questions, Ashena clarified that the model can handle various input modalities and that the loss function, along with weight decay, implicitly encourages separation of latent variables without explicit constraints.

Evaluation of RPL on Synthetic and Natural Data: Ashena and Manu described experiments where the RPL model was trained on synthetic animal movement videos and natural datasets, demonstrating its ability to extract and represent latent variables, with Sarah and Manu discussing the implications of constant versus dynamic variables.

Synthetic Dataset Design: Ashena described the creation of a synthetic dataset with moving animals, where variables such as velocity and orientation are randomly sampled and constant within each video, allowing controlled evaluation of the model's ability to learn latent variables.

Latent Variable Decoding: After training, the RPL model's representations were evaluated by training linear readouts to decode object identity, position, velocity, orientation, and rotation speed, showing performance close to supervised baselines.

Discussion on Variable Constancy: Sarah questioned the impact of constant versus varying parameters, to which Ashena and Manu responded that while velocity is constant within a video, it changes upon collisions, and the model can handle both invariant and dynamic variables.

Simulation of Future Trajectories: Ashena demonstrated that the RPL model can simulate future trajectories by autoregressively feeding its own outputs, maintaining accurate latent variable decoding for several steps before errors accumulate.

Comparison with Alternative Learning Architectures: Manu compared the RPL model to alternative architectures such as invariance learning (IL) and context predictive learning (CL), showing that RPL uniquely supports robust latent variable encoding across synthetic and natural datasets, while IL and CL have specific failure modes.

Invariance Learning (IL) Limitations: Manu explained that IL, which predicts at the same hierarchical level, learns invariant features like object identity but fails to encode dynamic variables such as position and velocity, as shown in both synthetic animal videos and natural speech datasets.

Context Predictive Learning (CL) Failure Modes: CL, which uses recurrent encoders for both prediction and target, is prone to generating self-predictable signals and degrades significantly with increased input noise or occlusion, unlike RPL which remains robust.

Empirical Results Across Datasets: Manu presented results showing that RPL outperforms IL and CL in encoding both invariant and dynamic latent variables in datasets including animal movement, naturalistic mouse videos, and speech (Libri Speech).

Qualitative Comparison with Experimental Neural Data: Manu and Ashena discussed how the RPL model qualitatively reproduces experimental findings from human and macaque studies, including successor representations in V1 and sequence encoding in prefrontal cortex, with Jerome, Sarah, and Tim seeking clarification on the comparison methodology.

Successor Representation in Human V1: Manu described modeling a paradigm from Loris Lange's lab, where the RPL model, when habituated to sequences, showed activation patterns for successor stimuli similar to those observed in fMRI recordings.

Comparison Methodology: In response to Jerome and Sarah, Ashena clarified that the comparison is qualitative, focusing on functional similarity in activation patterns rather than direct representational similarity analysis, due to lack of access to raw neural data.

Sequence Encoding in Macaque PFC: Manu explained that the RPL model, when exposed to local-global deviant sequences, could encode temporal position and stimulus identity, matching findings from neuropixel recordings in macaque prefrontal cortex.

Hierarchical RPL (HRPL) and Outlook for Biological Mapping: Ashena introduced the hierarchical extension of RPL (HRPL), discussing its potential for mapping model components to cortical layers and circuits, and outlined future directions including feedback connectivity, multimodal integration, and collaboration opportunities with Jerome's group.

HRPL Model Structure: Ashena described HRPL as a model with shallow encoders and local predictive circuits at each hierarchical level, allowing for representational similarity analysis between model components and different cortical areas.

Feedback Connectivity and Modulation: Ashena discussed plans to investigate how feedback connections between integrators can modulate representations, drawing parallels to dual counter-stream architectures in cortical hierarchies.

Open Questions and Future Work: Ashena outlined open research questions, including the emergence of structured latent representations, the role of inductive biases, mapping computational components to cell types, and extending the model to multimodal and sensory-motor integration settings.

Collaboration and Data Sharing: Jerome offered access to extensive neural recording datasets across cortical layers and modalities, proposing collaborative experiments to compare model predictions with neural data, and Ashena and Manu expressed interest in pursuing these analyses.

Technical Details and Training Methods: Lucas inquired about the training of the recurrent network, with Ashena and Manu confirming the use of backpropagation through time and discussing possible approximations for local learning in hierarchical models.

Training Algorithms: Ashena confirmed that the RPL and HRPL models are currently trained using backpropagation through time, with local learning approximations considered for short sequence lengths in hierarchical settings.