The rapid evolution of Vision Large Language Models (VLLMs) has unlocked new capabilities in video understanding. However, processing long-form video content remains computationally prohibitive. Standard architectures rely on feeding thousands of visual tokens into the LLM's context window, leading to unsustainable KV-cache bloat, memory exhaustion, and temporal amnesia over long horizons.
The VideoCartridge project introduces a paradigm shift: instead of passing raw visual tokens to the model at inference time, we compress the entire spatiotemporal context of a video into a highly dense, fixed-length set of vectors. The goal is to decouple video processing from text generation, creating a plug-and-play, reusable memory module that acts as a strict information bottleneck, retaining only the semantic and structural essence of the video.
The starting point: Self-Study approach
The conceptual foundation of this project is rooted in the original textual Cartridge architecture, which was designed to mitigate the context window limitations of Large Language Models when processing extensive documents.
In its original form, the framework compresses lengthy textual contexts into a compact, reusable set of learned embeddings (soft prompts). By decoupling context ingestion from the generative inference phase, the textual Cartridge acts as a fixed-length memory module. Through the Self-Study approach, the model is trained to answer queries or reconstruct information by relying solely on this bottleneck representation. This drastically reduces inference costs and KV-cache overhead while preserving the logical structure and semantic core of the original text.
The multimodal evolution
Building upon the success of textual compression, the VideoCartridge framework represents a necessary evolution into the multimodal domain, addressing the extreme computational complexity of sequential visual tokens. To achieve this spatiotemporal compression, we modified and adapted the original Self-Study Context Distillation architecture for video data.
Even in this case, the training pipeline operates without relying on massive human-annotated datasets, utilizing instead an automated Teacher–Student dynamic. During data generation, a high-capacity Teacher LLM processes raw video chunks to generate rich, descriptive question-and-answer pairs about the scene's content.
In the training phase, the Student model is tasked with reconstructing the Teacher's insights without access to the raw visual frames, relying solely on the VideoCartridge through a specific prefix-tuning strategy.
The VideoCartridge serves as the sole bridge of visual information. By optimizing the model to minimize the generative loss against the Teacher's ground truth, the Cartridge is forced to encode complex visual features (e.g., object permanence, kinematics, spatial topology) into a strictly constrained token space.
The road ahead
So far, the results are promising: with a very small number of tokens, we have managed to achieve the same performance as a full-length video, and have even surpassed it in some cases.
The next phase of research will address the limits of current dataset creation and semantic bleeding. Future development will focus on two main pillars:
-
Hierarchical and Multiple VideoCartridges: Moving beyond static compression into a single VideoCartridge for the entire video, we aim to implement dynamic, variable-length segmentation mapped across multiple Cartridges. These will be structured hierarchically, enabling the large language model (LLM) to access micro-level action details and macro-level video states without overloading the context window.
-
Video-Native Seed Prompts: To optimize the information bottleneck, we are reorganizing the data-generation pipeline. The new approach will use highly domain-specific initial prompts, moving away from prompts based entirely on text and going beyond simply adapting them to the visual domain. We are currently analyzing which prompts are most effective and significant for final performance.
References
- Eyuboglu, S., Ehrlich, R., and Arora, S. (2025). Cartridges: Lightweight and general-purpose long context representations via self-study. Paper
- Snell, C., Klein, D., and Zhong, R. (2022). Learning by Distilling Context. Paper
- Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. Paper
- Pope, R., Douglas, S., Chowdhery, A., et al. (2022). Efficiently Scaling Transformer Inference. Paper
- Li, X. L., and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Paper