Video Generation & Reasoning
Every week we pick one paper and go deep — video generation, world models, physical reasoning, diffusion, flow matching, and everything in between.
Every Friday 7:30 PM PT on Zoom · All sessions & registration on luma.com/video-model
Jump to next event ↓Special Event
Frontier Research in World Model
A day at SF Tech Week on the frontier of world models — the video, 3D and simulation models learning to predict, generate and reason about the physical world.
Sessions
Embodied Reasoning with World Models
I'll present a couple methods showing how we can effectively reason with video models. I'll first talk about how we can directly reason with video models. I'll then talk about how video models can be combined with VLMs for hybrid reasoning and with policies for low level reasoning.
Can Models Think Without Language? Video as the Next Substrate of Intelligence
Reasoning is often viewed as inseparable from language, driven by the remarkable success of large language models. But is language the only medium through which intelligence can emerge? In this talk, I will present A Very Big Video Reasoning Suite (VBVR), an inaugural effort toward a new paradigm of model reasoning that uses video as the substrate of thought. Unlike language, video naturally captures rich spatial and temporal structure, opening new possibilities for how models perceive, reason about, and make predictions. I will also share recent findings on how reasoning emerges through video generation, and how these developments may reshape the future landscape of large multimodal models.
Think Visually, Reason Textually: Vision-Language Synergy in ARC
Abstract reasoning from minimal examples remains a core unsolved problem for frontier foundation models. The Abstraction and Reasoning Corpus (ARC-AGI) provides a rigorous testbed for this capability, demanding conceptual rule induction and transfer to novel tasks. We introduce two synergistic strategies: Vision-Language Synergy Reasoning (VLSR), which decomposes ARC-AGI into modality-aligned subtasks, and Modality-Switch Self-Correction (MSSC), which leverages vision to verify text-based reasoning for intrinsic error correction. Our approach yields up to 4.33% improvement over text-only baselines across diverse flagship models.
Video Models Can Reason with Verifiable Rewards
Video diffusion models are emerging as general-purpose visual foundation models. Yet, the power of such generalization can be hindered by a reasoning ability that lags behind large language models. The main gap is the mismatch between perceptual plausibility and objective correctness. SFT on ground-truth solution videos can teach the model the visual form of valid trajectories, yet it does not directly optimize the correctness of sampled outputs. As a result, models may imitate solution-like patterns while failing to satisfy the underlying rules that make those solutions valid. In this talk, I will share our recent work VideoRLVR, a framework that trains video diffusion models with reinforcement learning from verifiable rewards. I will also discuss lessons from our failed attempts and how they lead to a practical recipe for moving video models from visual imitation toward verifiable reasoning.
Video Models Are Zero-Shot Learners and Reasoners
The remarkable zero-shot capabilities of Large Language Models (LLMs) have propelled natural language processing from task-specific models to unified, generalist foundation models. Could video models be on a trajectory towards general-purpose vision understanding, much like LLMs developed general-purpose language understanding? We demonstrated that Veo 3 can zero-shot solve a broad variety of tasks it wasn't explicitly trained for: segmenting objects, detecting edges, editing images, understanding physical properties, recognizing object affordances, simulating tool use, completing symmetric patterns, solving mazes, and much more. These emergent abilities indicate that video models are on a path to becoming unified, generalist vision foundation models.
VChain: Chain-of-Visual-Thought for Reasoning in Video Generation
Recent video generation models can produce smooth and visually appealing clips, but they often struggle to synthesize complex dynamics with a coherent chain of consequences. Accurately modeling visual outcomes and state transitions over time remains a core challenge. In contrast, large language and multimodal models exhibit strong visual state reasoning and future prediction capabilities. To bridge these strengths, we introduce VChain, a novel inference-time chain-of-visual-thought framework that injects visual reasoning signals from multimodal models into video generation. Specifically, VChain contains a dedicated pipeline that leverages large multimodal models to generate a sparse set of critical keyframes as snapshots, which are then used to guide the sparse inference-time visual-state adaptation of a pre-trained video generator only at these key moments. Our approach is tuning-efficient, introduces minimal overhead and avoids dense supervision. Extensive experiments on complex, multi-step scenarios show that VChain significantly enhances the quality of generated videos.
Do Joint Audio-Video Generation Models Understand Physics?
Recent joint audio-video generation models such as Seedance 2.0, Kling 3.0 Omni, and Veo 3.1 can produce clips that look and sound very realistic. In the real world, however, vision and sound are two observations of the same physical event. Turning up a volume knob makes the music louder, and a ringing alarm clock sealed in a foam-lined box sounds muffled. Do current models understand the physics that connects what we see with what we hear, or do they simply put together plausible frames and sounds? This talk introduces AV-Phys Bench, the first comprehensive benchmark for physical commonsense in joint audio-video generation. The benchmark contains prompts organized by how a scene evolves — Steady State, Event Transition, and Environment Transition — together with an Anti-AV-Physics set where prompts deliberately ask for physical violations. The prompts instantiate 41 audio-visual physics principles, and each prompt comes with its own rubric that scores semantic adherence and physical commonsense within and across modalities. Evaluating 3 proprietary and 4 open-source models with about 58,000 human judgments reveals a consistent gap between semantics and physics, much weaker performance on transition scenes, and a sharp drop of 45% to 69% on anti-physics prompts. The talk also presents AV-Phys Agent, a ReAct-style evaluator that grounds a multimodal LLM with deterministic audio DSP measurements such as loudness, pitch, reverberation, and onset timing; it aligns with human ratings more closely than MLLM-as-judge baselines and enables scalable physics-aware evaluation, closing with thoughts on using such evaluators as verifiable reward signals for post-training.
Demystifying Video Reasoning
Recent advances in video generation have revealed an unexpected phenomenon: diffusion-based video models exhibit non-trivial reasoning capabilities. Prior work attributes this to a Chain-of-Frames (CoF) mechanism, where reasoning is assumed to unfold sequentially across video frames. In this work, we challenge this assumption and uncover a fundamentally different mechanism. We show that reasoning in video models instead primarily emerges along the diffusion denoising steps. Through qualitative analysis and targeted probing experiments, we find that models explore multiple candidate solutions in early denoising steps and progressively converge to a final answer, a process we term Chain-of-Steps (CoS). Beyond this core mechanism, we identify several emergent reasoning behaviors critical to model performance: (1) working memory, enabling persistent reference; (2) self-correction and enhancement, allowing recovery from incorrect intermediate solutions; and (3) perception before action, where early steps establish semantic grounding and later steps perform structured manipulation.
Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm
The "Thinking with Text" and "Thinking with Images" paradigms significantly improve the reasoning abilities of LLMs and VLMs — but images capture only single moments, and the separation of text and vision hinders unified multimodal understanding. "Thinking with Video" leverages video generation models such as Sora-2 to use video frames as a unified medium for multimodal reasoning. The accompanying Video Thinking Benchmark (VideoThinkBench) covers vision-centric tasks (e.g., eyeballing puzzles) and text-centric tasks (e.g., GSM8K, MMMU). Sora-2 emerges as a capable reasoner: comparable to SOTA VLMs on vision-centric tasks — surpassing GPT-5 by 10% on eyeballing puzzles — with 92% accuracy on MATH and 69.2% on MMMU.
Seeing What Matters, Learning How to Find Out: Toward Visual General Intelligence
Which visual differences should change a model's answer, and which should not? This talk discusses recent work on learning relations between 3D objects and on color-dependent judgments in vision-language models. These studies examine the importance of fine-grained geometry and the effects of changing how the same text is presented. Using these concrete examples, the talk connects the task-dependent use of visual information to questions in Visual General Intelligence: A White Paper about acquiring knowledge from visual experience and applying it to new tasks, and closes with a research perspective on a further challenge: when current observations are insufficient, can a system learn how to obtain useful evidence through observation and interaction, and can those ways of finding out transfer to unfamiliar situations? ~40-minute talk followed by open Q&A.
Video Generation Models are General-Purpose Vision Learners
Driven by next-token prediction, NLP shifted from task-specific models into powerful generalist foundation models. What, then, is the equivalent catalyst needed to achieve a general-purpose model in computer vision? This talk contends that large-scale text-to-video generation serves as a pivotal pre-training paradigm for computer vision, providing the necessary spatiotemporal priors, vision-language alignment, and scalability required for general visual intelligence. GenCeption leverages a pre-trained video generative diffusion backbone to define a feed-forward perception model, capable of performing various vision tasks steered by text instructions — achieving state-of-the-art performance across depth, surface normal, and camera pose estimation, expression-referring segmentation, and 3D keypoint prediction, often matching or surpassing specialized models (e.g., DepthAnything3, SAM3, D4RT, VGGT-Omega, Sapiens, David, Genmo, and Lotus-2). The video generative pretrained backbone outperforms alternative pretraining paradigms, exhibits preliminary data and model scaling properties along with exceptional data efficiency, and triggers intriguing emergent behaviors — suggesting that video generation is not merely a synthesis tool, but a foundational path toward generalist vision intelligence for the physical world.
Title TBA
TBD