1 Nanjing University2 KlingAI Research3 Peking University4 University of Hong Kong5 Hong Kong University of Science and Technology (Guangzhou)6 Hong Kong University of Science and Technology7 Tsinghua University
When to model, what to model, and what to leave to the video generator.
Natural hand motionFollowing a physical processState across camera revisits
Videos generated by Coda-DaVinci
Code as a guide for video generation
Coding agents can build executable 3D scenes to guide video generation. This gives us control over scene structure and physical events, but modeling every detail can also introduce stiff motion. The key is deciding how to combine the strengths of coding agents and video models.
We study modeling granularity, physics simulation, and synthesis prompts through controlled experiments. These findings lead to Coda-DaVinci: a task-adaptive framework that selects when to use 3D modeling, assigns each entity an appropriate level of detail, and binds the resulting proxy to a synthesis prompt. A persistent scene workspace and visual feedback support revision and long-video continuation.
Preserve the event; leave room for natural motion. Scene programs carry state across a camera revisit. Selective modeling gives the video generator freedom to synthesize fine hand movements.
A task-adaptive Code2Video framework
Coda-DaVinci uses explicit 3D scenes when the task needs precise control. It favors direct generation for natural body movement when a proxy offers little benefit. For modeled scenes, environments and objects receive structured geometry, while human proxies stay coarse. The coding agent inspects the proxy and writes a prompt that binds its subjects to their intended appearance and actions.
The harness coordinates construction, rendering, and video synthesis. Visual inspection and independent review guide repairs, while editable scene assets remain available for continuation.
Route by task
Use a scene program where explicit control helps; choose direct synthesis when natural motion is the priority.
Model selectively
Keep structured environments and objects, while leaving fine human articulation to the video model.
Ground the prompt
Link subject descriptions to proxy entities and appearance references before synthesis.
See the difference
Compare the generated videos, then inspect the proxies used by the two Code2Video methods.
0:00 / 0:00
All three videos play from the same timestamp.
Read the original instruction
Main results
All methods use Seedance 2.5 for video synthesis. We evaluate physical behavior, scene consistency, instruction following, and visual quality.
Codex-based methods, VideoCoCo, CodeVideoModel, and Coda-DaVinci use GPT-6-Astra. Claude-based methods use Claude Opus 5.5. Scores are averaged over valid evaluations. R2M reports its relative consistency metrics; T2AV-Compass uses the selected 200-task subset.
What makes Code2Video work?
Our controlled study finds that coarse proxies preserve more natural motion on motion-intensive tasks, while medium-grained geometry supports scene consistency. Physics simulation and proxy-grounded prompts further improve average quality.
Component ablations, rendering-engine comparisons, and long-video consistency studies.
Blender achieves higher scores than Three.js on the diagnostic subset, at roughly twice the agent token cost. Long-video experiments show stronger event structure and scene consistency, with remaining challenges in transition stability and adherence to scene settings.
Citation
@misc{ji2026codadavinci,
title = {Coda-DaVinci: Reimagining Video Generation with Code2Video},
author = {Yiyan Ji and Qixun Wang and Yang Shi and Yifan Xu and Zhuoran Zhang and Tengfei Liu and Xiaohan Zhang and Shujie Li and Yuqi Tang and Jiafu Tang and Xuanyu Zhu and Xinyu Liu and Yan He and Yushuo Guan and Zihao Wang and Xianghua Ying and Haotian Wang and Zhaoxiang Zhang and Jiaheng Liu and Yuanxing Zhang and Pengfei Wan},
year = {2026}
}