What is VideoAgent?
VideoAgent is a research framework from the University of Hong Kong's Data Intelligence Lab (HKUDS, also behind LightRAG and VideoRAG), accompanying an EMNLP 2026 paper. You describe the video you want in plain language; the system breaks the request into explicit and implied sub-goals, maps them to specialised agents and builds a graph-shaped workflow that it checks and revises in a feedback loop. Jobs include video question answering and summaries, beat-synced movie edits, commentary videos, meme remakes, music videos and cross-language comedy adaptations.
Under the hood it strings together open models for speech and audio (CosyVoice, Fish Speech 1.5, Seed-VC, DiffSinger, Whisper large-v3-turbo) and ImageBind for retrieval, with hosted LLMs doing the planning. The code is MIT-licensed Python; there are no tagged releases, and most of the code landed in July 2025, with a handful of fixes in June and July 2026.
Is VideoAgent worth it?
VideoAgent is worth reading more than deploying. Its contribution is the planning layer: a graph router that turns a vague brief ('turn this stand-up script into a special with matching footage') into a chain of agents for transcription, retrieval, voice cloning, singing synthesis and editing, then scores and repairs that chain over several reflection rounds. The authors report a workflow-composition success rate of 0.95 across their configurations and better results with Claude 3.7 than with GPT-4o or DeepSeek-V3 as the planner; those are the paper's own numbers on its own datasets.
Running it is a research-lab exercise. The README asks for an 8 GB GPU, Conda with Python 3.10, pynini and ffmpeg, then six separate model downloads. The configuration needs Claude for the router, which is mandatory, plus DeepSeek, OpenAI and Gemini keys for remixing, editing and fine-grained captioning respectively. Open issues include agents that fail with 'There is no suitable agent', graph-judgement failures and dependency conflicts across install paths, and a July 2026 pull request fixed orchestrator crashes and an unsafe eval() in the multi-agent code.
The licences need checking before any commercial use. The code is MIT, but Fish Speech 1.5 and ImageBind weights are both CC BY-NC-SA 4.0, so the default voice-cloning and retrieval paths are non-commercial. The demo videos remix films and broadcasts sourced from the internet, and the README says they are for research only; the tool itself does nothing about rights in the footage you feed it.
Activity has thinned: 236 commits in July 2025, single digits a month after that, and six commits in 2026. With 1,973 stars it is a well-watched reference design for agentic video editing rather than a maintained product.
Where it earns its place
A machine-learning researcher working on agent planning
Reproduce and extend a graph-based multi-agent planner for media tasks
Run the framework with Claude as the router, swap in your own agents, and compare against the paper's reflection-round results. Expect to assemble your own evaluation set; a public release of the VideoEdit benchmark has been requested in the issues.
An engineer at a short-video startup scoping an AI editing feature
See how intent analysis and tool routing could work before building in-house
Read the intent-to-agent mapping and storyboard retrieval code, and test it on owned footage. Shipping it as is would mean replacing the non-commercial Fish Speech and ImageBind components and paying four LLM providers per job.
A university media lab teaching AI video production
Demonstrate automated commentary and summary videos on lecture recordings
Use the overview and Q&A paths with Whisper transcription and the GPT and Gemini keys, on an 8 GB GPU. Non-commercial teaching sits within the bundled model licences.
Before you commit
- Setup
- Clone, create a Conda env with Python 3.10, install pynini and ffmpeg, pip install the requirements, then download CosyVoice, Fish Speech, Seed-VC, DiffSinger, Whisper and ImageBind checkpoints as needed. Run python main.py and type a requirement at the prompt.
- Integrations
- Claude (required for the agentic graph router), DeepSeek, OpenAI and Gemini APIs via config.yml; an opt-in TwelveLabs Pegasus agent has been proposed by a contributor.
- Data and privacy
- Video frames, transcripts and prompts are sent to the hosted LLM providers you configure. Code is MIT; Fish Speech 1.5 and ImageBind weights are CC BY-NC-SA 4.0, which rules out commercial use of those paths.
- Cost at scale
- Local models run on an 8 GB GPU; each job also bills four LLM providers, and reflection rounds multiply planner calls. No hosted or paid edition exists.
- Limits
- No releases or versioning, sparse maintenance since 2025, reported routing failures ('There is no suitable agent') and dependency conflicts. The interface is a console prompt, not an editor.
What's good
- Intent decomposition and self-correcting workflow graph, documented in an EMNLP paper
- Covers understanding, editing and generative remakes in one framework
- Runs on an 8 GB GPU for the local models
- MIT-licensed code
What's not
- Needs Claude, DeepSeek, OpenAI and Gemini API keys
- Fish Speech 1.5 and ImageBind weights are non-commercial
- No releases; six commits in 2026
- Open issues report agent-routing failures and dependency conflicts
Best for
Researchers and engineers studying how to plan multi-agent video pipelines, rather than teams that need a dependable editing tool.
VideoAgent alternatives
VideoAgent: common questions
- Is VideoAgent worth it?
- VideoAgent is an impressive paper demo and an awkward thing to run, held together by four API keys and two non-commercial model licences. We rate it 3.2 out of 5.
- How much does VideoAgent cost?
- Free under MIT. Running it needs paid API keys for Claude, plus DeepSeek, OpenAI and Gemini for different features; some bundled models are non-commercial.
- What are the downsides of VideoAgent?
- Needs Claude, DeepSeek, OpenAI and Gemini API keys; fish Speech 1.5 and ImageBind weights are non-commercial; no releases; six commits in 2026; open issues report agent-routing failures and dependency conflicts.
- Who is VideoAgent best for?
- Researchers and engineers studying how to plan multi-agent video pipelines, rather than teams that need a dependable editing tool.
- How much does VideoAgent cost for a team or in production?
- Local models run on an 8 GB GPU; each job also bills four LLM providers, and reflection rounds multiply planner calls. No hosted or paid edition exists.
- Is VideoAgent safe for company data?
- Video frames, transcripts and prompts are sent to the hosted LLM providers you configure. Code is MIT; Fish Speech 1.5 and ImageBind weights are CC BY-NC-SA 4.0, which rules out commercial use of those paths.
- What are the best VideoAgent alternatives?
- The closest alternatives we have reviewed are ComfyUI and LightRAG.
Answers checked Oct 26, 2026.