Problem
I created a music production tool and am working with artists to create marketing videos for it. I get back a bunch of loosely related video files and need to put together a coherent video for marketing and ads.
For this I need to:
Come up with a cohesive story for the video
Edit the video
Make variants so that we can test out different versions for paid advertising
This can be pretty time consuming and it’s hard to keep track of what variants are possible if you get too deep into the footage. It’s better to work at a higher level to define the variants and then get into the details on each video.
So I made an AI agent that does this. I point it at my raw footage archive, give it some loose directions and then it can come up with some video concepts, mock them up, and let me decide which directions I want to pursue and refine into videos.
My Agent
How it works
Simple version: the agent runs on my computer
(Tech nerd version: I run OpenClaw in an Ubuntu VM on my NAS and I have a video editor sub-agent)
It works on files saved on a folder synced to the cloud and this whole folder syncs across to all my other computers
The agent transcribes all the videos in the footage archive whenever new footage is added
I talk to the bot over Telegram and give it specific instructions eg “Hey I just added new footage for artist X, can you please come up with a couple story ideas around marketing videos for the footage we have?”
It will then send me a couple draft videos - these first drafts tend to be quite unpolished and nowhere close to good but I can get a very good idea for what the story is
I will work with it to iterate and polish the versions I like till it gives me something good eg. add in subtitles, replace specific scenes with other scenes, etc
I initially let the agent use FFMpeg for video editing which just outputs a video file but have been experimenting with using hyperframes video editor (link).
This has the very nice benefit of I can pick up editing exactly where the agent left off so it’s like collaborating with the agent on multiple projects in parallel.
It can also generate copy for posting this onto social media which is super useful
Results
Here’s the first cut video:
It’s not perfect but it’s definitely a good enough jumping off point for manual editing after. I could keep talking to it to make it pixel perfect but I feel like it would be quicker to finish the last 20% manually.
My next step is to integrate into Da Vinci Resolve’s latest version (released 2 days ago) which has an MCP so the agent can use Resolve directly. I think Da Vinci is much nicer to use than Hyperframes so quite happy to have that option now!
Other Solutions
Before I made my own agent I decided to have a look at what’s available on the market. There were a bunch of very interesting video editing solutions (Descript is great, tried it for a month) but none of them have this workflow of brainstorming multiple concepts and editing them into multiple deliverables - they focus more on either generating AI videos or on using text as a way to drive the video editor. Premier Pro had an MCP that could be used with Claude but I prefer not to use Adobe out of principle so now that Da Vinci has an MCP - that’s the next step!
I’m sure this space will change very soon and hopefully we will have some better off-the-shelf solutions but in the meanwhile - always nice to build something that does exactly what you need!