AI Music Video Experiment 1 – Rarotonga String Band – ‘Me Ito Roa’
Greetings, this video was a experiment in using Stable Diffusion for video content.
Stable Diffusion is an Open-Source Text to Image generator that can also do imgtoimg.
The original video is here: https://www.youtube.com/watch?v=0UC3x…
The technique that I used for this video was
1. Break the video down into smaller clips
2. Extract 10 frames per second using ffmpeg (a few dancing clips were extracted at 25 fps)
3. take a frame from each clip and use imgtoimg to try a few prompts and settings and pick the best one.
4. Process every frame from the clip using the settings and prompt picked from step 3.
5. Upscale the results using Real-ESRGAN
6. Turn the frames back into clips using ffmpeg
7. Use kdenlive to edit the video back together.