Example 2
24 Frames @ 12 FPS -> 48 Frames @ 24 FPS (x2)
Before FILM
After FILM
Example
24 Frames @ 8 FPS -> 72 Frames @ 24 FPS (x3)
Before FILM
After FILM
Here are some huggingface spaces to test it:
https://huggingface.co/spaces/fffiloni/video_frame_interpolation https://huggingface.co/spaces/jbilcke-hf/video_frame_interpolation
this is also in the default install of deforum on auto1111
Database here - https://huggingface.co/spaces/jbilcke-hf/media-server/raw/main/database.json
Example -
{
"version": 1,
"startAtShotId": "",
"sequences": [
{
"sequenceId": "d344637d-247f-4dd1-8700-02741f36a94a",
"skip": false,
"lastGenerationAt": "",
"videoPrompt": "3D animation, blender renderings, simulations showcasing a variety of different effects",
"audioPrompt": "electronic goal music from the 90s",
"tags": [
"trailer",
"cgi"
],
"channel": "main",
"shots": [
{
"shotId": "fa3b74d1-f27f-4fc4-a5f3-f29dd0d2750e",
"index": 0,
"lastGenerationAt": "",
"videoPrompt": "stunning 3D video of rippling water waves, reflected on hovering metal beads in an underwater setting. Rendered in Blender with subtle, whaling ambient light, captured in 8K UHD.",
"audioPrompt": ""
},
{
"shotId": "f89e600c-91e8-401d-a870-3286b779e5a3",
"index": 1,
"lastGenerationAt": "",
"videoPrompt": "high-quality 3D animation of a cloth-like structure falling in a windless room, displaying the realistic physics of fabric movement. Rendered in ultra-detailed Blender, shot in 4K Cinema4D.",
"audioPrompt": ""
},
{
"shotId": "333aae1c-db8c-424e-bb05-d7161952beea",
"index": 2,
"lastGenerationAt": "",
"videoPrompt": "splendid video showing the fluid simulation of liquids as they interact with static and moving objects, set in a minimalistic, clean environment. Captured with 4K Cinema4D, showcases Blender render.",
"audioPrompt": ""
},
{
"shotId": "a19a3e22-9009-449c-a011-6d6cdc660477",
"index": 3,
"lastGenerationAt": "",
"videoPrompt": "detailed 3D animation of cellular automata, using simple elements to create complex patterns in a uniform space. Exhibits award-winning lighting techniques and Blender rendering, shot in Cinema4D studio quality.",
"audioPrompt": ""
},
{
"shotId": "694cb3a9-7f51-4f7e-86df-7ae56d34e433",
"index": 4,
"lastGenerationAt": "",
"videoPrompt": "fabulous Blender rendering of a high-speed camera capture of metal beads impacting a flat surface, displaying brilliant n-body collision physics. Captured at a ripe evening with Cinema4D.",
"audioPrompt": ""
},
{
"shotId": "d59ec76d-4769-44ab-9aea-09362764d35d",
"index": 5,
"lastGenerationAt": "",
"videoPrompt": "breathtakingly crisp 3D animation featuring a fluid mechanism, where treacle-thick liquid pours into a large vat. Beautifully lit by diffused, soft light. Rendered in Blender, captured at 8K UHD with Cinema4D.",
"audioPrompt": ""
},
{
"shotId": "e170953f-4913-428d-a551-53128920e4c8",
"index": 6,
"lastGenerationAt": "",
"videoPrompt": "surreal yet hyper-realistic 3D animation of cloth simulation, showcasing the realism of Blender's material simulation capabilities. With studio lighting illuminating the scene, shot at close-up with Cinema4D.",
"audioPrompt": ""
},
{
"shotId": "1dcca91d-c9a4-4ed2-b011-1a55d990ca46",
"index": 7,
"lastGenerationAt": "",
"videoPrompt": "contemporary fine-art 3D animation showcasing the stunning interaction between thousands of metal beads under n-body simulation. Rendered with precision in Blender and captured at a high detail with Cinema4D.",
"audioPrompt": ""
},
{
"shotId": "a9296116-b5e5-43ef-914e-d75c7ad2750e",
"index": 8,
"lastGenerationAt": "",
"videoPrompt": "Blender-rendered 3D animation of a glass cube falling into a pool of liquid, captured to showcase fluid dynamics and refraction of light effects. Caught under a soft diffused light with Cinema4D.",
"audioPrompt": ""
},
{
"shotId": "6c8e7622-268e-491a-a6bc-0ad5b927c7c0",
"index": 9,
"lastGenerationAt": "",
"videoPrompt": "impressive 3D animation, of cellular automata, forming fascinating patterns in a confined environment. The geometrically exact elements are rendered with Blender. Shot at dusk, in crisp 8K UHD with Cinema4D.",
"audioPrompt": ""
}
]
},
{
Some notes from here: https://huggingface.co/spaces/jbilcke-hf/AI-WebTV/discussions/8
The main code of the webtv is located inside the media-server :
manual steps:
- human input to write a short paragraph describing a multi-shot video sequence
- manual submit it to GPT-4 to generate a list of video captions for each shot (the system instructions are extracts from a stable diffusion guide)
- commit the captions to the playlist database
Inside the media-server space (generation process running in the background):
- for each prompt in the database
- generate a silent 3 seconds video clip with Zeroscope V2 576w (hosted on Hugging Face Spaces)
- upscale the clip with Zeroscope V2 XL (also a HF Space)
- perform frame interpolation with FILM (also a HF Space)
- storage in the Persistent Storage of the media-server Space
Inside the media-server space (streaming process running in the foreground):
- for each video file in the persistent storage folder
- add it to a new FFmpeg playlist (it's just a .txt file)
- broadcast it over the RTMP protocol using FFmpeg (in FLV format)
- diffusion of the stream using node-media-server
Inside the AI-WebTV space:
- display the stream using mpegts.js
- this doesn't work on iPhone, but now there is also a Twitch mirror
Do you see your instance on https://lemmy.world/instances or other servers /instances? That's typically a good sign that your server is federated.
A new instance doesn't federate on its own. It needs to discover the other servers. Use something like https://lemmyverse.net/ to make links with your home server and visit the other instances. Subscribe to a few and you'll start getting those posts.
Lemmy isn't allowing uploads bigger than 1MB. Working on finding a solution.
Batch of 4 with the same prompt.
Upscale using ZeroScope v2 XL
view more: next ›
First stream up with AI video. Now to work on configuration and automation!