• 24 Posts
  • 13 Comments
Joined 2 years ago
Cake day: July 3rd, 2023
  • DanQuix0te@feed.timeloop.tvOPMtoText 2 Video@feed.timeloop.tv•AI-WebTV Prompt GuideEnglish
    1·
    2 years ago

    Database here - https://huggingface.co/spaces/jbilcke-hf/media-server/raw/main/database.json

    Example -

    {
  "version": 1,
  "startAtShotId": "",
  "sequences": [
    {
      "sequenceId": "d344637d-247f-4dd1-8700-02741f36a94a",
      "skip": false,
      "lastGenerationAt": "",
      "videoPrompt": "3D animation, blender renderings, simulations showcasing a variety of different effects",
      "audioPrompt": "electronic goal music from the 90s",
      "tags": [
        "trailer",
        "cgi"
      ],
      "channel": "main",
      "shots": [
        {
          "shotId": "fa3b74d1-f27f-4fc4-a5f3-f29dd0d2750e",
          "index": 0,
          "lastGenerationAt": "",
          "videoPrompt": "stunning 3D video of rippling water waves, reflected on hovering metal beads in an underwater setting. Rendered in Blender with subtle, whaling ambient light, captured in 8K UHD.",
          "audioPrompt": ""
        },
        {
          "shotId": "f89e600c-91e8-401d-a870-3286b779e5a3",
          "index": 1,
          "lastGenerationAt": "",
          "videoPrompt": "high-quality 3D animation of a cloth-like structure falling in a windless room, displaying the realistic physics of fabric movement. Rendered in ultra-detailed Blender, shot in 4K Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "333aae1c-db8c-424e-bb05-d7161952beea",
          "index": 2,
          "lastGenerationAt": "",
          "videoPrompt": "splendid video showing the fluid simulation of liquids as they interact with static and moving objects, set in a minimalistic, clean environment. Captured with 4K Cinema4D, showcases Blender render.",
          "audioPrompt": ""
        },
        {
          "shotId": "a19a3e22-9009-449c-a011-6d6cdc660477",
          "index": 3,
          "lastGenerationAt": "",
          "videoPrompt": "detailed 3D animation of cellular automata, using simple elements to create complex patterns in a uniform space. Exhibits award-winning lighting techniques and Blender rendering, shot in Cinema4D studio quality.",
          "audioPrompt": ""
        },
        {
          "shotId": "694cb3a9-7f51-4f7e-86df-7ae56d34e433",
          "index": 4,
          "lastGenerationAt": "",
          "videoPrompt": "fabulous Blender rendering of a high-speed camera capture of metal beads impacting a flat surface, displaying brilliant n-body collision physics. Captured at a ripe evening with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "d59ec76d-4769-44ab-9aea-09362764d35d",
          "index": 5,
          "lastGenerationAt": "",
          "videoPrompt": "breathtakingly crisp 3D animation featuring a fluid mechanism, where treacle-thick liquid pours into a large vat. Beautifully lit by diffused, soft light. Rendered in Blender, captured at 8K UHD with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "e170953f-4913-428d-a551-53128920e4c8",
          "index": 6,
          "lastGenerationAt": "",
          "videoPrompt": "surreal yet hyper-realistic 3D animation of cloth simulation, showcasing the realism of Blender's material simulation capabilities. With studio lighting illuminating the scene, shot at close-up with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "1dcca91d-c9a4-4ed2-b011-1a55d990ca46",
          "index": 7,
          "lastGenerationAt": "",
          "videoPrompt": "contemporary fine-art 3D animation showcasing the stunning interaction between thousands of metal beads under n-body simulation. Rendered with precision in Blender and captured at a high detail with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "a9296116-b5e5-43ef-914e-d75c7ad2750e",
          "index": 8,
          "lastGenerationAt": "",
          "videoPrompt": "Blender-rendered 3D animation of a glass cube falling into a pool of liquid, captured to showcase fluid dynamics and refraction of light effects. Caught under a soft diffused light with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "6c8e7622-268e-491a-a6bc-0ad5b927c7c0",
          "index": 9,
          "lastGenerationAt": "",
          "videoPrompt": "impressive 3D animation, of cellular automata, forming fascinating patterns in a confined environment. The geometrically exact elements are rendered with Blender. Shot at dusk, in crisp 8K UHD with Cinema4D.",
          "audioPrompt": ""
        }
      ]
    },
{
  • DanQuix0te@feed.timeloop.tvOPMtoText 2 Video@feed.timeloop.tv•AI WebTV - jbilcke-hfEnglish
    1·
    2 years ago

    Some notes from here: https://huggingface.co/spaces/jbilcke-hf/AI-WebTV/discussions/8

    The main code of the webtv is located inside the media-server :

    manual steps:

    • human input to write a short paragraph describing a multi-shot video sequence
    • manual submit it to GPT-4 to generate a list of video captions for each shot (the system instructions are extracts from a stable diffusion guide)
    • commit the captions to the playlist database

    Inside the media-server space (generation process running in the background):

    • for each prompt in the database
    • generate a silent 3 seconds video clip with Zeroscope V2 576w (hosted on Hugging Face Spaces)
    • upscale the clip with Zeroscope V2 XL (also a HF Space)
    • perform frame interpolation with FILM (also a HF Space)
    • storage in the Persistent Storage of the media-server Space

    Inside the media-server space (streaming process running in the foreground):

    • for each video file in the persistent storage folder
    • add it to a new FFmpeg playlist (it’s just a .txt file)
    • broadcast it over the RTMP protocol using FFmpeg (in FLV format)
    • diffusion of the stream using node-media-server

    Inside the AI-WebTV space:

    • display the stream using mpegts.js
    • this doesn’t work on iPhone, but now there is also a Twitch mirror

Moderates