FFmpeg

A few months ago, I took an interest in unconventional methods to manipulate and create visual artwork. I wanted to share some of thoughts. As well as the tools and examples that helped me develop skills.

If you really want an exceptional understanding of FFmpeg -- and just how magic it is -- you need to destroy media. Yes. Counterintuitively, destroying things with FFmpeg is the best way to illustrate how much power it has compared to anything else out there. Sure, you can add quick easy transitions and choose from a curated list of aspect ratios with most GUI video editing software. But that's BORING. Because they always work right the first time!There is little room for human conversation, little room to make mistakes. The software -- and by extension, it's creators -- decide for you what they believe is visually appealing. And because of that, there is almost no room for uniqueness. And it's my sincere belief that raw audio-visual data is a neglected medium that could flourish if not for a serious lack of interest, and free tools for experimentation.

You can create cursed and hilarious glitch art, convert ANY file into raw video/audio data, or just mess around with bitrates until your media looks and sounds like it was recorded in 1970.

Something important to remember is that FFmpeg can't be gleamed from ChatGPT. Or by copy-pasting random commands you find on the internet. If you do use ChatGPT to get something done quick, that's fine. But read the documentation! whenever you run into something unfamiliar. That's the only way you'll be able to add that filter/codec/muxer/argument to your mental toolbox.

For anyone who is interested, there's a neat little webpage which has a plethora of bash scripts, commands, walkthroughs and examples on this github page. There are activities and exercises, animated pictures, examples, and tutorials.

For anyone with a slightly more advanced understanding of FFmpeg's filter pipeline who would like a virtual playground to tinker around with, you can use this FFmpeg explorer. It's an easier to use visual analog that allows you to drag and drop filters and modifiers into a pipeline. There's even a little preview and demo video that updates in real-time as you make changes!

Hello. Based from existing posts from the internet, I was able to create a slideshow video in a folder containing numerous files, as in numerous images named in sequential order (eg. 01.png, 02.png, and so on).

ffmpeg.exe
   -reinit_filter 0
   -framerate 1/3
   -start_number 0
   -i "%02d.png"
   "output.mp4"

But, I would really like to add a transition effect between the image files.

There are existing posts out there, but they need to write down the actual files into the command. But what if the folder contains too many image files, like hundreds of them?

Thank you in advance.

UPDATE:

Found a working command from Superuser, the one answered by Michael. Transition effect is crossfade. I don't understand it myself though, but it indeed works after I tested it with my numerous image files in a folder.

Some commenters back there in Superuser seem to had errors when they used it for themselves. Likely, they didn't change the values of A, B, and WxH to whatever numerical value they actually needed, and filenames of their images were not 100% sequential. Or there was a gap like a missing numbered image file which would trigger FFmpeg to fully stop when it encountered that gap.

ffmpeg.exe
  -i "%02d.jpg"
  -vf "zoompan=d=(A+B)/B:s=WxH:fps=1/B, framerate=25:interp_start=0:interp_end=255:scene=100"
  "output.mp4"

where A is the duration in seconds how long each picture is shown (without crossfade duration), B is the crossfade duration in seconds, and WxH is the size of the output video.

Resizing then padding a centered video, maintaining aspect ratio, to a final 720 x 1280, with the same top an bottom colors, say darkgray, is below, which I think I got from Stack Overflow or somewhere (it works).

• scale=720:1280:flags=spline:force_original_aspect_ratio=decrease:eval=frame,pad=720:1280:-1:-1:eval=frame:color=darkgray

But I would like to have different colors for top and bottom paddings. See the example pic below, which I want. Thanks in advance.

Hi, As there is not much online about coding video material for lossless codecs I thought that since I gained knowledge from various forums and research I would share my conclusions. Don't criticize or slander for mistakes.

If anyone does not understand the concept of losslessness, loss, coding, bits, open source... then I do not recommend reading further and for experienced people I invite.

Well, yeah. In digital form, it is possible to save the moving image by saving the colors of each pixel and its location on each frame of the film. Such a form of recording is very ineffective and makes the size of a given material grow linearly because each second of the film consists of at least a dozen pictures. One photo of size 1MPX (~1080p) can weigh in uncompressed form up to 100MB. This makes even a film lasting one minute can start weighing 6GB which is very much. To deal with this, codecs are used which are divided into lossless and lossless which, depending on the function, provide a different form of compression. Lossless codecs are algorithms using magic tricks to save a given information in simple and smaller form, for example:

000111000 can be written 3;0.3;1.3;0 further 3;0.1,0 is one example. Or use Huffman coding. Losing codecs are algorithms that have the ability to delete certain information in such a way as to compress information without loss more efficiently, for example: 000211210400 is deleted for 2 and 4 and remains 3;0.3;1.3;0 further 3;0.1,0

Unfortunately, as the name itself indicates "loss" with 3;0,1,0 you can no longer get 000211210400 and the bits representing visual colored pixels were not fully preserved. Of course, this is very simplistic and in practice it does not work that way. Personally, I don't like the fact that something's lost. So I started learning and so I discovered some codecs that are as lossless as huffyuv and ffv1 and codecs that have the possibility of losslessness av1,vp9, x264, h264 (the latter are newer and more advanced).

So I did the tests and on the preliminary I would like to inform about not using any GUI-type programs to ffmpeg due to them, I will complete the specific parameters, at least in the thoughtless mode. I recommend using the command line with ffmpeg. I'll give you what I use.

So getting a video coded without loss is not easy because most recording programs use loss codecs. On happiness, the OBS (Open Broadcaster Software) program allows recording to ULH0 codec. It is a simple lossless codec that allows very fast encoding so that recording, for example, the desktop is done in real time. Let's assume that one second of recording takes 30MB of space which easily gives 1.8GB per minute. Of course, it's just mathematical calculation, but in practice, algorithms code information differently and in a more unpredictable way. When you get the video material (including the lossless audio codec that is irrelevant here) you can decode it to a completely uncompressed form and see how much it will take. With my material 18.5 seconds after decoding it takes about 10GB of 569MB at ULH0. Here are the commands for encoding to the lossless uncompressed codec: "ffmpeg -i input.avi -c:v r210 -an output.mkv" This shows how much space you can save using a really simple algorithm and short time. Therefore, in the form of uncompressed losslessness, it is pointless.

Since OBS encodes to ULH0 all progress and references I will base on it rather than on the uncompressed version (r210). Now you can encode the material (ULH0 569MB 18.5 seconds) using "huffyuv" (a code based mainly on Huffman coding) with the command: "ffmpeg -i input.avi -c:v huffyuv -an output.mkv" and you can get a file size of 1.3GB or less than a non-compressed version of about 10 times but also more than ULH0 therefore this encoding is pointless for archiving but it can be useful later to verify whether the loss codec/loss codec actually encodes without loss. You can also use "ffmpeg -i input.avi -c:v ffvhuff -an output.mkv" which will use an improved version of Huffyuv codec and miscellaneous material will be about 900MB less than Huffyuv but more than ULH0.

Now you can see how small video material can take away literally gigabytes of disk space and how important it is to code (compression).

Now you can enter already more advanced codecs and the first one who managed to compress the material more tightly is FFV1 or lossless codec compressed by ffmpeg. Its main goal was to effectively compress and at the same time maintain the speed of compression and decompression. The size I managed to get through the command: "ffmpeg -i input.avi -c:v ffv1 -an output.mkv" is 160MB, i.e. a reduction of to 26.8% where for lossless compressions it is a very good result. Here, I would like to compare the result to the video encoding (ULH0) by compressing LZMA2 compression which is not very effective with multimedia and does not allow to play the film without decompression. The result I managed to get is 26% ULH0 size, which is comparable to FFV1. But it's just a trivia and I don't recommend encode media in this way, especially already heavily encoded without loss or loss. By default, FFV1 uses level three and has an available 0 to 3 where 0 is the largest file size but the fastest encoding and 3 is the smallest file size and slowest encoding. Here I would like to inform you that the FFV1 encoding is "hell" fast and the time saved between mode 3 and 0 is meaning, therefore I recommend to use the default value, i.e. to set nothing but "-c:v ffv1". Here it is worth to emphasize that every time you want to encode a video file it is worth to uncode the original file and the output file to huffyuv and compare the size to bits, that is, the size to the end of is the same. If the size of the huffyuv from the output is smaller than the huffyuv from the original means the compression was lost. Here it is worth to emphasize that not always because the difference e.g. 4 or 8 bit does not mean the loss of information, but simply another recording of metadata where the output file may contain less metadata than the original file e.g. because the codec name is smaller. It's okay. However, if the difference is more than 20 bits then you can expect that a loss compression has been performed. Here, too, I do not recommend using PSNR, SIMM and VFAM indicators because they evaluate visual quality rather than mathematical quality, which can cause distortion and do not give the highest values despite the fact that the material has been encoded without loss. I would also like to inform you that recently HandBrake (for movie conversion) introduced support to FFV1. Unfortunately, when I used it to encode using FFV1 which is only lossless then after decoding to huffyuv by ffmpeg the sizes did not agree so I do not recommend to use HandBrake for lossless compression (unless I did something wrong. Returning when the size of huffyuv files after being decoded by ffv1 agrees you can try to use the command: "ffmpeg -i input.avi -c:v ffv1 -level 3 -threads 0 -compression_level 9 -coder 1 -context 1 -slides 24 -lossless 1 output.mkv" unofficially know that this command compresses the material more slowly and thus you can get 23.7% of the ULH0 size, which is less than normal FFV1 setting when I checked the size after the decoding to huffyuv agreed but I did not check the parameters in this comet exactly what they do so advise to be careful. The railways want to say here that this encoding is very fast and you can get a relatively small file size and be sure that no information has been lost because FFV1 does not support loss. It can also be added that FFV1 is fully open and free (free from patents which is different from the rest of later codecs). However, if someone needs stronger lossless compression no matter the speed of coding, you can reach for another codec: h264_nvenc.

In most graphics cards there is acceleration of encoding h264, so this can be used. Here is a command that encodes in the basic way material: "ffmpeg -i input.avi -c:v h264_nvenc -tune lossless -an output.mkv" Thanks to acceleration the compression lasts relatively much like with FFV1 but without acceleration it will not be possible to use it. The size that was achieved was 18.9% ULH0 which is definitely stronger than FFV1 with stronger settings. It is worth remembering here that after decoding to huffyuv I noticed the equators 4bit less in the output file which would mean that some metadata was lost. I think it might be related to the mkv format because when the output file is in .avi there is no difference of 4 bits. I don't think it matters much. Personally prefers mkv from avi because this first is fully open and free avi is closed technology by microsoft. We also have to remember that codec h264 is not open and is proprietary so it is a defect that makes me not personally use it. You can also use stronger encoding h264_nvenc by command: "ffmpeg -i input.avi -c:v h264_nvenc -tune lossless -preset p7 -an output.mkv" Here the encoding time of the insignificant increases but it is still fast no and is smaller file size unfortunately little because 16.5% of the size of ULH0. As before, there is also a variety of 4 bits after decrypting to huffyuv. Now we're going into slower codecs, and let's start with VP9. It is a fully open and free codec created by Google (it is uncertain with patents). It supports lossless and lossless mode. To encode the video to VP9 in a lossless mode at normal speed you need to use: "ffmpeg -i input.avi -c:v libvpx-vp9 -lossless 1 -an output.mkv" The compression time here is much longer than in the case of h264 but it is still quite fast. The file size to be obtained is 14.5% ULH0 which is stronger than the strongest mode h264_nvenc. At first glance, the 2% difference compared to h264_nvenc is not impressive but with the approach of compression in % to 0, progress is greatly decreasing due to lossless compression limits. However, after calculating the difference of these 2% in the file size 569 makes differences because in my case it is a reduced size from 92MB to 83MB which for lossless compressions is already a lot. And here comes another option with the slowest VP9 encoding in lossless mode but getting the least possible file size: "ffmpeg -i input.avi -c:v libvpx-vp9 -lossless 1 -deadline best -an output.mkv" Thanks to it we managed to compress the material to 10.9% ULH0 size but the coding time has already increased significantly. So much so that coding 22 minute material lasts all day. This is an option for someone who wants a really small size and it is not important for him to code time. In these three options after decoding the original and output file to huffyuv file sizes were always the same so it is certain that these commands cause lossless encoding. Now it came to codec AV1, quite a strong improvement in relation to VP9 because it originates from it. Also fully open and free (with small patents). Even though the codec has a loss and lossless mode, I understand why there is little talk about lossless mode. The speed with which av1 encodes in lossless mode is frighteningly long and the benefits obtained are not adequate to the time required for coding. Here is an AV1 encoding command in lossless mode that took 2 hours of file 18.5 seconds: "ffmpeg -i input.avi -c:v libaom-av1 -aom-params lossless=1 -crf 0 -an output.mkv" And the result is 10.4% ULH0 size, which is little less than the strongest VP9 but significantly longer. Due to on this personally I do not recommend to use AV1 for lossless mode. Codec is great for loss mode because with the same bitrade as VP9, he keeps more information. And with VP9 railways more than x264 at the same bitrade. For crazy people who are mentally ill and should be treated you can use the "-cpu-used 0" flag to command with av1 to try to slow down coding even more and get even less file size but I haven't tested this because it would probably take a week to record 18.5 seconds. In addition, it should be remembered that I noticed the differences of 4 bits when decoding huffyuv as with h264_nvenc which could mean that some metadata had changed. Either way, it's the strongest coding you can get through an open codec in a lossless way.

There is still a question of codec X264 which is an open implementation of h264 which is closed from the railways. Anyway, x264 is limited by patents, and this is his biggest flaw. The X264 for encoding uses the processor like vp9, av1 and ffv1. Interestingly, it achieves an even better result than encoding with av1 in lossless mode and to make it more fun with faster time than VP9. Here is a command that compresses to x264 normally in lossless mode: "ffmpeg -i input.avi -c:v libx264 -crf 0 -an output.mkv" The size that we got is 9.52% ULH0 size, which is less than av1 which is fascinating. Additions did not notice the difference in bits when decoded to huffyuv. For an even smaller size and thus the smallest possible video size in lossless mode can be used: "ffmpeg -i input.avi -c:v libx264 -preset placebo -crf 0 -an output.mkv" and the result is 9.32% ULH0 size, which is little and probably meaningless to normal people. But the good thing is that time has increased but not much in pregnancy is the quicksy than coding the slowest VP9. Unfortunately, because x264 uses closed h264 I would not personally use it, but it is nice that you can get less than 10% of the size of your video without loss.

In summary, each codec has its disadvantages and advantages and some are newer and compress better and others with railways are slower and compress better. It gives results so that everyone can choose what they want to get and easily copy and paste their command so they don't have to look for settings or test unnecessarily. It would be good to say that the size of the maximum compression with x264 or 53MB with 569MB is not a big result at all, looking at larger files, such as at ULH0 at the level of 50GB is x264 would be almost 5GB and it is a film about 30 minutes. From my calculations, VP9 with the strongest settings is about 200MB per minute and I think it's a reasonable value given the coding time. Note that the results that give are pure mathematical calculations and tests on one video file. But different results can be obtained on different videos. It may be that x264 does not compress to 9% and to e.g. 2% if the whole film is a black image, so there will be a lot of a recurring element or e.g. 30% if there are many unpredictable things that are poorly compressed. Anyway, personally for my archiving I use VP9 because it is open at the same time fast enough and relatively small. An additional advantage is that its equivalent of webp graphic format in lossless mode better compresses the file than AVIF format in lossless mode, or AV1 equivalent. If you want to get the smallest possible value in a relatively short time, use X264 if you depend on the full openness of the codec and guarantee of losslessness of high sissiness use FFV1, and if you like me on openness and small size is VP9.

Finally, I will give all encoding commands and their result and the accepted coding time.

TESTED LOSSLESS HUFFYUV

Basic (ULH0): 569MB video 18,5s File uncompressed (r210): 9,2GB (10s) >>> ffmpeg -i input.avi -c:v r210 -an output.mkv

Codec (huffyuv): 1,3GB (5s) >>> ffmpeg -i input.avi -c:v huffyuv -an output.mkv

Codec (ffvhuff): 949MB (73% size from huffyuv) (10s) >>> ffmpeg -i input.avi -c:v ffvhuff -an output.mkv

LZMA2: (26% from ULH0) (1m)

Codec (ffv1 basic): 152MB (26,8% from ULH0 ) (15s) >>> ffmpeg -i input.avi -c:v ffv1 -an output.mkv

Codec (ffv1 strongest): 134MB (23,7% from ULH0) (30s) >>> ffmpeg -i input.avi -c:v ffv1 -level 3 -threads 0 - compression_level 9 -coder 1 -context 1 -slices 24 -lossless 1 -an output.mkv

Codec (h264_nvenc basic): 107MB (18,9% from ULH0) (15s) >>> ffmpeg -i input.avi -c:v h264_nvenc -tune lossless -an output.mkv (loss 4bit)

Codec (h264_nvenc strongest): 93MB (16,5% from ULH0) (30s) >>> ffmpeg -i input.avi -c:v h264_nvenc -tune lossless -preset p7 -an output.mkv (loss 4bit)

Codec (vp9 basic): 82MB (14,5% from ULH0) (1m) >>> ffmpeg -i input.avi -c:v libvpx-vp9 -lossless 1 -an output.mkv

Codec (vp9 strongest): 62MB (10,9% from ULH0) (10m) >>> ffmpeg -i input.avi -c:v libvpx-vp9 -lossless 1 -deadline best -an output.mkv

Codec (av1 basic): 59MB (10,4% from ULH0) (1h) >>> ffmpeg -i input.avi -c:v libaom-av1 -aom-params lossless=1 -crf 0 -an output.mkv (loss 4bit) (Don't use that!)

Codec (av1 strongest): untested >>> ffmpeg -i input.avi -c:v libaom-av1 -aom-params lossless=1 -crf 0 -cpu-used 0 -an output.mkv (loss 4bit) (Don't use that!)

Codec (x264 basic): 54MB (9,52% from ULH0) (30s) >>> ffmpeg -i input.avi -c:v libx264 -crf 0 -an output.mkv

Codec (x264 strongest): 53MB (9,32% from ULH0) (1m) >>> ffmpeg -i input.avi -c:v libx264 -preset placebo -crf 0 -an output.mkv

Thank you for reading the whole thing and I very much hope that my post will help people like me to whom losing information is something terrible. Greetings, everyone.

cross-posted from: https://discuss.tchncs.de/post/18870157

I'm trying to extract the frames of a video as individual images but it's really slow, except when I'm using jpeg. The obvious issue with jpegs is the data loss from the compression, I want the images to be lossless. Extracting them as jpegs manages about 50-70 fps but as pngs it's only 4 fps and it seems to continue getting slower, after 1 minute of the 11 minute video it's only 3.5 fps.

I suspect it's because I'm doing this on an external 5tb hard drive, connected over USB 3.0 and the write speed can't keep up. So my idea was to use a different image format. I tried lossless jpeg xl and lossless webp but both of them are even slower, only managing to extract at about 0.5 fps or something. I have no idea why that's so slow, the files are a lot smaller than png, so it can't be because of the write speed.

I would appreciate it if anyone could help me with this.

Looks very interesting. Did anyone see a significant speed up in their usage of ffmpeg?

I'm trying to add the audio of a video to another one so I can have multiple tracks for my media service (Jellyfin).

The issue is the new track doesn't have any sound, it's just there with the proper metadata but it's just mute.
I tried playing it with Jellyfin and with VLC.
The original audio tracks play fine.

This is my command.

ffmpeg -i .\3.english.mkv -i .\3.french.mp4 -c copy -map 0 -map 1:a:0 -y .\3.mix.mkv

I also already tried adding -c:a:1 ac3 since this is the format of the audio in the mkv file.

The data of the original audio of the mkv is

Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 384 kb/s
    Metadata:
      title           : English
      BPS-eng         : 384000
      DURATION-eng    : 02:21:42.176000000
      NUMBER_OF_FRAMES-eng: 265693
      NUMBER_OF_BYTES-eng: 408104448
      _STATISTICS_WRITING_APP-eng: mkvmerge v25.0.0 ('Prog Noir') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2018-07-23 09:18:58
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES

The data of the file I'm trying to inject from the mp4 is

Stream #0:1[0x2](fre): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s (default)
    Metadata:
      creation_time   : 2014-04-17T18:14:55.000000Z
      handler_name    : movie.track_2.aac
      vendor_id       : [0][0][0][0]

do you guys have any idea of what might be the issue?

I also tried extracting the audio to a file and the aac file works fine, it's just when adding it to the mkv which doesn't work.

Was wondering what settings I should set on a VPS to get fast ffmpeg performance.