Video Encoding with Large Files

Posted on by Denis G Noonan

This week’s issue relates to encoding large video files. I had previously integrated open source software to encode video from a variety of formats to H.264 (using an mp4 container). The software worked fine with test data from small video files. Recently however, we have been testing with more realistic data from longer videos.

Soon into this stage of testing, we found that the encoding software which was running on our test server was failing.

We use the open source software ‘melt’ on Linux to do the video encoding. ‘Melt’ usually runs as a two-pass process.

So I went to look at our logs and determined that the ‘melt’ first pass in particular was failing. Next, I checked from running ‘top’ in another terminal that melt was using more and more memory and was terminating at 92% of total memory.

As I suspected that the Linux Out-Of-Memory killer had terminated the process, I went to check the kernel logs, as follows.

$ tail /var/log/kern.log
kernel: [32941851.928772] Out of memory: Kill process 3318 (melt) score 927 or sacrifice child
kernel: [32941851.928788] Killed process 3318 (melt) total-vm:4400676kB, anon-rss:3551936kB, file-rss:140kB

This confirms that the OOM-killer terminated the melt process as memory needs to be reclaimed to ensure that Linux could continue to run.

Now that I knew the source of the problem, first I tweaked the melt parameters to reduce its workload. However, the same problem occurred again with only a slight improvement in how far the encoding went.

Following one from this, I checked our Linux box and saw that no swap space had been allocated. From here, I could see that the solution was to configure swap memory in Linux to allow the melt process to complete.

So, we allocated 2GB of swap memory in addition to the existing 4GB of RAM and ran the encoding test again. This time, everything ran smoothly, with the swap memory being used to supplement RAM.