Distributed multithreaded cloud transcoding with hadoop, ffmpeg and nodejs

Most systems called cloud transcoding have few cloud features and are simply server-side transcoding software. True cloud transcoding should include a distributed file system, unlimited expansion, intelligent multithreaded processing and task allocation, single-machine multithreading, unlimited clusters and subclusters, and distributed processing across thousands of servers.

After prolonged research, I found a suitable approach: hadoop for clusters and the distributed file system; ffmpeg for image watermarks, scrolling text and multiresolution transcoding within that system; nodejs for intelligent control and task assignment; and mongodb for returned distributed-system data.

Diagram

Technical approach

The hadoop distributed file system supports command-line file control, much like local commands. Upload local files to distributed servers or retrieve them locally using simple commands.

Step one, upload: A nodejs API uploads videos to the controller. It divides videos into small segments and uploads them by command line to different distributed servers.

Step two, transcode: Nodes receive intelligently assigned segments and report to the controller. The controller sends transcoding commands; nodes transcode their segments. With six servers, six segments transcode simultaneously on separate nodes.

Step three, return: The controller waits for completion from every node, retrieves the results locally and joins the video.

Step four, cloud storage: Store the joined video simultaneously on several large storage servers and provide playback.

Additional technology: single-machine multithreaded transcoding. When there are too few nodes for a large video, split it further within a node, process multiple segments in parallel to use the entire CPU, then join and return the result to the controller.

Advantages: Expand the distributed system to thousands of servers, control the whole distributed file system through simple operations, and use the performance of the nodejs controller for tens of millions of concurrent requests and intelligent task assignment through nodejs. Optimized ffmpeg parameters preserve advanced operations such as watermark removal, watermark addition and scrolling text even on segmented videos.