Recorder & Transcoder Project

Project Concluded:   April 05, 2023

DB Reference:   asl_recorder

Last Updated:   September 26, 2023

screenshot of convert code in Python

Introduction

It's not uncommon to have friends who live stream, and some of them opt to keep a copy of their streaming videos to produce a highlight collection later. However, storing a large amount of video can consume a lot of space, and recording simultaneously can also impact streaming performance. Given the considerable amount of free computational capacity within the server clusters, I decided to lend a hand to my friends.

The project comprises three main components: recorders that support multiple platforms, such as twitchTV, Bilibili, or TikTok; a transcoder that converts all recorded video into H264(.mp4) format after the streaming has ended; and a data storage solution.

Specification & Structures

As of April 2023, the system has the following specification:

Hardware: 1 * 4 Core CPU & 4G Memory Virtual Server(Windows)

Hardware: 1 * 4 Core CPU & 6G Memory Virtual Server(Debian 11)

Hardware: 1 * 24 Core CPU & 12G Memory VIrtual Server(Debian 11)

Software: BililiveRecorder + ffmpeg + Python3

Detailed Description

This project has three VMs: two for hositng the recorder, and one for the transcoder.

The first part of this project is the recorder. Using the open-source project like BililiveRecorder or twitch-stream-recorder, It is easy to automatically record videos and store them in the backend storage whenever the streaming is on.

The second part of the project involves a transcoder, which is necessary because some recorders output video in a format that is not compatible with editing software, such as ".flv". Since format conversion can be time-consuming, it is better to do it early. To address this, I developed an automatic transcoder that monitors a list of locations in real-time using the popular ffmpeg as the core. Once a new video file is found, it is added to the monitoring list. Every 10 seconds, the script checks the file size of those on the list. If the size of any of the files does not change, indicating that the current streaming session is over and it is safe to manipulate the file, that file will be added to the conversion queue and processed in the high-performance converter.

screenshot of bilibili recorder

The third part of the project is focused on storage and is divided into two tracks based on different requirements. The first track involves storing videos in the storage cluster, which has limited space, typically 500 GB per user. Once the source files have been used for video editing, they are removed to ensure that the total space does not exceed the threshold. The second track involves storing the converted files on tape. Tape storage is much cheaper than disk arrays. Based on the existing tape library system, we will be able to store these cold data for a long time. For more information about the tape storage system, please refers to Hybrid Storage Project