Introduction to SEI

In audio and video streaming, supplemental enhancement information (SEI) is additional data inserted into the bitstream to convey extra information, which can be received in accurate synchronization with the related audio and video content.

SEI can be inserted during the encoding and transmission of audio and video content. Various types of information can be added into SEI, such as parameters of the camera or encoder; time code; closed captions; lyrics; and copyright info. While standards define the syntax and semantics of how SEI messages should be inserted into the bitstreams, there is no requirement for decoders to be able to process this information. As such, SEI data may be discarded at some point within the content distribution chain if the equipment does not support it.

In addition to conveying technical information related to the bitstream, SEI can also be used for specific applications purposes. As an example, SEI could be used to send questions in live interactive quizzes, with the audience answering on an app to earn points or rewards. The diagram below shows a sample implementation of this workflow. While the host asks questions, the on-site staff sends instructions to the app server to insert several SEI messages (such as the content of the questions, and time remaining to answer) into the live stream. After the app receives the video stream, it analyzes the SEI messages and immediately requests the corresponding content from the app server for display, completing the quizzing process.

Magewell Ultra Encode devices can insert time code and closed captions as H.264 and HEVC SEI messages. Ultra Encode HDMI can extract time code from the Vendor Specific InfoFrame of the HDMI input signal and write it into the SEI; Ultra Encode SDI supports the system time code or SMPTE RP-188 time code extracted from the SDI input signal. Ultra Encode also preserves closed captions from the input signal that comply with the CEA-608 or CEA-708 standard.