JPEG Video¶

JPEG (Motion JPEG) video over RTP, as defined by RFC 2435. Every frame is an independent baseline JPEG image, so there is no inter-frame dependency: a lost packet costs one frame, never a group of pictures, and there is no keyframe wait when a receiver joins.

Overview¶

JPEG suits short, reliable links and sources that already produce JPEG, for example IP cameras, where H.264 latency or recovery behaviour is unwanted.

Building¶

The JPEG backend is built when libjpeg is found (libjpeg-dev on Debian and Ubuntu, libjpeg-turbo-devel on Fedora, RHEL and AlmaLinux). It can be switched off with -DBUILD_JPEG=OFF.

Transmitting JPEG Video¶

#include "jpeg/libjpeg/rtp_jpeg_payloader.h"

mediax::rtp::jpeg::RtpJpegPayloader payloader;

mediax::rtp::StreamInformation stream_info;
stream_info.session_name = "jpeg-stream";
stream_info.hostname = "239.192.1.1";
stream_info.port = 5004;
stream_info.width = 1280;
stream_info.height = 720;
stream_info.framerate = 25;
stream_info.encoding = mediax::rtp::ColourspaceType::kColourspaceJpeg;

payloader.SetStreamInfo(stream_info);
payloader.SetEncoderConfig({85, mediax::rtp::jpeg::JpegSubsampling::k422});  // quality 1-100, chroma subsampling
payloader.SetMtu(1400);                                                      // optional, the default is 1400
payloader.Open();
payloader.Start();

// Send RGB24 frames of the configured size, the payloader compresses and packetises them
std::vector<uint8_t> rgb_buffer(1280 * 720 * 3);
// Fill buffer with video frame...
payloader.Transmit(rgb_buffer.data(), true);

payloader.Close();

The quantisation tables are sent with every frame (RFC 2435 Q = 255), so a receiver needs no prior knowledge of the quality. The size fields of RFC 2435 count 8 pixel units, a frame that is not a multiple of 8 pixels is sent padded and receivers that know the real size cut it back.

Recording (OutputMode::kRecordOnly and kStreamAndRecord) is not supported by the JPEG payloader, Open() fails.

Receiving JPEG Video¶

#include "jpeg/libjpeg/rtp_jpeg_depayloader.h"

mediax::rtp::jpeg::RtpJpegDepayloader depayloader;

mediax::rtp::StreamInformation stream_info;
stream_info.session_name = "jpeg-stream";
stream_info.hostname = "239.192.1.1";
stream_info.port = 5004;
stream_info.width = 1280;   // optional, 0 uses the size from the stream
stream_info.height = 720;
stream_info.framerate = 25;
stream_info.encoding = mediax::rtp::ColourspaceType::kColourspaceJpeg;

depayloader.SetStreamInfo(stream_info);
depayloader.Open();
depayloader.Start();

mediax::rtp::RtpFrameData frame;
if (depayloader.Receive(&frame, 1000)) {
  // frame.cpu_buffer holds frame.resolution.width x height RGB24 pixels
}

depayloader.Close();

Frames can also be delivered to a callback with RegisterCallback(), and the compressed JPEG file of every frame is available before decoding through RegisterEncodedFrameCallback() (the data starts with FF D8 and ends with FF D9, annex_b is false).

The depayloader accepts what other RFC 2435 senders produce: types 0 and 1 with or without restart markers (types 64 and 65), quantisation tables sent in the stream (Q 128 to 255, including cached tables) and computed ones (Q 1 to 99). A frame with a lost packet is dropped as a whole, GetDroppedFrames() counts them. The receive queue holds the newest 8 frames when frames are read with Receive().

Hardware JPEG (VAAPI)¶

On Intel and AMD GPUs with a VAAPI JPEG engine, RtpJpegVaapiPayloader and RtpJpegVaapiDepayloader compress and decode the frames in hardware. They are built when libva is found, together with the other VAAPI codecs, and produce and accept the same stream as the libjpeg classes, so either end of a stream can be hardware or software.

#include "jpeg/vaapi/rtp_jpeg_vaapi_depayloader.h"
#include "jpeg/vaapi/rtp_jpeg_vaapi_payloader.h"

mediax::rtp::jpeg::vaapi::RtpJpegVaapiPayloader payloader;   // same API as RtpJpegPayloader
mediax::rtp::jpeg::vaapi::RtpJpegVaapiDepayloader depayloader;   // same API as RtpJpegDepayloader

if (mediax::rtp::jpeg::vaapi::RtpJpegVaapiPayloader::IsVaapiAvailable()) { /* the GPU encodes JPEG */ }

• The classes derive from the libjpeg ones. When the machine has no VAAPI JPEG engine, the frame size is below what the hardware accepts (16 x 16 pixels on Intel), or a frame fails on the hardware, libjpeg takes over, so they work on any machine. IsHardwareActive() and GetHardwareFrames() tell which path ran.
• The encoder and decoder are separate engines: some GPUs only decode. IsVaapiAvailable() is per class.
• The hardware encoder always produces the standard Huffman tables RFC 2435 requires.
• The hardware takes YCbCr surfaces and the RGB24 frames are converted by the CPU (full range, as JFIF needs), and the decoded surface is converted back the same way. The engine itself needs about 3 ms for a 1920 x 1080 frame, but the conversion and the copies to and from the surface cost more, so on a fast desktop CPU libjpeg-turbo, which uses SIMD throughout, is as quick or quicker in latency (measured at 1080p: hardware 8 to 9 ms encode and 7 to 8 ms decode, libjpeg-turbo 4 to 5 ms). The hardware pays off when the CPU is the scarce resource, for example several streams on an embedded Intel or AMD part, because the compression work is off the CPU.
• A quality of 100 is sent as 99, the largest Q value RFC 2435 can describe. 4:2:2 and 4:2:0 are both supported.
• DepayloaderFactory::DetectDecoders(ColourspaceType::kColourspaceJpeg) lists vaapi before libjpeg when the decoder is available, and CodecQuery lists "JPEG VAAPI".
• The hardware shares the process wide VAAPI display with the H.264, H.265 and AV1 backends.

SAP/SDP¶

JPEG streams are announced with the static payload type:

m=video 5004 RTP/AVP 26
a=rtpmap:26 JPEG/90000
a=fmtp:26 width=1280; height=720
a=framerate:25

and SapListener reports them as ColourspaceType::kColourspaceJpeg.

RTSP¶

IP cameras that stream JPEG over RTSP are received by the RTSP depayloader like any other codec. The static payload type 26 is recognised even when the camera leaves out the a=rtpmap line.

Examples¶

# Send a test card, then receive it and keep the last frame as a PPM image
build/bin/jpeg_transmit 127.0.0.1 5004 640 480 85 25 100
build/bin/jpeg_receive 127.0.0.1 5004 50 /tmp/last.ppm

Both examples take a trailing backend argument, auto (VAAPI hardware when present, the default), vaapi or libjpeg:

build/bin/jpeg_transmit 127.0.0.1 5004 1920 1080 85 25 100 libjpeg
build/bin/jpeg_receive 127.0.0.1 5004 50 /tmp/last.ppm vaapi

jpeg_transmit writes /tmp/jpeg_stream.sdp, which other software can use to view the stream:

ffplay -protocol_whitelist file,udp,rtp /tmp/jpeg_stream.sdp

MediaX also receives RFC 2435 streams made by other software, for example GStreamer (videotestsrc ! video/x-raw,format=YUY2 ! jpegenc ! rtpjpegpay ! udpsink) and ffmpeg (-c:v mjpeg -huffman default -force_duplicated_matrix 1 -pix_fmt yuvj420p -f rtp). RFC 2435 requires the standard Huffman tables, so an encoder that optimises them, as the ffmpeg mjpeg encoder does by default, cannot be used.

Qt6¶

QtRtpJpegPayloader and QtRtpJpegDepayloader wrap the classes above like the other Qt6 codec wrappers. Decoded RGB24 frames arrive through the newFrame(Frame) signal, with the size of the decoded image:

#include <qt6/QtRtpJpegDepayloader.h>
#include <qt6/QtRtpJpegPayloader.h>

mediax::qt6::QtRtpJpegDepayloader depayloader;
depayloader.setStreamInfo(stream_info);
QObject::connect(&depayloader, &mediax::qt6::QtRtpJpegDepayloader::newFrame, this, &Viewer::showFrame);
depayloader.open();
depayloader.start();

mediax::qt6::QtRtpJpegPayloader payloader;
payloader.setStreamInfo(stream_info);
payloader.setQuality(85);            // 1-100
payloader.setSubsampling420(false);  // 4:2:2 (default) or 4:2:0
payloader.open();
payloader.start();
emit payloader.sendFrame(frame);     // or payloader.transmit(&frame)

droppedFrames() reports the frames lost to missing packets. The wrappers are built with BUILD_QT6=ON and BUILD_JPEG=ON; include <qt6/QRtp> with MEDIAX_HAS_JPEG defined to get them with the other wrappers.

Viewing MJPEG Web Cameras¶

HTTP MJPEG streams and JPEG snapshot URLs are not RTP. They are shown by the qt6-rtsp-player example, see RTSP Receiver.

See Also¶

• JPEG 2000 Video, a different codec that is not yet available
• Streaming Formats