SRT Video (Drones)¶

MediaX sends and receives H.264, H.265 and AV1 video over SRT (Secure Reliable Transport), a UDP protocol that recovers lost packets by retransmission within a fixed latency window. It suits a drone video link: a radio path with a round trip time of tens of milliseconds, bursts of loss and a ground station that needs the picture a few hundred milliseconds after it was taken.

The protocol is implemented natively in mediax::srt (draft-sharabayko-srt, live streaming mode). There is no dependency on libsrt. Encryption uses OpenSSL, if it is not found the library builds without encryption and refuses streams that ask for it. It interoperates with libsrt (and so with ffmpeg, ffplay, VLC, OBS and srt-live-transmit).

Build option: -DBUILD_SRT=ON (the default, not available on Windows).

How it works¶

Encoder (H.264 / H.265 / AV1) -> TS multiplexer -> SRT sender (ARQ, pacing)
        |
        |  UDP over the lossy radio link
        v
SRT receiver (NAK, timed release) -> TS demultiplexer -> RTP packetiser -> decoder -> RGB24

• Every packet is stamped when it is sent. The receiver releases it exactly latency later, so jitter and retransmissions are absorbed by the delay and the picture arrives at a constant rate.
• The receiver reports gaps immediately and then every max((RTT + 4 x RTTvar) / 2, 20 ms). The sender retransmits ahead of new data.
• A packet that cannot be recovered before its release time is skipped (too-late packet drop) and the sender is told, so nothing ever stalls. The video decoder is then held back until the next key frame so a damaged picture is never shown (see Loss handling).
• Sending is paced to the measured input rate plus an overhead for retransmissions, so a burst such as a key frame does not flood a slow radio.

Tuning for a drone link¶

Choose the latency from the round trip time of the radio link. The recommended value is 3 to 4 times the RTT, plus a margin for a burst of loss.

auto config = mediax::srt::SrtConfig::ForDroneLink(/*expected_rtt_ms=*/50);
// latency 260 ms (4 x RTT + 60 ms, limited to 100..500 ms), 30 % retransmission overhead

Other settings worth knowing:

• maxbw (bytes per second): the send rate limit. By default it follows the video bit rate plus oheadbw percent. Set it to the capacity of the radio link when that is known.
• oheadbw (percent, default 25, 30 with the drone preset): room left for retransmissions. Raise it on lossy links.
• peeridletimeo (ms, default 5000): how long the link may be silent before it is declared lost.
• A key frame interval of one second means a recovery after an unrecoverable loss takes at most one second.

URLs¶

srt://host:port?mode=caller|listener|rendezvous&latency=ms&passphrase=...&pbkeylen=16|24|32&streamid=...

Connection modes¶

Direction and mode are independent: a listener can be the sender or the receiver. After a connection ends, a listener accepts the next caller, and a caller reconnects, so a drone that flies out of range and back resumes by itself.

Receiving¶

mediax::srt::SrtDepayloader is a drop-in RtpDepayloader, like the RTSP and HLS receivers. It finds the codec in the stream, creates the decoder (VAAPI or OpenH264 for H.264, VAAPI for H.265 and AV1) and delivers RGB24 frames.

#include "srt/srt_depayloader.h"

mediax::srt::SrtDepayloader rx;
rx.SetUrl("srt://:9000?mode=listener&latency=250&passphrase=correct-horse-battery");
rx.RegisterErrorCallback([](mediax::srt::SrtStatus status, const std::string& message) {
  std::cerr << message << "\n";
});
rx.Open();   // validates the settings
rx.Start();  // returns at once, connects (and reconnects) in the background

mediax::rtp::RtpFrameData frame;
while (rx.Receive(&frame, 5000)) {
  // frame.cpu_buffer holds RGB24 data
}

auto stats = rx.GetStatistics();  // RTT, loss, recoveries, drops, reconnects, frames
rx.Close();

For GUI programs use mediax::qt6::QtSrtDepayloader, which delivers frameReady(QImage) signals ("latest wins") and never blocks the event loop.

Loss handling¶

When packets are lost and cannot be recovered in time, the transport stream has a hole. By default the receiver discards access units until the next key frame (SetWaitForKeyFrameAfterLoss(true)), freezing the last good picture instead of showing corrupted ones. GetStatistics() reports ts_continuity_errors and frames_discarded. Pass false to decode everything and accept artefacts until the next key frame.

Sending¶

mediax::srt::SrtTransmitter muxes access units into MPEG-TS and sends them. Feed it from any encoder that produces Annex B H.264 or H.265, or AV1 temporal units (see AV1):

#include "srt/srt_transmitter.h"

mediax::srt::SrtTransmitter tx;
tx.SetUrl("srt://gcs.example:9000?latency=250&passphrase=correct-horse-battery");
tx.SetCodec(mediax::rtp::ColourspaceType::kColourspaceH265);  // default is H.264
tx.RegisterKeyFrameRequestCallback([&] { encoder.ForceKeyFrame(); });
tx.Open();
tx.Start();

while (capturing) {
  tx.SendAccessUnit(nal_data, nal_size, pts_90khz, is_key_frame);
}
tx.Close();

• Parameter sets (SPS, PPS and VPS) are repeated in front of key frames that lack them, and an access unit delimiter is added, so a receiver can join at any key frame.
• With a key frame request callback registered, a new peer is held until a key frame arrives and the callback asks the encoder for one. Without a callback everything is sent.
• If the link cannot take a picture (the peer is not acknowledging), the picture is dropped, the encoder is asked for a key frame and sending restarts there. Nothing queues up, so latency never grows.

MediaX encoders¶

The OpenH264, VAAPI H.264, VAAPI H.265 and VAAPI AV1 encoders can send straight to SRT. Attach() switches off their RTP output and passes each encoded access unit on:

mediax::rtp::h264::openh264::RtpH264OpenH264Payloader encoder;
encoder.SetStreamInfo(stream_info);
tx.Attach(&encoder);
encoder.Open();
tx.Start();
encoder.Transmit(rgb_frame);   // encoded, muxed and sent over SRT

Other encoders can use SendAccessUnit() directly. To attach another RtpPayloader backend, call NotifyEncodedUnit() from its Transmit() as the four above do.

AV1¶

AV1 travels in the same MPEG-TS and SRT framing as H.264 and H.265. Select it with SetCodec(); the access unit you send is an AV1 temporal unit, a sequence of OBUs, instead of an Annex B access unit:

tx.SetCodec(mediax::rtp::ColourspaceType::kColourspaceAv1);
tx.RegisterKeyFrameRequestCallback([&] { encoder.ForceKeyframe(); });
tx.Attach(&encoder);            // mediax::rtp::av1::vaapi::RtpAv1VaapiPayloader
// or, from any AV1 encoder: tx.SendAccessUnit(temporal_unit, size, pts_90khz, is_key_frame);

• The receiver needs no setting: it recognises AV1 in the transport stream, creates the VAAPI AV1 decoder and delivers RGB24 frames like any other codec. Without VAAPI AV1 decode hardware no decoder is created and an error is reported.
• Each temporal unit is sent starting with a temporal delimiter and with a size field on every OBU. The transmitter adds the temporal delimiter and size fields if the encoder left them out, and repeats the sequence header in front of key frames that lack one, so a receiver can join at any key frame.
• Key frames are shown key frames (show_frame = 1). They are found from the frame header, so SendAccessUnit(data, size) without a key_frame argument works for AV1 too.
• The picture size comes from the sequence header. The loss handling is the same as for H.264 and H.265: after an unrecoverable loss the decoder waits for the next key frame.
• In the transport stream AV1 is private PES data (stream_type 0x06) described by an AV01 registration descriptor and an AV1 video descriptor in the PMT, as in the AOM specification "Carriage of AV1 in MPEG-2 TS".

The MediaX AV1 depayloader decodes 8-bit 4:2:0 (Main profile) streams only.

RTP to SRT gateway¶

MediaX can carry H.264 and H.265 between RTP and SRT without decoding or re-encoding: the NAL units are only repacked (RFC 6184 / RFC 7798 RTP on one side, MPEG-TS on the other), so a gateway costs very little CPU and adds no picture latency of its own. mediax::gateway::SrtGateway does the receiving, converting, retransmitting and announcing; the application only says what to convert.

#include "gateway/srt_gateway.h"

mediax::gateway::SrtGateway gateway;

// A SAP announced RTP stream goes out over SRT
mediax::gateway::RtpToSrtRouteConfig camera;
camera.session_name = "Camera 1";                      // The SAP session name to convert
camera.srt_url = "srt://ground-station.example:9000?latency=200";
gateway.AddRtpToSrt(camera);

// An SRT source is retransmitted as RTP and announced with SAP
mediax::gateway::SrtToRtpRouteConfig drone;
drone.session_name = "Drone 7";                        // The name to announce
drone.srt_url = "srt://:9100?mode=listener&latency=300";
drone.rtp_address = "239.192.10.7";                    // Multicast or unicast
drone.rtp_port = 5004;
gateway.AddSrtToRtp(drone);

gateway.Start();
for (const auto& route : gateway.GetStatus()) {
  // route.state, route.last_error, route.rtp_to_srt.conversion, route.srt_to_rtp.stream ...
}

• RTP to SRT: the route waits for the SAP announcement with the given session name, joins the multicast group (or listens on the unicast address) and port it describes, and sends the video to the SRT URL. A deleted announcement stops the route, a new one resumes it, and a changed address, port or codec restarts it. Both H.264 and H.265 are taken from the announcement.
• SRT to RTP: the route connects to the SRT source and, once the first key frame shows the codec and picture size (read from the sequence parameter set), starts the RTP output and announces it with SAP. The announcement is withdrawn when the SRT link drops and made again when it returns. The announced frame rate is framerate from the configuration, as the stream does not carry one.
• After a packet loss that could not be recovered, nothing is forwarded until the next key frame, so receivers never get damaged pictures. On the RTP side the route can ask the source for a key frame through RtpToSrt::RegisterKeyFrameRequestCallback() if the gateway uses the converters directly.
• Any number of routes can run in one process. The gateway registers a catch-all callback with the SapListener singleton, which replaces any other catch-all callback the application registered.

Limits of the conversion¶

• The gateway handles H.264 and H.265 only. AV1 is not converted: RtpToSrt::SetCodec() refuses it with kNotSupported and SrtToRtp does not forward AV1 temporal units.
• Frame reordering (B frames) is not supported from RTP to SRT: RTP carries no decode time stamps, so the transport stream gets DTS = PTS. SRT to RTP is not affected.
• RTP packets are not reordered: a gap in the sequence numbers drops the picture it falls in. The first access unit of a stream is dropped, as it may be incomplete.
• Only the non-interleaved RTP modes are supported (single NAL unit, STAP-A / AP and FU-A / FU packets).
• The SAP announcement is the minimal one MediaX produces (picture size, no sprop-parameter-sets). The parameter sets are sent in the stream in front of every key frame.

Building your own gateway¶

The pieces of SrtGateway can be used on their own, for instance when the RTP comes from somewhere other than SAP:

Examples¶

# Ground station: wait for a drone, encrypted, tuned for a 50 ms link
srt-receive --url="srt://:9000?mode=listener&passphrase=correct-horse-battery" --drone --rtt=50

# Drone (or a test): send a test pattern
srt-transmit --url="srt://127.0.0.1:9000?passphrase=correct-horse-battery" --drone --rtt=50

# Interop: ffmpeg sends to MediaX
ffmpeg -re -f lavfi -i testsrc=size=1280x720:rate=25 -pix_fmt yuv420p -c:v libx264 -tune zerolatency -g 25 -f mpegts \
  "srt://127.0.0.1:9000?mode=caller&latency=250000&passphrase=correct-horse-battery"

# Interop: ffplay receives from MediaX
ffplay "srt://:9000?mode=listener&latency=250000"

Test loss recovery without hardware by impairing the loopback interface:

sudo tc qdisc add dev lo root netem delay 20ms 5ms loss 5%
# ... run srt-transmit and srt-receive, watch lost= and recovered= in the receiver output ...
sudo tc qdisc del dev lo root

Statistics¶

SrtStatistics (also in SrtReceiveStatistics::link and SrtTransmitStatistics::link) reports, per connection: packets and bytes sent and received, retransmissions, packets lost, recovered, dropped too late and dropped by the sender, duplicates, ACK and NAK counts, RTT and its variance, the latency in use in each direction and the send and receive rates.

Limits¶

• Live mode only: no file transfer mode, message mode with multi-packet messages, packet filter (FEC) or connection groups (bonding).
• One key is used for a connection. libsrt refreshes the key after 2^25 packets (about 9 hours at 1000 packets per second), MediaX does not rekey a running connection. Reconnect to rotate the key.
• The receiver decodes the first video stream of the first program. Audio is ignored.
• A transport stream whose video PES packets have no length (many encoders, including ffmpeg's) is delivered one picture late, when the next picture starts. Streams from MediaX carry the length and are delivered at once.
• Rendezvous mode is tested against itself. It could not be tested against libsrt here because ffmpeg binds a rendezvous socket to the peer's address, so it can only rendezvous with a peer on another host.
• Windows is not supported.