Python Bindings

Python API Reference

Python bindings for MediaX, generated with SWIG. They cover stream descriptions, the uncompressed payloader, the H.264, H.265 and AV1 encoders and decoders, the RTSP receiver and SAP/SDP announce and discover. Everything else (JPEG, SRT, HLS, Chromecast, GigE Vision, colourspace conversion) is C++ and Qt6 only, see the C++ API and Qt6 API.

Behaviour to know about

  • Frames are passed as buffers. Transmit() accepts any contiguous buffer (bytes, bytearray, memoryview, a C-contiguous numpy array), checks it holds at least width * height * bytes_per_pixel bytes and raises ValueError if not. It always blocks until the frame is sent, so the blocking argument is ignored.
  • Received frames are read with RtpFrameData.tobytes(), which copies the pixels out. RtpFrameData.cpu_buffer itself is a raw pointer owned by the depayloader.
  • std::string_view arguments take a Python str.
  • C++ exceptions are raised as RuntimeError, they never terminate the interpreter.
  • Destroying a started payloader or depayloader is safe, but call Stop() and Close() when you are finished.
  • Callbacks (RegisterCallback() and similar) and factories cannot be used from Python, frames are polled with Receive().

Installation

Build with -DBUILD_PYTHON=ON (needs swig and the Python 3 development headers), or install the package:

sudo dpkg -i mediax-python_<version>_<arch>.deb

The modules are installed as the mediax package. The location depends on the package format:

Install Location
.deb (Ubuntu) /usr/lib/python3/dist-packages/mediax/
.rpm (AlmaLinux) /usr/lib64/python3.X/site-packages/mediax/
make install The same directory as the package for the distribution, under <prefix> (for example /usr/local/lib/python3/dist-packages/mediax/)

The extension modules are compiled for one Python minor version, so the packages depend on exactly that version (for example python3 (>= 3.12), python3 (<< 3.13)). Build the bindings with the interpreter you will run them with.

The package locations are on Python's default path. For make install add the directory that contains mediax to sys.path (or set PYTHONPATH), then import the modules from the package:

import sys
sys.path.append('/usr/local/lib/python3/dist-packages')

from mediax import rtp_types
from mediax import sap_listener

import mediax also makes the modules available as attributes (mediax.rtp_types). There is no mediax.rtp object, the stream types are in rtp_types. Importing by bare module name (import rtp_types with the mediax directory on sys.path) is not supported, because the modules import each other relatively.

Modules

Module Built when Contents
rtp_types Always StreamInformation, RtpFrameData, ColourspaceType_*, StatusCode_*, OutputMode_*
rtp_payloader Always RtpPayloader base class (abstract)
rtp_depayloader Always RtpDepayloader base class (abstract)
rtp_uncompressed_payloader Always RtpUncompressedPayloader
rtp_uncompressed_depayloader Always RtpUncompressedDepayloader
rtsp_depayloader BUILD_RTSP=ON RtspDepayloader, RtspClientConfig, RtspStatistics, RtspTrackInfo, status and state enums, RtpFrameData
rtp_h264_openh264 BUILD_OPENH264=ON RtpH264OpenH264Payloader, RtpH264OpenH264Depayloader, EncoderConfig, DecoderConfig
rtp_h264_vaapi libva found RtpH264VaapiPayloader, RtpH264VaapiDepayloader, VaapiEncoderConfig, VaapiDecoderConfig, profile and level enums
rtp_h265_vaapi libva found RtpH265VaapiPayloader, RtpH265VaapiDepayloader, VaapiH265EncoderConfig, VaapiH265DecoderConfig, profile and level enums
rtp_av1_vaapi libva found RtpAv1VaapiPayloader, RtpAv1VaapiDepayloader, VaapiAv1EncoderConfig, VaapiAv1DecoderConfig, Av1Profile
sap_announcer Always SapAnnouncer (singleton)
sap_listener Always SapListener (singleton), SdpMessage

SWIG drops C++ namespaces, so mediax::rtp::ColourspaceType::kColourspaceRgb24 becomes the module-level constant rtp_types.ColourspaceType_kColourspaceRgb24. Integer types (uint8_t, uint16_t, uint32_t) map to Python int.

Stream Information

from mediax import rtp_types

stream_info = rtp_types.StreamInformation()
stream_info.session_name = "python-stream"
stream_info.hostname = "239.192.1.1"
stream_info.port = 5004
stream_info.width = 1920
stream_info.height = 1080
stream_info.framerate = 30
stream_info.encoding = rtp_types.ColourspaceType_kColourspaceRgb24

Fields: session_name, hostname, port, width, height, framerate, encoding, deleted.

Constants

# Colourspace / encoding (rtp_types.ColourspaceType_*)
kColourspaceRgb24        # 24-bit RGB
kColourspaceYuv422       # YCbCr 4:2:2 (UYVY)
kColourspaceYuv420p      # YUV 4:2:0 planar
kColourspaceMono8        # 8-bit greyscale
kColourspaceMono16       # 16-bit greyscale
kColourspaceRgba         # Not a GVA uncompressed video format
kColourspaceBgra         # Not a GVA uncompressed video format
kColourspaceNv12         # Not a GVA uncompressed video format
kColourspaceJpeg2000     # JPEG 2000 (not currently implemented)
kColourspaceH264Part4    # H.264 (Part 4)
kColourspaceH264Part10   # H.264 (Part 10)
kColourspaceH265         # H.265 / HEVC
kColourspaceAv1          # AV1
kColourspaceJpeg         # JPEG (RFC 2435)
kColourspaceUndefined

# Status (rtp_types.StatusCode_*): kStatusOk (0), kStatusError
# Output mode (rtp_types.OutputMode_*): kStreamOnly, kRecordOnly, kStreamAndRecord

The compressed encodings select the codec of a stream. The payloaders and depayloaders are in the codec modules, see Compressed Video.

Uncompressed Streaming

Transmit

RtpUncompressedPayloader follows the same lifecycle as the C++ class: SetStreamInfo(), Open() (returns bool), Start(), Transmit(), Stop(), Close().

from mediax import rtp_types
from mediax import rtp_uncompressed_payloader

stream_info = rtp_types.StreamInformation()
stream_info.session_name = "rgb-stream"
stream_info.hostname = "239.192.1.1"
stream_info.port = 5004
stream_info.width = 640
stream_info.height = 480
stream_info.framerate = 30
stream_info.encoding = rtp_types.ColourspaceType_kColourspaceRgb24

payloader = rtp_uncompressed_payloader.RtpUncompressedPayloader()
payloader.SetStreamInfo(stream_info)
if not payloader.Open():
    raise RuntimeError("Failed to open the payloader")
payloader.Start()

frame = bytes(640 * 480 * 3)               # RGB24, any contiguous buffer works
if payloader.Transmit(frame) != rtp_types.StatusCode_kStatusOk:
    print("Frame not sent")

payloader.Stop()
payloader.Close()

With numpy:

import numpy as np

frame = np.zeros((480, 640, 3), dtype=np.uint8)
frame[:, :, 0] = 255                       # Red
payloader.Transmit(np.ascontiguousarray(frame))

The buffer must hold at least width * height * bytes_per_pixel bytes for the configured encoding, otherwise ValueError is raised. Transmit() always blocks until the frame has been sent.

Other members: SetWidth(), SetHeight(), SetFrameRate(), SetIpAddress(), SetPort(), SetColourSpace(), SetOutputMode(), SetRecordingFilename(), GetBufferSize() and the matching getters. Transmit() returns an int, StatusCode_kStatusOk (0) on success.

Receive

from mediax import rtp_types
from mediax import rtp_uncompressed_depayloader

stream_info = rtp_types.StreamInformation()
stream_info.hostname = "239.192.1.1"
stream_info.port = 5004
stream_info.width = 640
stream_info.height = 480
stream_info.encoding = rtp_types.ColourspaceType_kColourspaceRgb24

depayloader = rtp_uncompressed_depayloader.RtpUncompressedDepayloader()
depayloader.SetStreamInfo(stream_info)
if not depayloader.Open():
    raise RuntimeError("Failed to open the depayloader")
depayloader.Start()

frame = rtp_types.RtpFrameData()
if depayloader.Receive(frame, 1000):       # True when a frame arrived within the timeout (ms)
    pixels = frame.tobytes()               # Copy of the pixels, valid after the next Receive()
    print(frame.resolution.width, frame.resolution.height, len(pixels))

depayloader.Stop()
depayloader.Close()

With numpy:

image = np.frombuffer(frame.tobytes(), dtype=np.uint8).reshape((frame.resolution.height, frame.resolution.width, 3))

tobytes() raises ValueError if no frame has been received. To find a stream to receive, use the SAP listener below.

Compressed Video (H.264, H.265, AV1)

Each codec module holds the payloader (encoder and RTP sender) and the depayloader (RTP receiver and decoder) of one backend. They are used exactly like the uncompressed classes: Transmit() takes an RGB24 frame and Receive() returns decoded RGB24, the codec work happens inside.

Codec Module Backend Stream encoding
H.264 rtp_h264_openh264 OpenH264, software ColourspaceType_kColourspaceH264Part10
H.264 rtp_h264_vaapi Intel/AMD VAAPI ColourspaceType_kColourspaceH264Part10
H.265 rtp_h265_vaapi Intel/AMD VAAPI ColourspaceType_kColourspaceH265
AV1 rtp_av1_vaapi Intel/AMD VAAPI ColourspaceType_kColourspaceAv1

The H.264 NVENC, Jetson, V4L2 M2M and Android backends and the Jetson AV1 decoder have no Python module. A module is only present when its backend was built into MediaX (BUILD_OPENH264=ON for OpenH264, libva for VAAPI), so from mediax import rtp_h265_vaapi raises ImportError otherwise.

Transmit and receive

from mediax import rtp_types
from mediax import rtp_h264_vaapi

stream_info = rtp_types.StreamInformation()
stream_info.session_name = "h264-stream"
stream_info.hostname = "239.192.1.1"
stream_info.port = 5004
stream_info.width = 1280
stream_info.height = 720
stream_info.framerate = 30
stream_info.encoding = rtp_types.ColourspaceType_kColourspaceH264Part10

if not rtp_h264_vaapi.RtpH264VaapiPayloader.IsVaapiAvailable():
    raise RuntimeError("no VAAPI device")

# Receiver first, so no frame is missed
rx = rtp_h264_vaapi.RtpH264VaapiDepayloader()
rx.SetStreamInfo(stream_info)
rx.Open()
rx.Start()

tx = rtp_h264_vaapi.RtpH264VaapiPayloader()
tx.SetStreamInfo(stream_info)
tx.Open()
tx.Start()

tx.Transmit(rgb_frame)                     # bytes, bytearray, memoryview or numpy array, 1280 * 720 * 3 bytes

frame = rtp_types.RtpFrameData()
if rx.Receive(frame, 1000):                # True when a decoded frame arrived within the timeout (ms)
    pixels = frame.tobytes()               # RGB24

tx.Stop()
tx.Close()
rx.Stop()
rx.Close()

The other codecs differ only in the module and class names: rtp_h264_openh264.RtpH264OpenH264Payloader, rtp_h265_vaapi.RtpH265VaapiPayloader, rtp_av1_vaapi.RtpAv1VaapiPayloader, and the matching ...Depayloader classes. IsVaapiAvailable() is a static method on the VAAPI classes. The frame buffer must hold at least width * height * 3 bytes, otherwise Transmit() raises ValueError.

Encoder and decoder configuration

The size and frame rate come from SetStreamInfo(). The rest of the encoder is tuned with a configuration object:

Codec Encoder configuration Decoder configuration
OpenH264 EncoderConfig: bitrate, keyframe_interval, max_nal_size, enable_denoise, complexity DecoderConfig: width, height, error_concealment
H.264 VAAPI VaapiEncoderConfig: bitrate, keyframe_interval, profile (H264Profile_*), level (H264Level_*), qp, enable_cabac, num_ref_frames, max_slice_size VaapiDecoderConfig: width, height, error_concealment, max_ref_frames
H.265 VAAPI VaapiH265EncoderConfig: bitrate, keyframe_interval, profile (H265Profile_*), level (H265Level_*), qp, num_ref_frames, ctu_size VaapiH265DecoderConfig
AV1 VAAPI VaapiAv1EncoderConfig: bitrate, keyframe_interval, profile (Av1Profile_*), qp, tier, level_idx VaapiAv1DecoderConfig 
config = rtp_h264_vaapi.VaapiEncoderConfig()
config.width = 1280                        # The configuration also carries the frame size and rate
config.height = 720
config.framerate = 30
config.bitrate = 4000000
config.profile = rtp_h264_vaapi.H264Profile_kMain
config.keyframe_interval = 30

tx.SetStreamInfo(stream_info)
tx.SetEncoderConfig(config)                # Call before Open(). The last of SetStreamInfo() and SetEncoderConfig() sets the size
tx.Open()

SetEncoderConfig() replaces the whole configuration, including the frame size and frame rate, and the stream takes the configured size, so set config.width, config.height and config.framerate as well. SetDecoderConfig() on the depayloader takes the decoder configuration. Other members: SetMtu() and GetMtu(), ForceIdr() (ForceKeyframe() for AV1) to make the next frame a key frame, GetBytesSent() and GetFrameCount() on the payloader, and GetBytesReceived(), GetDecodedWidth() and GetDecodedHeight() on the depayloader.

RTSP Streaming

Built with -DBUILD_PYTHON=ON and BUILD_RTSP=ON. The rtsp_depayloader module wraps RtspDepayloader. Callbacks and factories cannot be expressed in Python, so frames are polled with Receive().

from mediax import rtsp_depayloader as rtsp

config = rtsp.RtspClientConfig()
config.transport = rtsp.RtspTransportMode_kTcp   # kAuto, kUdp, kMulticast or kTcp
config.auto_reconnect = True

rx = rtsp.RtspDepayloader()
rx.SetClientConfig(config)
rx.SetUrl("rtsp://camera/stream")          # returns RtspStatus_kOk (0) on success
rx.SetCredentials("user", "password")

if rx.Open():                              # returns bool
    rx.Start()

    frame = rtsp.RtpFrameData()
    if rx.Receive(frame, 5000):            # True when a frame arrived within the timeout (ms)
        print(frame.resolution.width, frame.resolution.height)

    stats = rx.GetStatistics()
    print(stats.reconnects, stats.rtcp_reports_sent)
    rx.Stop()
    rx.Close()
else:
    print(rtsp.RtspStatusToString(rx.GetLastStatus()), rx.GetLastError())
Member Description
SetUrl(url), SetCredentials(user, password) Stream URL and optional credentials
SetClientConfig(RtspClientConfig) transport, keepalive, auto_reconnect, max_reconnect_attempts, connect_timeout_ms, response_timeout_ms, media_timeout_ms, enable_rtcp, rtcp_interval_ms, rtp_port_min/rtp_port_max, socket_buffer_bytes, user_agent
Open(), Start(), Stop(), Close() Lifecycle
Receive(RtpFrameData, timeout_ms) Poll for a decoded frame
GetLastStatus(), GetLastError() RtspStatus_* code and text. RtspStatusToString() converts the code
GetStatistics() RtspStatistics: reception, reconnects, rtcp_reports_sent
GetTracks(), GetTrack(i) RtspTrackInfo: index, media_type, control_url, encoding, encoding_name, width, height, framerate, multicast_address, multicast_port, multicast_ttl

RtpFrameData has resolution (width, height), encoding, cpu_buffer (a raw pointer) and tobytes(), which returns a copy of the pixels (RGB24 for decoded video). See the RTSP Receiver guide.

SAP/SDP

SapAnnouncer and SapListener are singletons. They have no constructor in Python, get the shared object with GetInstance().

Announcer

import time
from mediax import rtp_types
from mediax import sap_announcer

stream_info = rtp_types.StreamInformation()
stream_info.session_name = "announced-stream"
stream_info.hostname = "239.192.1.1"
stream_info.port = 5004
stream_info.width = 1920
stream_info.height = 1080
stream_info.framerate = 30
stream_info.encoding = rtp_types.ColourspaceType_kColourspaceRgb24

announcer = sap_announcer.SapAnnouncer.GetInstance()
announcer.AddSapAnnouncement(stream_info)
announcer.Start()

time.sleep(60)          # The announcer runs in a background thread

announcer.Stop()
announcer.DeleteAllSapAnnouncements()
Method Description
AddSapAnnouncement(StreamInformation) Add a stream to announce
DeleteSapAnnouncement(name), UndeleteSapAnnouncement(name) Withdraw or restore one announcement by session name
DeleteAllSapAnnouncements() Withdraw every announcement
Start(), Stop(), Restart(), Active() Control and query the announcer
GetActiveStreamCount() Number of active announcements
GetStreams() StreamInformationVector of the announced streams (len(), indexing, iteration)
SetSourceInterface(index) Choose the network interface
GetInterfaces() MapUint32String, interface index to name (items())

AddSapAnnouncementRaw(name, sdp_text) and DeleteSapAnnouncementRaw(name) announce a hand-written SDP.

Listener

import time
from mediax import sap_listener

listener = sap_listener.SapListener.GetInstance()
listener.Start()

time.sleep(2)           # Wait for announcements

for name, sdp in listener.GetSapAnnouncements().items():
    print(f"Stream: {name}")
    print(f"  Address: {sdp.ip_address}:{sdp.port}")
    print(f"  Resolution: {sdp.width}x{sdp.height}")
    print(f"  Framerate: {sdp.framerate}")

listener.Stop()

GetSapAnnouncements() returns a SapAnnouncementsMap (a std::map proxy supporting len(), items() and find()) of SdpMessage. SdpMessage fields: session_name, ip_address, ip_address_source, port, width, height, framerate, sampling, bits_per_pixel, protocol_version, hash, deleted and sdp_text. GetStreamInformation(name, StreamInformation) fills the given object and returns True if the stream is known. RegisterSapListener() and GetSapAnnouncement() (callbacks and out-pointers) are not available from Python, use GetSapAnnouncements().

Pixel Formats and NumPy

Frame buffers are packed, with no row padding. Sizes in bytes are width * height * bytes_per_pixel:

Encoding Bytes per pixel NumPy shape
kColourspaceRgb24 3 (height, width, 3), uint8
kColourspaceYuv422 (UYVY) 2 (height, width, 2), uint8
kColourspaceMono8 1 (height, width), uint8
kColourspaceMono16 2 (height, width), uint16

These shapes apply to Transmit() input and RtpFrameData.tobytes() output. Reshape with numpy, for example np.frombuffer(frame.tobytes(), dtype=np.uint8).reshape((height, width, 3)).

Error Handling

Most calls report failure through return values rather than exceptions:

Call Success
RtpUncompressedPayloader.Open(), RtspDepayloader.Open(), Receive() True
RtpUncompressedPayloader.Transmit() rtp_types.StatusCode_kStatusOk (0)
RtspDepayloader.SetUrl() rtsp.RtspStatus_kOk (0) 
rx = rtsp.RtspDepayloader()
status = rx.SetUrl("rtsp://camera/stream")
if status != rtsp.RtspStatus_kOk:
    raise ValueError(rtsp.RtspStatusToString(status))

if not rx.Open():
    raise ConnectionError(f"{rtsp.RtspStatusToString(rx.GetLastStatus())}: {rx.GetLastError()}")

Argument problems raise exceptions: TypeError for the wrong type (for example an int where a str is expected, or str data passed to Transmit()), ValueError for a frame buffer that is too small or an empty RtpFrameData, and RuntimeError for a C++ exception.

Complete Example

Receive an RTSP stream and count frames:

#!/usr/bin/env python3
import sys
import time

sys.path.append('/usr/lib/python3/dist-packages')
sys.path.append('/usr/local/lib/python3/dist-packages')
from mediax import rtsp_depayloader as rtsp


def main(url):
    rx = rtsp.RtspDepayloader()
    status = rx.SetUrl(url)
    if status != rtsp.RtspStatus_kOk:
        print(rtsp.RtspStatusToString(status))
        return 1

    if not rx.Open():
        print(rtsp.RtspStatusToString(rx.GetLastStatus()), rx.GetLastError())
        return 1
    rx.Start()

    frame = rtsp.RtpFrameData()
    frames = 0
    start = time.time()
    while time.time() - start < 10:
        if rx.Receive(frame, 1000):
            frames += 1
            if frames == 1:
                print(f"First frame {frame.resolution.width}x{frame.resolution.height}")

    print(f"{frames} frames, {rx.GetStatistics().reconnects} reconnects")
    rx.Stop()
    rx.Close()
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "rtsp://camera/stream"))

The installed examples (rtp_transmit_example.py, rtp_receive_example.py, sap_announcer_example.py and sap_listener_example.py) are in /usr/share/doc/mediax-python/examples/.

Support

Edit this page on GitHub