Stream hardware-encoded H.264 video from a Raspberry Pi to a web page using WebRTC

I use the wonderful 3D printing distribution OctoPi (which in turn uses OctoPrint) on a Raspberry Pi Zero to start and monitor my 3D prints. It has built-in webcam functionality, which means you can monitor your printer and see in the web interface if the print is going well or not - really nice! The problem with the webcam is that video is encoded to MJPEG (Motion JPEG), which can be displayed in the web interface <img> tag, but is a bandwidth hog. Streaming a 320x240 picture at 10fps generates about 800kB/s of traffic. The good thing is that the RPi hardware JPEG encoder is used, so CPU usage is very low (~2%). My 3D printer is in the basement and a WiFi connection from there is only possible using an external antenna and with only a really low bandwidth available.
Knowing the RPi has an H.264 hardware encoder I set out to find a solution to stream H.264 video onto the OctoPrint web interface. These instructions apply to a Debian Jessie based Raspbian / OctoPi 0.13.
But first things first:

Video / audio formats and container format

A generally misunderstood part of digital video is codecs and container formats. Video and audio codecs define how video or audio is encoded / compressed. These are formats like MPEG2, AVC/H.264, VP8/9, WMV9, etc. for video, and e.g. MP3 or AAC for audio and they generate raw data packets. These data packets are put (multiplexed or "muxed") into a container (together with other streams like subtitles etc.) to save it to a file or stream it via a network. These container formats are e.g. MPEG-TS, AVI, MP4, MKV, FLV etc.

Streaming formats and web browser video support

Streaming video systems are fragmented and often proprietary. There is streaming via HTML (a web browser requests a resource via a regular HTTP GET request), RTP/RSTP (older streaming format), RTMP (A proprietary format used by Flash), HSL (Newer streaming protocol not supported very well yet) and more. All of these system support different codecs internally.

The <video> element

Now the problem is that modern web browsers support the MP4 / H.264 video format, but the MP4 container format is as-it-is not suitable for streaming it to web browsers via the <video> element. Let that sink in - It is simply not possible. You need browser plugins or external programs. You can play a "finished" MP4 video file via the web browser, which will do a regular HTTP GET request, download, cache and play the file, but you can not STREAM every possible format.
The standard for the <video> element only defines "open" container formats to be supported in the browser: WEBM and OGG. It supports video/audio codecs like VP8, AVC/H.264, Theora, Vorbis.

Streaming Theora / OGG video to VLC with GStreamer

To stream video with with GStreamer first install the necessary packages with:

sudo apt-get install gstreamer1.0-omx gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad libgstreamer1.0-0 libgstreamer1.0-dev libgstreamer-plugins-base1.0-0 libgstreamer-plugins-base1.0-dev

gstreamer1.0-omx gives us the GStreamer filter omxh264enc which enables H.264 hardware-encoding. gstreamer1.0-tools provides gst-launch1.0 which is used to build a GStreamer pipeline and start playing / streaming it. You can now try to start streaming in Theora / OGG format:

gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw,width=320,height=240,framerate=10/1' ! theoraenc ! oggmux ! tcpserversink host=RPI_IP port=7272

Streaming Theora, 320x240@10fps, data rate ~150kb/s, 90-100% CPU

To stream in H.264 / FLV format, use:

gst-launch-1.0 v4l2src ! 'video/x-raw,width=320,height=240,framerate=10/1' ! omxh264enc target-bitrate=500000 control-rate=1 ! 'video/x-h264,profile=high' ! h264parse ! flvmux ! tcpserversink host=RPI_IP port=7272

Streaming H.264, 320x240@10fps, data rate ~360kb/s, ~20% CPU

You could even increase the compression in sacrifice of image quality, which will produce a data rate of ~200kb/s. Image quality is okish:

gst-launch-1.0 v4l2src ! 'video/x-raw,width=320,height=240,framerate=10/1' ! omxh264enc target-bitrate=100000 control-rate=1 ! 'video/x-h264,profile=high' ! h264parse ! flvmux ! tcpserversink host=RPI_IP port=7272

You can watch both streams in VLC when you open the address "tcp://RPI_IP:7272".
Note that the Theora stream creates a lower data rate stream, but a very high CPU usage. The video is very jerky, so I suspect it drops frames, because the CPU can not sustain encoding at 10fps, which in turn lowers the data rate...
The H.264-encoded video is smooth though and visually ok for the data rate.

Using WebRTC via Janus / Nginx

Now we know the RPi can hardware-encode and stream H.264 video. But how do we manage to view our video on a webpage?
The Firefox API page mentions RTP/RTSP as a source for the <video> tag, but I couldn't get that to work. After searching the internet for a while I ran across WebRTC, which is a realtime audio / video communication standard and seems to be supported in all major browsers. I decided to give it a try and followed these instructions and it worked:

Streaming H.264 using WebRTC

Note that you need to stop the haproxy and octoprint services before installing and starting nginx and janus, otherwise it won't work!

Using WebRTC in OctoPrint?

Ok. Now we know this works, but how can we integrate that into OctoPrint?! It is a bit unfortunate to use Janus and Nginx, another web server. OctoPrint already uses the Tornado web server, so we have to convert the example page from above to use that. I found this example on how to set up a WebRTC chat application and went from that...


To be continued...