Knowing the RPi has an H.264 hardware encoder I set out to find a solution to stream H.264 video onto the OctoPrint web interface. These instructions apply to a Debian Jessie based Raspbian / OctoPi 0.13.
But first things first:
Video / audio formats and container format
A generally misunderstood part of digital video is codecs and container formats. Video and audio codecs define how video or audio is encoded / compressed. These are formats like MPEG2, AVC/H.264, VP8/9, WMV9, etc. for video, and e.g. MP3 or AAC for audio and they generate raw data packets. These data packets are put (multiplexed or "muxed") into a container (together with other streams like subtitles etc.) to save it to a file or stream it via a network. These container formats are e.g. MPEG-TS, AVI, MP4, MKV, FLV etc.
Streaming formats and web browser video support
Streaming video systems are fragmented and often proprietary. There is streaming via HTML (a web browser requests a resource via a regular HTTP GET request), RTP/RSTP (older streaming format), RTMP (A proprietary format used by Flash), HSL (Newer streaming protocol not supported very well yet) and more. All of these system support different codecs internally.
The <video> element
Now the problem is that modern web browsers support the MP4 / H.264 video format, but the MP4 container format is as-it-is not suitable for streaming it to web browsers via the <video> element. Let that sink in - It is simply not possible. You need browser plugins or external programs. You can play a "finished" MP4 video file via the web browser, which will do a regular HTTP GET request, download, cache and play the file, but you can not STREAM every possible format.
The standard for the <video> element only defines "open" container formats to be supported in the browser: WEBM and OGG. It supports video/audio codecs like VP8, AVC/H.264, Theora, Vorbis.
Streaming Theora / OGG video to VLC with GStreamer
To stream video with with GStreamer first install the necessary packages with:
sudo apt-get install gstreamer1.0-omx gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad libgstreamer1.0-0 libgstreamer1.0-dev libgstreamer-plugins-base1.0-0 libgstreamer-plugins-base1.0-devgstreamer1.0-omx gives us the GStreamer filter omxh264enc which enables H.264 hardware-encoding. gstreamer1.0-tools provides gst-launch1.0 which is used to build a GStreamer pipeline and start playing / streaming it. You can now try to start streaming in Theora / OGG format:
gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw,width=320,height=240,framerate=10/1' ! theoraenc ! oggmux ! tcpserversink host=RPI_IP port=7272
|Streaming Theora, 320x240@10fps, data rate ~150kb/s, 90-100% CPU|
gst-launch-1.0 v4l2src ! 'video/x-raw,width=320,height=240,framerate=10/1' ! omxh264enc target-bitrate=500000 control-rate=1 ! 'video/x-h264,profile=high' ! h264parse ! flvmux ! tcpserversink host=RPI_IP port=7272
|Streaming H.264, 320x240@10fps, data rate ~360kb/s, ~20% CPU|
gst-launch-1.0 v4l2src ! 'video/x-raw,width=320,height=240,framerate=10/1' ! omxh264enc target-bitrate=100000 control-rate=1 ! 'video/x-h264,profile=high' ! h264parse ! flvmux ! tcpserversink host=RPI_IP port=7272
You can watch both streams in VLC when you open the address "tcp://RPI_IP:7272".
Note that the Theora stream creates a lower data rate stream, but a very high CPU usage. The video is very jerky, so I suspect it drops frames, because the CPU can not sustain encoding at 10fps, which in turn lowers the data rate...
The H.264-encoded video is smooth though and visually ok for the data rate.
Using WebRTC via Janus / Nginx
Now we know the RPi can hardware-encode and stream H.264 video. But how do we manage to view our video on a webpage?
The Firefox API page mentions RTP/RTSP as a source for the <video> tag, but I couldn't get that to work. After searching the internet for a while I ran across WebRTC, which is a realtime audio / video communication standard and seems to be supported in all major browsers. I decided to give it a try and followed these instructions and it worked:
|Streaming H.264 using WebRTC|
Using WebRTC in OctoPrint?
Ok. Now we know this works, but how can we integrate that into OctoPrint?! It is a bit unfortunate to use Janus and Nginx, another web server. OctoPrint already uses the Tornado web server, so we have to convert the example page from above to use that. I found this example on how to set up a WebRTC chat application and went from that...
To be continued...