GStreamer

Programmable Media Pipelines

GStreamer allow you to work with media (audio/video) using pipelines. Just like standard UNIX pipeline can be used to piece different software together to manipulate / filter / transform (mainly) text, GStreamer's pipelines are designed to for instance: pull data from a camera, filter video frames (to say overlay some text), transcode formats, and ultimately either display on screen, save to a file, or stream over a network.

GStreamer provides a programming interface (bindings) in Python via PyGST. This means you can use Python to dynamically manipulate media pipelines, and make connections to other kinds of information systems (internet, email, chat, databases, etc).

GStreamer is cross-platform, with versions for Mac OSX and Windows. (don't know how capable / stable these are though)

GStreamer is part of the Gnome desktop project, but is used by apps using any window manager. GStreamer is used as the "backend" in many popular linux applications. For instance: the Totem and Kaffeine movie players, Rhythymbox and amarok music players, and Gnash (an open source flash plugin).

Playing with GStreamer on the Command line

gst-launch

Use can test out GStreamer media pipelines via some commandline programs: gst-launch, gst-inspect. Note that in some installs of the tools, the programs may end with a version number (as in gst-launch-0.10). The examples shown here are all using version 0.10 of GStreamer.

The "hello world" of audio / video pipelines use the "videotestsrc" and "audiotestsrc" elements:

gst-launch-0.10 videotestsrc ! ximagesink

gst-launch-0.10 audiotestsrc ! alsasink

Note that GStreamer pipelines use the exclamation mark ( ! ) as their "pipe" character. If you're used to regular UNIX pipelines you will (at least once) make the mistake of typing a regular pipe ( | ) which of course results in an error as the shell tries to interpret what you've written.

Documentation of all available plugins is available online.

gst-inspect

gst-inspect is like the "man" command of GStreamer elements. Use gst-inspect alone to get a big list of all elements (plugins) available. gst-inspect <name of plugin> will display detailed info for a particular element.

gst-inspect-0.10 videotestsrc | less

gst-inspect without any option will dump a list of all available elements. Combine with grep to search for a particular element:

gst-inspect-0.10 | grep crop

On an elements inspect page, you can find out information about the available properties and their values. For instance, the change the video test pattern, you can use the 'pattern' property:

gst-launch-0.10 videotestsrc pattern=1 ! ximagesink

Webcams (v4lsrc)

Webcams cam generally be accessed via the video4linux system. GStreamer has two sources for connecting to these devices, v4lsrc, and v4l2src. Use the one that works with your camera.

Viewing a USB cam (you may need v4l2src):

gst-launch-0.10 v4lsrc ! video/x-raw-yuv,width=320,height=240 ! xvimagesink

Alternatively, you can use the normal (non-accelerated) ximagesink, though in this case it's necessary to add the "ffmpegcolorspace" element to help negotiate the correct format that ximagesink needs to display the video.

gst-launch-0.10 v4lsrc ! video/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! ximagesink

Make sure that plugging in your camera results in the creation of a "video" device (typically /dev/video0). If your camera has a different name (or for instance when using multiple cameras), set the device property of the v4lsrc element:

gst-launch-0.10 v4lsrc device=/dev/video1 ! video/x-raw-yuv,width=320,height=240 ! xvimagesink

Saving to file (OGG format, video only):

gst-launch-0.10 v4lsrc ! video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! theoraenc ! oggmux ! filesink location=foo.ogg

Saving to file (OGG format, audio & video): This example uses several tricks to adjust the rates, and queue to improve performance (and hopefully not lose any data).

gst-launch-0.10 v4lsrc ! video/x-raw-rgb,width=320,height=240 ! queue ! videorate ! video/x-raw-rgb,rate=15/1 ! ffmpegcolorspace ! theoraenc ! oggmux name=mux ! filesink location=foo.ogg alsasrc ! audio/x-raw-int,channels=1,rate=22025,depth=16 ! queue ! audioconvert ! vorbisenc ! mux.

Taking a picture (example from OLPC wiki)

gst-launch-0.10 v4l2src ! ffmpegcolorspace ! pngenc ! filesink location=foo.png

building bridges

Common "bridge" elements are: audioconvert, audioresample, ffmpegcolorspace or videoscale. Often a non-functional pipeline can be fixed by plugging in the right bridge element between two conflicting pieces. Once a pipeline is working, it may be a good idea though to revist your pipeline and attempt to remove any unnecessary elements as these may slow down the processing.

capsfilter

The second element in the two pipelines above ("video/x-raw-yuv,width=320,height=240") is a special syntax that can be used in GStreamer pipeline descriptions. It sets a mime type to specify a desired video format (in this case rgb color), along with (optionally) information like frame size and rate.

These get translated into gstreamer "capsfilter" elements that then enforce a particular format of data. In this case we request a particular framesize (as the default is typically smaller). The size requested may be limited by the capabilities of your camera (so if one size fails, try another -- 320 by 240 should be relatively safe).

Screencasts with Istanbul

Installing the Istanbul adds a special element designed to make screencasts (movies of your screen contents). (Note: built into gstreamer is the ximagesrc element; istximagesrc seems to be further developed for screencasting -- though honestly I haven't tested ximagesrc enough to know of it's shortcomings).

using istanbul's istximagesrc GStreamer module to create a screen cast (with audio):

gst-launch-0.10 oggmux name=mux ! filesink location=foo.ogg istximagesrc name=videosource use-damage=false endx=320 endy=240 ! video/x-raw-rgb,framerate=10/1 ! videorate ! ffmpegcolorspace ! videoscale method=1 ! video/x-raw-yuv,width=512,height=384,framerate=10/1 ! theoraenc ! queue ! mux. gconfaudiosrc name=audiosource ! audioconvert ! vorbisenc ! queue ! mux.

using istanbul's istximagesrc GStreamer module (& shout2send) to do a live screencast:

gst-launch-0.10 oggmux name=mux ! shout2send password=mlmpass mount=live istximagesrc name=videosource use-damage=false endx=320 endy=240 ! video/x-raw-rgb,framerate=10/1 ! videorate ! ffmpegcolorspace ! videoscale method=1 ! video/x-raw-yuv,width=512,height=384,framerate=10/1 ! theoraenc ! queue ! mux. gconfaudiosrc name=audiosource ! audioconvert ! vorbisenc ! queue ! mux.

Network Streaming with Icecast

The shout2send element allows sending the output of a audio/video stream to an icecast server.

Send a test image to a local icecast (change 'hackme' to your own icecast password):

gst-launch-0.10 videotestsrc ! video/x-raw-yuv,width=320,height=240 ! theoraenc ! oggmux ! shout2send password=hackme mount=live

Test it with:

vlc http://localhost:8000/live

Programming GStreamer with Python

• PythonGStreamer

Related pages on this wiki

• GStreamerPlugins
• ListeningAndWatching

Related / Comparable Projects

• MLT

GStreamer Resources Online

• GStreamer Website
• PyGST Tutorial

• GStreamer & Python Tutorial
• GStreamer on OLPC Wiki
• barcamp presentation with cool examples