ListeningAndWatching: Difference between revisions
Line 85: | Line 85: | ||
== Watching with Motion == | == Watching with Motion == | ||
[ | [[Motion]] |
Latest revision as of 11:02, 1 June 2009
listening and watching
The idea here is to look into how the sound and image/video capabilities of the computer could be used as sensors. In particular, we extend the idea of command line programs and the Linux/UNIX pipeline as a way of making novel connections between different software.
Listening with GStreamer
["GStreamer"] provides a number of modules interesting for both input and output of audio & video. For instance, the "level" module is great for getting at the relative "loudness" of a digital recording.
A callback function receives messages related to the level (relative volume) of the audio signal. Using a low and high variable, translates the raw numbers to a relative value (from 0 being at the low, to 1 being the high).
import gst, pygst, struct (pmin, pmax) = (None, None) def listener_on_message (bus, message, data): global pmin, pmax s = message.structure if s and s.get_name() == "level": rms = s['rms'] # also available, but not used here # peak = s['peak']; decay = s['decay'] # numbers are in pairs (left, right) -- we use just the first one (left?) p = rms[0] # check against/update min and max if (pmin==None or p<pmin): pmin = p if (pmax==None or p>pmax): pmax = p print "level", p, pmin, pmax return True listener_desc = 'alsasrc ! level ! fakesink' listener = gst.parse_launch(listener_desc) listener.get_bus().add_watch(listener_on_message, None) import gobject mainloop = gobject.[[MainLoop]]() listener.set_state(gst.STATE_PLAYING) try: mainloop.run() except: # an interruption from Ctrl-C print "stopping" listener.set_state(gst.STATE_NULL)
Displaying the result
The GStreamerWindow is a class that makes it easy to create a window displaying the output of a GStreamer pipeline (a pipeline ending in an x(v)imagesink). Using this class, the 'f' key toggles a fullscreen mode, and pressing 'Esc' should quit the program.
attachment:GStreamerWindow.py
Using the GStreamerWindow:
from GStreamerWindow import GStreamerWindow import gtk, pygst, gst pipe_desc = "videotestsrc ! ximagesink" pipeline = gst.parse_launch(pipe_desc) win = GStreamerWindow() win.set_pipeline(pipeline) win.play() win.show() gtk.main()
Pipeline to display an ogg/theora video
Let's display an ogg-format video as an output. To begin, here's a basic gstreamer pipeline to display an ogg/theora video stream:
filesrc location=foo.ogg ! \ oggdemux ! \ theoradec ! \ x(v)imagesink