ListeningAndWatching

listening and watching

The idea here is to look into how the sound and image/video capabilities of the computer could be used as sensors. In particular, we extend the idea of command line programs and the Linux/UNIX pipeline as a way of making novel connections between different software.

Listening with GStreamer

["GStreamer"] provides a number of modules interesting for both input and output of audio & video. For instance, the "level" module is great for getting at the relative "loudness" of a digital recording.

A callback function receives messages related to the level (relative volume) of the audio signal. Using a low and high variable, translates the raw numbers to a relative value (from 0 being at the low, to 1 being the high).

import gst, pygst, struct

(pmin, pmax) = (None, None)

def listener_on_message (bus, message, data):
	global pmin, pmax
	s = message.structure
	if s and s.get_name() == "level":
		rms = s['rms']
		# also available, but not used here
		# peak = s['peak']; decay = s['decay']
		
		# numbers are in pairs (left, right) -- we use just the first one (left?)
		p = rms[0]
		
		# check against/update min and max
		if (pmin==None or p<pmin): pmin = p
		if (pmax==None or p>pmax): pmax = p
		print "level", p, pmin, pmax

	return True

listener_desc = 'alsasrc ! level ! fakesink'
listener = gst.parse_launch(listener_desc)
listener.get_bus().add_watch(listener_on_message, None)

import gobject
mainloop = gobject.[[MainLoop]]()
listener.set_state(gst.STATE_PLAYING)

try:
	mainloop.run()
except: # an interruption from Ctrl-C
	print "stopping"

listener.set_state(gst.STATE_NULL)

Displaying the result

The GStreamerWindow is a class that makes it easy to create a window displaying the output of a GStreamer pipeline (a pipeline ending in an x(v)imagesink). Using this class, the 'f' key toggles a fullscreen mode, and pressing 'Esc' should quit the program.

attachment:GStreamerWindow.py

Using the GStreamerWindow:

from GStreamerWindow import GStreamerWindow
import gtk, pygst, gst

pipe_desc = "videotestsrc ! ximagesink"
pipeline = gst.parse_launch(pipe_desc)

win = GStreamerWindow()
win.set_pipeline(pipeline)
win.play()
win.show()
gtk.main()

Pipeline to display an ogg/theora video

Let's display an ogg-format video as an output. To begin, here's a basic gstreamer pipeline to display an ogg/theora video stream:

filesrc location=foo.ogg ! \
oggdemux ! \
theoradec ! \
x(v)imagesink

Watching with Motion

Motion