Sonification
https://pad.pzimediadesign.nl/p/ounupo-29012018
Sound and Listening
What is sound
We experience sound thought displacement of air molecules from a source to a the receiver.
When a source emits a sound, it makes the air molecules compress and decompress, traveling away from the source.
The rate of this compression-decompression variation is what is called the frequency of a sound. And the amount of air molecules displace give us the amplitude or volume of a sound.
Sound wave properties
- frequency = pitch
- measured in Hertz (Hz) or cycle per second. 1Hz = 1cycle/sec
- amplitude = volume
- measured in decibel (db)
Sound waves on liquid
audible sound
Human audible sound, or the hearing spectrum, is located between 20Hz and 20 000Hz of of the acoustic spectrum
How do we listen?
How does the movement of air molecules gets perceived as sound?
Spacial Perception Sound
How do we detect the position where sound is coming from?
- arrival time difference
- amplitude difference
- filtering, by the outer ear
Janet Cardiff - binaural mics
Hearing as composing
Recording and listening exercise.
- record 1 to 2 minutes of sound
- write down all you here during those minutes, either with or without headphones
4'33
4'33 (1952) is a probably the most famous John Cage composition, where the performance stands in front of the audience without playing a single note, for 4 minutes and 33 seconds.
http://hyperallergic.com/85779/the-original-john-cage-433-in-proportional-notation-19521953/
- hearing as composing
- all sound can become music
Expanded listening
contact mics
- listen to how solid objects respond to vibration
- due to its physical properties each object will respond - resonate - differently to vibration
- The resonant frequencies are the frequencies that are emphasized by the object
David Tudor - Rainforest (1968)
- http://composers-inside-electronics.net/rainforest/rainforest/RAINFOREST.html
- https://www.youtube.com/embed/iT9HP48sHTg
- http://www.getty.edu/research/tools/guides_bibliographies/david_tudor/av/rainforest.html
- Holzer, Derek, and Mads BEch Paulszewski. n.d. “Rainforest 2012.” http://macumbista.net/files/rainforest_overview.pdf
Electromagnetic pickups
Instead of capturing acoustic waves, like acoustic microphones, electromagnetic pickups capture electromagnetic fields.
Christina Kubisch Electrical Walks explore cities electromagnetic fields.
- http://www.christinakubisch.de/en/works/electrical_walks
- http://cabinetmagazine.org/issues/21/cox.php
- https://www.youtube.com/watch?v=5tCphr8pbFk
Digital audio
Analog audio is stored in a continuous analogue to its source.
http://www.synthgear.com/2014/audio-gear/record-grooves-electron-microscope
More grooves: http://sessionville.com/articles/vinyl-through-the-microscope-looking-glass
Digital audio on the other hand is stored discretely (in segments or samples).
sampling rate and bit rate
Two parameters are essential to digital audio:sampling rate and bit rate
sampling rate: how frequently the analogue signal is sampled File:Discrete-signal.png
- The sampling rate of CDs is 44.1KHz: discrete samples of a signal are stored 44100 times each second.
- Digital video cameras tend to store audio at 48KHz.
bit rate: accuracy of sampling - how many values are used to represent the amplitude of a signal
- The bit rate of a CD is 16-bit: 2¹⁶ (65536) values are used to encode the audio digitally
- 8 bit 2⁸ (256) values
audio formats
- uncompressed: aiff, wav, flac
- compressed: mp3, ogg
Editing
With Audacity http://www.audacityteam.org/
“Audacity.” 2017. FLOSS Manuals. Accessed January 11. http://write.flossmanuals.net/audacity. http://archive.flossmanuals.net/audacity/_booki/audacity/audacity.pdf http://archive.flossmanuals.net/audacity/_booki/audacity/audacity.epub
Bibliography
- Collins, Nicolas. 2006. Handmade Electronic Music. London: Routledge.
- “Sound:An Interactive eBook on the Physics of Sound.” 2017. Sound: An Interactive eBook on the Physics of Sound. Accessed January 11. https://soundphysics.ius.edu/.
♣
BitMaps
bitmap images
You probably have heard before that bitmap images are grids of pixels, where
- resolution: the number of horizontal and vertical pixels. E.g. 1920x1080 is 1920 pixels wide and 1080 high
- color depth: the number of bits used to indicate the color of a single pixel
for a good introduction to bit maps see: See Bourke, Paul. n.d. ‘A Beginners Guide to Bitmaps’. Accessed 2 January 2018. http://paulbourke.net/dataformats/bitmaps/.
Color depth
- 1-bit color (21 = 2 colors): monochrome, often black and white, compact Macintoshes, Atari ST.
- 2-bit color (22 = 4 colors): CGA, gray-scale early NeXTstation, color Macintoshes, Atari ST.
- 3-bit color (23 = 8 colors): many early home computers with TV displays, including the ZX Spectrum and BBC Micro
- 4-bit color (24 = 16 colors): as used by EGA and by the least common denominator VGA standard at higher resolution, color Macintoshes, Atari ST, Commodore 64, Amstrad CPC.
- 5-bit color (25 = 32 colors): Original Amiga chipset
- 6-bit color (26 = 64 colors): Original Amiga chipset
- 8-bit color (28 = 256 colors): most early color Unix workstations, VGA at low resolution, Super VGA, color Macintoshes, Atari TT, Amiga AGA chipset, Falcon030, Acorn Archimedes.
- 12-bit color (212 = 4096 colors): some Silicon Graphics systems, Color NeXTstation systems, and Amiga systems in HAM mode.
- 8-bit color: A very limited but true direct color system, there are 3 bits (8 possible levels) for each of the R and G components, and the two remaining bits in the byte pixel to the B component (four levels), enabling 256 (8 × 8 × 4) different colors.
- Grey scale (8-bit)
- True color (24-bit): Usually, true color is defined to mean 256 shades of red, green, and blue, for a total of 224, or alternately 2563, or 16,777,216 color variations. The human eye can discriminate up to ten million colors
converting resolution
with Imagemagick
to 100px wide
convert myimage.jpg -resize 100x output.jpg
to 100px hight
convert myimage.jpg -resize x100 output.jpg
to 100x100px (image will loose its proportion; hence the "!" so imagemagick is sure that you wanna do that:)
convert myimage.jpg -resize 100x100! output.jpg
converting color (bit depth)
convert an image into monochrome (1 bit depth) image: 2color values (b/w)
convert myimage.jpg -colorspace gray -depth 1 out.png
(2 bit depth) image: 4 color values
convert myimage.jpg -colorspace gray -depth 2 out.png
(3 bit depth) image: 8 color values
convert myimage.jpg -colorspace gray -depth 3 out.png
(4 bit depth) image: 16 color values
convert myimage.jpg -colorspace gray -depth 4 out.png
(8 bit depth) image: 256 color values
convert myimage.jpg -colorspace gray -depth 4 out.png
How can we be certain about this? How do can we see the grid that forms a bitmap image?
- magnify any digital screen
- a Plain-text bitmap: where you can read each pixel: X PixMap (XPM)
X PixMap (XPM) is a plain-text bitmap image format, where each color is assigned a character, which is mapped onto the text space.
convert myimage.jpg out.xpm
/* XPM */
static char *output[] = {
/* columns rows colors chars-per-pixel */
"50 73 2 1 ",
" c black",
". c white",
/* pixels */
" ",
" ",
" ",
" ",
" ................ ",
" ...... . .. . . ....... ",
" .. . . ... ",
" . . .. . . .. ",
" .. . . . . . . . ",
" . . . . . . . .. ",
" . . . . . . . ",
" ... .. . .. ",
" ... .. . ... ",
" .. . ",
" . .. ",
" .. . ",
" . .. ",
" ... . . ",
" . . . ",
" .. . ",
" .. . . . . ",
" . . . ",
" .. . ..... . ",
" . .. .. . ",
" . . .. . . ",
" . .. . .. ",
" .. . . . ..... ",
" . ... . . . ",
" . . .... .. . .. ",
" . .. .... . . ",
" . . . . .... ",
" . . . .. ... ",
" . . . .. ",
" .. . . . . ",
" . . .. . . ",
" . .. .. .. .. ",
" .. ... ... . ",
" .. .. ",
" . . . . . ",
" . . .. . ",
" . . .... . . ",
" . .. ... . .. .. ",
" .. . .. ..... . .. . ... . ",
" . ... .. .... .... . ",
" . ..... ...... ...... . ",
" .. . ..... . ......... ... . ",
" .. ......... .. . ...... . ",
" . .. .................... ",
" . .. . ............... . ",
" . .. .. ............. . ",
" . .. . . ..... . ",
" ... . ... . .. ...... . ",
" ... . ............. . ",
" .. . .. . ",
" ... .. ",
" ... . . .. ",
" .. . . ",
" .. . ",
" .... .. ",
" ..... .... ",
" ..... ",
" ",
" . ",
" .. . ... ",
" ... .. .. ... .. .. ...... ... ",
" ............................ .. ",
" ...... ................. ... . ",
" ...... ... ... ..... .... .. . ",
" .. ",
" . ",
" ",
" ",
" "
};
♣
Raw audio
RAW Audio is an audio file format for storing uncompressed audio in raw form. Comparable to WAV or AIFF in size, RAW Audio file does not include any header information (sampling rate, bit depth, endian, or number of channels)
convert an mp3 to raw
Using
- ffmpeg: " A complete, cross-platform solution to record, convert and stream audio and video."
- Sox: the Swiss Army knife of sound processing programs
Find out the properties of the audio file with ffmpeg
ffmpeg -i file.mp3 sox -V file.wav
Sox has no handler for mp3 formate, hence you have need to convert from mp3 to wav, ffmpeg does it quickly and well
ffmpeg -i file.mp3 -acodec pcm_s16le -ar 44100 file.wav
-acodec pcm_s16le: audio codec is PCM signed 16bit-ar 44100: audio rate 44.1kHz
Sox: display information on the wav file
sox -V file.wav
Sox: convert the wav file to raw
sox file.wav -b 16 -c 1 -r 44100 -t raw file.raw
-b 16bit-rate-c 1number of channels-r 44100sampling rate-t rawtarget raw
play (sox): play back the raw file at same bit-rate and sample-rate
play -b 16 --endian little -e signed -r 44100 -c 1 Zong3.raw
--endian little: "specify whether the byte-order of the audio data is, respectively, `little endian'... Endianness applies only to data encoded as floating-point, or as signed or unsigned integers of 16 or more bits. It is often necessary to specify one of these options for headerless files"-e signedencoding. Signed is commonly used with a 16 or 24 -bit encoding size.
Same thing but with aplay:
aplay -f S16_LE -c 1 -r 44100 Zong3.raw
-fsample forma+bit rate+endian:- 8bit: S8 U8
- 16bit: S16_LE S16_BE U16_LE U16_BE
- 24bit: S24_LE S24_BE U24_LE U24_BE
- 32bit: S32_LE S32_BE U32_LE U32_BE
-c: channel number-r: sample rate. 4000 Hz is the minimum allowed by aplay. 44100Hz is the default sample rate for CDs
convert image to raw audio
copy the image to another file, with the extension raw
cp img.png img.raw
play: use sox previously use command to play that audio
play -b 16 --endian big -e signed -r 44100 -c 1 img.raw
- lowering the sampling rate
-rwill slow-down the playback
Save the result of playing the raw file into audio file with a head (wav)
sox -r 44100 -b 16 -c 1 -e signed img.raw img.wav
Notice that the same audio parameters need to be declared: sample-rate, bit-rate, channel number, encoding
play: without having to declare any parameters
play img.wav
Links
- Raw Audio File Formats Information (at the Wayback Machine)
- SOX Conversions, Raw Files, Splitting And Merging Channels
- Audio Conversion Cheat Sheet
- [SoX] in the Cookbook
♣
Raw images
Raw: This is the simplest of all ways to store images, just as "raw" bytes. For example one byte per pixel for grey scale or 3 bytes per pixel for RGB colour. There is no standard header and so even the size of the image needs to be specified for programs that might read the image. [2]
Imagemagick can convert a bit-map file to a raw .dat – a not specific file type
convert image.png image.dat
And, it can also convert the .dat back into a bit-map
Andre Castro (talk) 10:14, 29 January 2018 (CET) if you run:
identify image.dat img.dat PNG 2102x2799 2102x2799+0+0 8-bit RGB 256c 33.6KB 0.000u 0:00.000
Hence the image still with PNG headers
If you stick to the size, bit-depth and color profile of the original, the conversion will "preserve" the original
convert -depth 8 -size 2102x2799 rgb:image.dat image2.png
If you change those parameters, it is likely that the resulting bitmap will be something else!
convert -depth 8 -size 1000x1000 rgb:image.dat image2.png
convert -depth 24 -size 1000x1000 rgb:image.dat image2.png
convert -depth 8 -size 1000x1000 gray:image.dat image2.png
What we are doing here is, not unlike playing back audio files at bit-rate and sample-rates, different from the ones they were encoded in.
But we can convert from a raw audio to an image file with
convert -depth 8 -size 1000x1000 rgb:Zong3.raw Zong3.jpg
or even an animated gif
convert -depth 8 -size 1000x1000 rgb:Zong3.raw Zong3.jpg
What gets lost in a conversion ??
...
code experiments with Raw images
With a small amount of code, it's easy to dump out a stream of data as bytes:
# raw.py
import struct, sys
out = open("image.data", "wb")
for x in range(100):
out.write(struct.pack('B', 255))
out.write(struct.pack('B', 0))
out.write(struct.pack('B', 0))
out.write(struct.pack('B', 128))
The "wb" option in the open command means that we want to "write" "binary" data (the default behaviour of read would be to read a text file).
When you run the script:
python raw.py
In Python, struct.pack is a way of converting a number (from 0 to 255) into it's corresponding binary representation as a "byte" or 8 bits (where 0 is all bits off 00000000 and 255 is all bits on 11111111). The code above loops 100 times outputting the bits of the sequence 255, 0, 0, 128.
255 0 0 128 255 0 0 128 255 0 0 128 255 0 0 128...
We can then use an image application like the GIMP to interpret this raw data as an image... In this case we tell the GIMP that it should interpret the bytes as being RGBA formatted pixels in the size 10 x 10:
It then interprets the numbers as a stream of pixels in the form:
RED GREEN BLUE ALPHA, RED GREEN BLUE ALPHA, ...
So for the code above:
255 (full) RED, 0 GREEN, 0 BLUE, 128 (half) ALPHA, ...
The result is an image 10 pixels wide by 10 pixels tall, where every pixel is red with 50% transparency.
Put a header on it
Working with raw data file can be inconvenient however since everytime you want to view the data as an image, you need to explicitly tell an application (such as GIMP) what the size and format of the image is. We can improve the situation by attaching preceding the raw data with a simple "header" to and follow the guidelines (which specify the order of the bytes) of a specific simple image format.
Targa is an early very simple format for images. It comes from an early manufacturer of video display cards, named Targa, who created a minimal format for files to display on their hardware. The format is still popular today in Game development and other communities for whom the simplicity of the format is useful for "wrapping" raw image data with a header so that it's self-contained and directly loadable by different programs without needing to explicitly specify information width and height and bit depth.
import struct, sys
out = open("image.tga", "wb")
width = 320
height = 240
header = struct.pack("<BBBHHBHHHHBB",0,0,2,0,0,8,0,0,width,height,32,1<<5)
out.write(header)
for y in xrange(height):
for x in xrange(width):
r = 0
g = 0
b = 0
a = 128
if y < 32:
r = 255
a = 255
if x > 64 and x < 256:
g = 255
if y > 120:
r = 128
out.write(struct.pack('B', b))
out.write(struct.pack('B', g))
out.write(struct.pack('B', r))
out.write(struct.pack('B', a))
This script outputs a TGA format, which has been opened in the GIMP and exported to PNG
Next: Raw image sequence
Resources
- This example is based on this Raw image example (in Portugese)
references
- ↑ https://en.wikipedia.org/wiki/Color_depth
- ↑ Bourke, Paul. n.d. ‘A Beginners Guide to Bitmaps’. Accessed 2 January 2018. http://paulbourke.net/dataformats/bitmaps/.
♣