User:Ssstephen/take-the-entrain

this room for sound residency will be an exploration of tying digital connections and using them as an instrument. connexion used to be spelled with an x which shows the origin of the word: a binding or joining together. opening a network connexion creates a string of data which can and will be plucked. a work in the form of a net: strings interwoven and interconnetted. starting from short strings inside a single computer, over the two weeks this network will expand to cover the entire universe.

This is a two week residency in the Willem de Kooning Academy's Room for Sound, from 16th to 27th January 2023. I will be experimenting with networked music, in particular I plan to use some ESP32 boards and a Raspberry Pi to send and receive RTP-MIDI over LAN and hopefully WAN, and to translate this MIDI data to and from audio (via microphones, sensors, buzzers, speakers, etc). I would like to get other people involved in this process as well as getting the machines to interact: the machines could be not just musicians but also instruments.

Pseudocode

Get the computers to entrain to eachother dynamically, clap with eachother

While
    Countup++ per ms
    If(hearbeep) or (countup= random ~ 200ms)
        If(countup >40ms)
            Wait(countup ms)
            Sendbeep
            Countup=0

Day 1

Get the computers to sing

Today I am working with two Seeed Studios Xiao ESP32-C3 and some buzzers. Writing Arduino (C++) scripts for them to make beeps and also blinkenlights because they're fun.

/**
 * Singing 
 *
 * Plays predefined tones in a predefined order from a buzzer on pin D9.
 * Also flashes an LED on pin D8 just for fun.
 */
#include "Arduino.h"
// #include "pitches.h"

// Define some pitches of beeps, a basic time unit, a status
int dt1=200;
int dt2=90;
int dt3=360;
int dt4=400;
int duration=50;
int ledstatus=0;

void beep(int tone1,int length) {
  //this part switches the LED status
  if(ledstatus==0){
    digitalWrite(D8,HIGH);
    ledstatus=1;
  }else{
    digitalWrite(D8,LOW);
    ledstatus=0;
  }
  //This part pulses the buzzer.
   for (int tonecounter=1;tonecounter<=length;tonecounter=tonecounter+1){
     analogWrite(A0,127);
     delayMicroseconds(tone1);
     analogWrite(A0,0);
     delayMicroseconds(tone1);
   }
  //Or you could for more exact tones and lengths but no analog
  //tone(D9, tone1, length);
}

void setup()
{
  // initialize LED digital pin as an output.
  pinMode(D8, OUTPUT);
  pinMode(A0, OUTPUT);
}

void loop()
{
  //A sequence of buzzes
  beep(dt1,duration*2);
  beep(dt2,duration);
  beep(dt1,duration*2);
  beep(dt2,duration*2);
  beep(dt1,duration);
  beep(dt2,duration*4);
  beep(dt1,duration*2);
  beep(dt2,duration);
  beep(dt1,duration*2);
  beep(dt3,duration);
  beep(dt1,duration*2);
  beep(dt4,duration*4);
}

Someone else sending data from arduino xto ChucK

Get the computers to clap

Some tweaks to that code (tweaks only below) gave me two buzzers that beep at predefined tempi (metronomes I guess). They can make nice polyrhythms together. Timing is tricky!

int notelength=20;
int bpm = 100;
unsigned long delay1=(60000000/bpm) - (1000 * notelength);

void clap(int tone1,int bpm) {
  tone(A0, tone1, notelength);
  delayMicroseconds(delay1);
}

void loop()
{
  //A clapping speed
  clap(dt3,180);
}

How do you embed audio on the wiki? Here's a link instead.

Day 2

Get the computers to listen

To sound or to digital signals? If they're digital I probably need them to listen more generally to wifi or bluetooth first. But I dont have any microphones for the C3's today so if its audio input I'll need to use another board.

The ESP32-A1S: Audio Inputs

How to work this board. First I am following the instructions here. Installing ESP-IDF and ESP-ADF. (IoT and Audio Development Frameworks). Actually I'm going to give up on this and use Arduino. I found a video tutorial and a library that look good. Hmm this board is actually quite tricky to get going. I have installed another library to work with this exact board but failing to upload after compile (A fatal error occurred: Could not open COM70, the port doesn't exist). I have been experimenting with the platformio.ini file below but still no luck. I think I'll give up on this part for today.

; PlatformIO Project Configuration File for arduino-audiokit-hal
[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino
lib_deps = https://github.com/pschatzmann/arduino-audiokit-hal

	c:\Users\user\Downloads\arduino-audio-tools-main.zip
lib_ldf_mode = deep+
build_flags = -DCORE_DEBUG_LEVEL=5 -DAUDIOKIT_BOARD=5 
monitor_speed = 115200

(Also this serial plotter looks cool)

Explore timbre

I didnt expect to be so interested in the tone of the buzzers but they are very cool. The digitalWrite function (and analogWrite, and delay) seem interesting to explore further to see what happens. Can you keep a constant pitch and change tone to explore what is possible? Two first attempts at playing with tone by varying the pulse length and gap:

void beepdown(int tone1,int length) {
  //This part pulses the buzzer.
  for (int tonecounter=1;tonecounter<=length;tonecounter=tonecounter+1){
    analogWrite(A0,250);
    delayMicroseconds(tone1*((length-tonecounter+1)/10));
    analogWrite(A0,0);
    delayMicroseconds(tone1*(tonecounter/10));
  }
}

void beepup(int tone1,int length) {
  //This part pulses the buzzer.
  for (int tonecounter=1;tonecounter<=length;tonecounter=tonecounter+1){
    analogWrite(A0,250);
    delayMicroseconds(tone1*(tonecounter/10));
    analogWrite(A0,0);
    delayMicroseconds(tone1*((length-tonecounter+1)/10));
  }
}

They came out very squealy and uncomfortable sounding in the recording, but interesting tones for sure.

Listen here

Day 3

ESP32C3 Listening

Ok back to the ESP32-C3, today I want to get WiFi or Bluetooth working on these, maybe sending MIDI data over these channels, bluetooth first because I'm scared of the network here (although the ESP32 could be a wifi access point which may be interesting too). I'm kinda following this tutorial. First I copied a bluetooth scanner that prints to the serial port. Success! Then a different script that sets it as a bluetooth server, and I managed to send some data (text) from my phone. Now I'll tweak this and see if the ESP will play a sound when my phone sends data. Yesssss it does indeed.

/**
 * Sing for me 
 *
 * Plays tones from a buzzer on pin A0 when bluetooth commands are received.
 * Also flashes an LED on pin D8 just for fun.
 */

#include <Arduino.h>
#include <BLEDevice.h>
#include <BLEUtils.h>
#include <BLEServer.h>
#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>
 
// See the following for generating UUIDs:
// https://www.uuidgenerator.net/
 
#define SERVICE_UUID        "4fafc201-1fb5-459e-8fcc-c5c9c331914b"
#define CHARACTERISTIC_UUID "beb5483e-36e1-4688-b7f5-ea07361b26a8"

// Define some pitches of beeps, a basic time unit, a status
int dt1=400;
int dt2=360;
int dt3=480;
int dt4=400;
int dt5=960;
int dt6=200;
int duration=50;
int ledstatus=0;
int notelength=14;
int bpm = 240;
unsigned long delay1=(60000000/bpm) - (1000 * notelength); 
 
 
// functions for making sounds

void ledSwitch(){
  //this part switches the LED status
  if(ledstatus==0){
    digitalWrite(D8,HIGH);
    ledstatus=1;
  }else{
    digitalWrite(D8,LOW);
    ledstatus=0;
  }
}

void beep(int tone1,int length) {
  ledSwitch();
  //This part pulses the buzzer.
  for (int tonecounter=1;tonecounter<=length;tonecounter=tonecounter+1){
    analogWrite(A0,250);
    delayMicroseconds(tone1);
    analogWrite(A0,0);
    delayMicroseconds(tone1);
  }
}

void clap(int tone1) {
  tone(A0, tone1, notelength);
  delayMicroseconds(delay1);
}

void beepsweep(int tone1,int length, int direction) {
  ledSwitch();
  //This part pulses the buzzer.
  for (int tonecounter=1;tonecounter<=length;tonecounter=tonecounter+1){
    analogWrite(A0,250);
    delayMicroseconds(tone1*(abs((1-direction)*length-tonecounter+1)/10));
    analogWrite(A0,0);
    delayMicroseconds(tone1*(tonecounter/10));
    delayMicroseconds(tone1*(abs(direction*length-tonecounter+1)/10));
  }
  ledSwitch();
}

// Class for BLE interaction

class MyCallbacks: public BLECharacteristicCallbacks {
    void onWrite(BLECharacteristic *pCharacteristic) {
      std::string value = pCharacteristic->getValue();
 
      if (value.length() > 0) {
        Serial.println("*********");
        for (int i = 0; i < value.length(); i++)
          Serial.print(value[i]);
        Serial.println();
        Serial.println("*********");

        beepsweep(stoi(value),duration*2, 1);
      }
    }
};


 
void setup() {
  Serial.begin(115200);

  // initialize LED digital pin as an output, and buzzer analog pin.
  pinMode(D8, OUTPUT);
  pinMode(A0, OUTPUT);
 
  BLEDevice::init("Bzzzzzzzt");
  BLEServer *pServer = BLEDevice::createServer();
 
  BLEService *pService = pServer->createService(SERVICE_UUID);
 
  BLECharacteristic *pCharacteristic = pService->createCharacteristic(
                                         CHARACTERISTIC_UUID,
                                         BLECharacteristic::PROPERTY_READ |
                                         BLECharacteristic::PROPERTY_WRITE
                                       );
 
  pCharacteristic->setCallbacks(new MyCallbacks());
 
  pCharacteristic->setValue("Hello World");
  pService->start();
 
  BLEAdvertising *pAdvertising = pServer->getAdvertising();
  pAdvertising->start();
}
 
void loop() {
  // put your main code here, to run repeatedly:
  delay(2000);
}

I also strapped a little flat buzzer to the window and sent bluetooth notes to it from across the building. Couldnt get it to work from the building next door though, is it because of the wind? Or the windowpane? Or the windy staircase? Anyway strapping it to the window is very interesting, these buzzers are really weak and the sound comes through so quietly I really like it.

Here's the sound

So that works great for BT signals in and serial out but I need to introduce the two machines to eachother, so make a connection and send BLE messages.

Wifey

Tutorial I'm trying. So first I'm making one ESP32C3 a wifi access point (like a hotspot on a phone). It works and I can connect to it on my phone or laptop. Now I'll try to set up the other module to connect to this network.

Day 4

La La LAN

Today: connect ESP32-C3s directly to eachother. On Wifi. On bluetooth. Lets try WiFi first, still using yesterday's tutorial. Connecting to the other ESP32 network: all works good. I also set up the connecting chip to have a webserver that turns on an LED, and plays a buzzer.

Here's the sound

Also managed to connect to eduroam. No cert needed here, just login credentials. So I can connect both ESPs to the eduroam wifi at the same time. I can ping them and have given them Wifi status LED's that flash if WiFi is connected. But I cant get a webserver running on eduroam hmmm. Maybe I should try to open a port, or maybe it wont be possible to use this network.

Now they need to have a little chat with eachother.

Day 5

The day I get the ESP32 to send more interesting data

This playlist gives me some information on how to MIDI on Arduino. But actually that is for MIDI over DIN, so maybe for a different day. This library does lots of MIDI stuff including MIDI over serial and BLE, and can later be expanded with this to do rtpMIDI.

Also tried MIDIUSB where I learnt that an ESP32 does not in fact have USB communication, rather a serial port going down a USB wire. Interessant.

First I'm getting the ESP32 to send MIDI over BLE, using this script and this BLE MIDI driver. The code below sends a 150bpm drum pattern over MIDI, while playing a melody itself on a buzzer. I really like how the synchronisation is all messed up, this is what I was hoping for. Next week the plan is to play with this effect some more and get it happeing on different networks. I am working with microsecond delays for the moment, but this is probably not the most accutate way to count time. Or maybe it is. The processor is 160Mhz so I'm not sure if I need to account for that, more likely I think it is to do with bluetooth latency (does bluetooth MIDI have strict timing? I doubt it).

It sounds like this.

#include <Arduino.h>
#include <BLEMIDI_Transport.h>
#include <hardware/BLEMIDI_ESP32.h>
BLEMIDI_CREATE_INSTANCE("Bammm",MIDI);
  
// Define some pitches of beeps, a basic time unit, a status
int melody[16] = {400, 360, 480, 400, 960, 200, 400, 360, 480, 400, 960, 200, 400, 360, 480, 400};
int melodyPos = 0;
int ledstatus=0;
int tempo = 100000; //a 16th note in 150bpm
 
// functions for making sounds

void ledSwitch(){
  //this part switches the LED status
  if(ledstatus==0){
    digitalWrite(D8,HIGH);
    ledstatus=1;
  }else{
    digitalWrite(D8,LOW);
    ledstatus=0;
  }
}

void clap(int tone1) {
  ledSwitch();
  tone(A0, tone1, 100);
  delayMicroseconds(tempo-10000);
  ledSwitch();
}

void setup()
{
  // initialize LED digital pin as an output, and buzzer analog pin.
  pinMode(D8, OUTPUT);
  pinMode(A0, OUTPUT);

  MIDI.begin(10);
}

void loop()
{
    for (int i =0; i<16; i++){
      if((i%4==0)||(i==14)){
        MIDI.sendNoteOn(36, 100, 10);
        MIDI.sendNoteOff(36, 0, 10);
      }
      MIDI.sendNoteOn(44, 100, 10);
      MIDI.sendNoteOff(44, 0, 10);
      if(i%3==0){
        clap(melody[melodyPos%16]);
        melodyPos++;
      }else{
        delayMicroseconds(tempo); 
      }
    }
}

Next week: rtpMIDI, receiving MIDI data to make sounds on the ESP, buttons, two ESPs communicating (via some sort of MIDI).