Wang SI24

Midi Controller

Main Idea

Rain Receiver is actually a midi controller but just by using sensor.

Most logic of a MIDI controller is to help users map their own values/parameters in the DAW, using potentiometers, sliders, buttons for sequencer steps (momentary), or buttons for notes (latching). The layout of the controls on these devices usually features a tight square grid with knobs, which is a common design for controllers intended for efficient control over parameters such as volume, pan, filters, and effects.

I would like to make some upgrades in the following two ways:

the layout

During an period of using pedals and MIDI controllers for performance, I've organized most of my mappings into groups, with each group containing 3 or 4 parameters.

For example,

the Echo group includes Dry/Wet, Time, and Pitch parameters, the ARP group includes Frequency, Pitch, Fine, Send C, and Send D.

In such cases, a controller with a 3x8 grid of knobs is not suitable for mapping effects with 5 parameters, like the ARP. A structure resembling a tree or radial structure would be more suitable for the ARP's complex configuration.

The structures, such as planets and satellites, demonstrate how a large star can be radially mapped to an entire galaxy; A tiny satellite orbiting a planet may follow an elliptical path rather than a circular one; Comets exhibit irregular movement patterns. The logic behind all these phenomena is similar to the logic I use when producing music.

Issues

In the process of changing the way of controlling sound, the core logic is to build a connection(mappings) of various items. Same as the Html sound experiment.

Chips like the UNO are tricky because they can't use 'midiusb.h'. The MIDI signal is hard to recognize even with Hairless MIDI. I could use the Standard Firmata with the Max plugin, but the value of the potentiometer is not stable. I attempted to add a 'potThreshold' but don’t know how to modify the Standard Firmata.

In this case, I have to switch to Leonardo or Micro.

Reference

https://dartmobo.com/dart-kombat/

https://www.matrixsynth.com/2017/01/polytik-new-project-via-artists.html

https://bleeplabs.com/product/thingamagoop-3000/

https://www.bilibili.com/video/BV1Bj411u72k/?spm_id_from=333.788.recommend_more_video.0

https://www.bilibili.com/video/BV1wx4y1k7pc/?spm_id_from=333.788.recommend_more_video.-1

https://www.youtube.com/watch?v=a8h3b26hgN4

https://ciat-lonbarde.net/ciat-lonbarde/plumbutter/index.html

Insperations

Stars

Chains/With saw gear

Layer

PCB etching

For patterns, and touch function

Format

A moving midi controller, using the same parameter to control the move, but then I’ll need a controller to control the midi controller.
A clock with various LDR, activated by the second clock pointer. This could become a clock sequencer.
A sandbox with buildings or cars or trees, the model will be the knob of the potentiometer.
A controller to hold, with several buttons on it.

Exterior

the box probably needs to use plastic but not iron(too heavy), or wood(not stable, even through wood still needs the laser cut) maybe the acrylic board.

Components

6-8 knobs for potentiometer;
4 or 8 buttons;
a slider or two?
Maybe a LDR?(but how to map the LDR to midi signal??- maybe a button to active the LDR, and a light to show of the LDR is turn on or off, and when using the LDR is the same logic to map in the live. But is the LDR need a cover?)
A arcade joystick potentiometer for controller the 360 degree values like the surrounding sounds.

ADXL345 - 3 axis(for the Hands)

Boards/Chips

Arduino Leonardo / Pro micro. Multiplexer

Code

with the modify X3 version the buttons and potentiometers are finished but still don’t know how to add the slider and the joystick.(for the Ableton live mapping part actually is not a problem, just need to map to the X and Y, the circle is just a appearance. So it is important to get the X and Y parameter from Arduino and transfer to the midi format.)

  #include <MIDI.h>
  #include <ResponsiveAnalogRead.h>
  const int N_POTS=3;
  int potPin[N_POTS] = { A1, A2, A3};
  int potCC[N_POTS] = {11, 12};
  int potReading[N_POTS] = {0};
  int potState[N_POTS] = { 0 };
  int potPState[N_POTS] = { 0 };
  int midiState[N_POTS] = { 0 };
  int midiPState[N_POTS] = { 0 };
  const byte potThreshold = 6;
  const int POT_TIMEOUT = 300;
  unsigned long pPotTime [N_POTS] = {0}; 
  unsigned long potTimer [N_POTS] = {0};
  float snapMultiplier = 0.01;
  ResponsiveAnalogRead responsivePot[N_POTS] = {};

void setup() {

 Serial.begin(9600);
 MIDI.begin(MIDI_CHANNEL_OMNI);
 for (int i = 0; i < N_POTS; i++){
   responsivePot[i] = ResponsiveAnalogRead(0, true, snapMultiplier);
   responsivePot[i].setAnalogResolution(1023);
 } 
  }

void loop() {

 for (int i = 0; i < N_POTS; i++){
   potReading[i] = analogRead(potPin[i]);
   responsivePot[i].update(potReading[i]);
   potState[i] = responsivePot[i].getValue();
   midiState[i] = map(potState[i], 0, 1023, 0, 128);
   int potVar = abs(potState[i] - potPState[i]);
   if (potVar > potThreshold){
     pPotTime[i] = millis();
   }
   potTimer[i] = millis() - pPotTime[i];
 if(potTimer[i] < POT_TIMEOUT) {
     if (midiState[i] !=midiPState[i]){
       Serial.print("Pot ");
       Serial.print(i);
       Serial.print(" | ");
       Serial.print("PotState: ");
       Serial.print(potState[i]);
       Serial.print(" - midiState: ");    
       Serial.println(midiState[i]);
       MIDI.sendControlChange(potCC[i], midiState[i], 1);
       midiPState[i] = midiState[i];
     }
       potPState[i] = potState[i];
   }  
 }
  }

shopping list

leonardo

expander cd4067

potentiometer 6

buttons

mpr121

joystick

slider

ledbutton

Puzzle blocks game

when I'm drawing the drafts, I still thinking about block game, use the wood stick/block to make a real puzzle game, so the logic is let the sliders influence each other in a physical way. The sliders will became to a path, some slider will also not able to use when the other sliders are occupied the place.

To build a puzzle blocks midi controller, I still need:

- [ ] Buy sliders for about 9 or more.

- [ ] Cutting Wood blocks in wood station.

- [ ] Design the puzzle path.

Javascript Club - Gamepad/Midi API

Joining Rosa's JavaScript club has been really inspiring and helpful. I think I finally found a way to link the HTML I made last trimester with physical controls.

Gamepad to control the Helicopter

I tried to use the gamepad to control the cursor at first, the movement is easy to map but the click/mouse down/mouse up is hard to simulate, with my broken coding, I failed and came up with more stupid ideas, use the WASD key to control the helicopter then map the Index 0,1,2,3,4,5 to the WASD and J K(which also failed at the end and took more time).

Protocol:

1, Key control - using WASD to control the helicopter.

2, Replace the draggable by key control in the main codes, also add the J and K to control the fill and drink.

3, Map the Index 0,1,2,3,4,5 of the gamepad to the WASD and J K - Failed

4, Using the same approach of key control, let the values of the helicopter movements echo to the Index 0,1,2,3,4,5 == button pressed.

In the end, I got a Beer Robber version with both able to use key and gamepad control.

Midi controller to control the slider block

At first I meet some issues that the midi controller is only controlling the slider(css) moving but not able to control the sound values, I found the midi code part is outside of the pitch loop, which caused the effect not able to defined.

After the issue fixed, I also mapped the buttons to control the pitch blocks in the html.

https://hub.xpub.nl/chopchop/~wang/__lab__/files/public_html/RE3%20MIDI.html?_xsrf=2%7Ce2ae7386%7C5af0ce2c1735f7814fb3b50b6e485ca6%7C1713860808

Browser extensions

The Browser Sound plugin is a browser-based tool that adds various effects to your audio and video.

This browser-based plugin could easily accessing samples for your projects.

Unlike plugins in digital audio workstations (DAWs), this plugin operates directly in your web browser. It will distort your audio or video when you play them in the browser.

Regarding my recent experiments, I've been using pizzicato.js for sound experiments, but I've found the selection of sound sources to be limited.

To address this limitation, I'm interested in creating a browser-based plugin that allows users turn the Browser into an instrument, a sampler, by searching and using their own samples from various websites and platforms.

https://pad.xpub.nl/p/BRNP

What is it?

The Browser Sound plugin is a browser-based tool that adds various effects to your audio and video, which could easily accessing samples for your projects.

Why make it?

1,The plugin addresses the limitation of traditional DAW by providing users with a convenient way to access and manipulate audio samples directly within their web browser. 2,It could expand the possibilities of sound experimentation, allows users turn the Browser into an instrument, a sampler, by searching and using their own samples from various websites and platforms.

Workflow

1, Research on the browser plugins. 2, Sound design part, list the effect could be using on the sampler, design the audio effect chain. 3, Coding part on browser extension. 4, Based on the coding part to fix the sound design. 5, Interface design for the plugin. 6, Based on the Interface design to redesign the code logic. 7, Test and fix any bugs. 8, Using the plugin to record a sound file.

Timetable

Total: 22 days 12 h 0 min 0 s. 1, Research: 3 days. 2, Sound design part: 3 days. 3, Coding part: 5 days. 4, Interface design: 3 days. break: 12 h 5, Bug fixing: 5 days. 6, test and record: 3 days.

Previous practice

Regarding my recent experiments, I've been using pizzicato.js for sound experiments, the sound source is using the SineWave with different frequencies, but I've found the selection of sound sources are limited, it should be able to use any audio source from the another websites and platforms.

Relation to a wider context

The Browser Sound plugin contributes to the broader landscape of digital audio production tools by offering a new approach to sound manipulation and sampling. It bridges the gap between traditional DAWs and web-based platforms, providing users with innovative ways to create and manipulate audio content.

Part II 17/04

I'm sticking with my focus on developing the "browser plugin" and finding ways to expand its functionality. As part of the project, I'm starting with research to gather insights and ideas.

Regarding references

I've explored feedback and found inspiration from projects like Pellow, which explores creating sound from browser- based data, and MIDI controllers, which could potentially be integrated to control the plugin. Experimenting with multiple browser tabs to a mixer is an interesting way to explore.

Accessibility

I've also thought about accessibility factors for the web page. Have you considered features like Speech to Text? There are existing browser extensions like Read in the Dark, Read Aloud, and Zoom In/Out that enhance accessibility. For sound-related extensions, most focus on volume enhancement or equalization, leaving room for a plugin with effects like tremolo, phaser, distortion, or even a sampler recorder, as suggested by Senka.

Does it has potential?

I think there's still some potential for the plugin, especially given the existing landscape of browser extensions and the need for more diverse sound-related tools. Im started working on it, and I'm considering using a simple JSON file and possibly an API to facilitate its development.

Part III 24/04

In the first step, I tried adding pizzicato.js to both the popup.html and background.js files, but meeting issues with it not being defined. Despite trying with a lot different approachs, even copying all the code directly into the background.js, the problem persisted.

When I adding the url src rather than library, I found there a CDN issue from the console, which is means with the Manifest version 2.0, Chrome is not able to accept using the external javascript to modifying real-time audio from tabs, due to "security restrictions", to protect user privacy and prevent malicious extensions from capturing sensitive audio content.

In this case, I have to try to use the Audio API to make this extension.