Eden and I decided to continue building upon our last project! Our goal for our final was to implement remote communication to create a p5js sketch that reacted to remote interpersonal synchrony! You can read up on our previous project here: https://www.moaw.art/post/streeeeeeetch-breathe-relax-it-s-project-2
You can check out our final sketch here! It includes all of the Arduino and server codes as well as instructions and visual references for how to join! https://editor.p5js.org/mnk2970/sketches/ylgyvcaia
MEET THE TEAM (Pussy Sublimity):
Image Description: A drawing of three different floating heads, these are cartoon characters meant to represent different members of the band, Pussy Sublimity. From left-to-right: A face with a garlic-shaped hood represents Moaw; a face with a rabbit-shaped hood represents Eric; and a face with a cat-shaped hood represents Eden
We split the project up into two parts! Eden worked on fabricating, calibrating, and fine-tuning the sensor, and I worked on learning about servers and polishing up the p5js sketch! As always, Eric, the third member of Pussy Sublimity was also an important source of support. He helped supply a server for our group's project, as well as resources for server-communication!
CONCEPT:
The project changed quite a bit from what we initially envisioned! For our p5js sketch, we originally planned on having two glowing and overlapping orbs to represent interpersonal synchrony. However, we decided to change the design of the sketch after our class emphasized the effect of latency in server communication.
image description: A drawing of our concept for the final project! The words, "THE PLAN", is scrawled into the corner of the image. At the top of the image is a computer window representing a p5js sketch with a feather floating in it. There is text to the left of the window, which reads as "collective force / # of clients". Three computers are drawn below, with wireless signals beaming towards the window, representing serial communication. Each computer is hooked to little buttons, which represent sensors.
Our new plan was to have our sketch be a single floating feather! The height of the feather would be determined by the accumulative sensor values being sent to and from the server, which would then be divided by the number of clients. We also planned on having a wave function which would control the movement of the feather, the shape and motion of the feather would be influenced by a separate "synchronicity" variable, to communicate the degree of interpersonal synchrony.
We thought that having a single feather would be a more rewarding method of feedback rather than the two overlapping orbs. Our concern was that the two overlapping orbs would feel punishing to participants who were not experiencing interpersonal synchrony, With our new sketch, participants would still be able to engage through collaborating to move a single object.
This new concept required several steps:
Coding a feather prototype sketch and determining how a single local sensor could affect feather movement
Establishing initial communication with the server with the p5js sketch
Creating a feather simulation in the server code
Incorporate the data from multiple clients through sensor fusion
Several things changed throughout the process. For example, while we originally planned on using a wave equation to represent synchrony, we ended up removing that from our sketch altogether. This ultimately effected the way we also chose to have the server send packages to each client.
THAT'S A LOT OF STUFF!! LET'S BEGIN AND GO IN ORDER!
THE FEATHER PROTOTYPE SKETCH:
You can check out the initial prototype here! https://editor.p5js.org/mnk2970/sketches/lJatpjA2w
I had the fun opportunity to review high school math that I had long abandoned. A lot of time was spent creating bounding boxes to make sure the feather stayed on screen! Even more time was needed for the dampening and velocity equations to make sure that the feather fell nice and slowly.
Comments are written in blue!
let feather
let x, y; // coordinates for feather movement equation
let vx
let vy // this is for our velocity
x = 70
y = 0
vx = 0
vy = 0
//████████████████ this is our image library
function preload() {
feather = loadImage('assets/3.png')
}
function setup() {
createCanvas(500, 700);
}
function draw() {
background(155);
image (feather,x, y )// the feather origin is at 70, 530
x = x + vx;
y = y + vy;
vy = vy + 1 // applies gravity
if (y < 0) {
vy = vy + 2;
}
if (y > 530) {
vy = vy -2;
if (abs(vy) < 2 ) {
y = 530
vy = 0 // when it reaches the bottom, this is to simulate a "floor" for our feather
}
}
if (x < 0) {
vx = vx + 2;
}
if (x > width - 350) {
vx = vx -2; // lines 33-44 keep our feather in bounds
}
if (abs(vy) > 1.0)
vx = vx + random (-100, 100)/1000 * abs(vy);
else
vx = vx / 1.1
vy = vy/1.01 // exponential dampening to determine the speed of how it falls
vx = vx/1.01// expoential dampening for my x value
if (abs(vy) > 2)
vy = vy > 0 ? 2 : -2We got the feather movement down, but then we had to tweak the Arduino code for the purpose of our sketch!
MAPPING AND CALIBRATING OUR SENSORS https://www.arduino.cc/reference/en/language/functions/math/map/
image description: A two-part cartoon drawing featuring the cartoon versions of me, Eric, and Eden!
Panel 1 Moaw: "I just want ALL of the sensors values to register the same on the p5js sketch."
Panel 2
Eric: What you're looking for is mapping!
Moaw: oh!
Panel 3
Eden is shown next an illustration of two sensors labeled as Sensor A and Sensor B. Sensor A has a value ranging from 5-659 and Sensor B has a value ranging from 11-720. After mapping, both sensors' values have been mapped to 1-100.
We revisited our Arduino sketch to map our sensor values. This step was very important to ensure that all of our sensors read the same! Mapping our sensors allowed us to take the low and high values of any sensor, as varied as they may be, and have our Arduino code share those values through a uniform range.
int minval = 5; // the minimum streeetch/press value of our sensor
int maxval = 690; // the maximum streeetch/press value of our sensor
//...
//...
average = map (average, minval, maxval, 0, 1023 );This snip of code mapped our force sensor's min value (5) to 0, and its max value (690) to 1023! We wanted the minval and the maxval to be easy to adjust so that anyone could easily edit the Arduino code to calibrate their sensor.
ESTABLISHING SERVER COMMUNICATION! LEARNING ABOUT WEBSOCKETS!
This sketch was our first attempt at using websockets! Check it out here! https://editor.p5js.org/mnk2970/sketches/um1ETaoDe
A DISCOVERY!!
image description: A two-part cartoon drawing! The two halves are vertically oriented. In order from top to bottom:
In the top panel, the cartoon versions of Eden and I are staring at laptops. Eric is standing to the right of both of us. There are speech bubbles that read as follows
Moaw: "This looks familiar"
Eden: It's like serial communication!
Eric: Good observation! It was designed that way intentionally!
This was Eden and my first time engaging with websockets and we weren't sure what to expect! Thankfully, the websocket javascript wasn't terribly unfamiliar, it was designed to be similar to serial communication! Just like with serial communication, the client and server take turns communicating with one another, and waiting for specific events to trigger certain functions!
The following code allowed our p5js sketch to receive and print a message in console.log which was received from our server.
webSocket = new
WebSocket("wss://electromatter.info/bingbong"); // this is a string that identifies the server, electromatter is the server, bingbong is our program! This is where all of the communication happens :- )
webSocket.onopen = function (event) {// websocket will listen for a specific event, which is when the socket gets connected
webSocket.send("MESSAGE");// when connected, this will send a text message
}
webSocket.onmessage = function(event) { // onmessage- data received callback. When p5 gets a mesage back from the server, this message is called
console.log(event.data); // websocket message will appear in the console log
}We used node.js to program the server's counterpart code. You can see the client and server codes and their respective rendezvous!
We then implemented server communication with our feather! We didn't implement multiple sensors/clients yet. For this sketch, we just wanted to send and receive sensor date from our server! You can check it out here: https://editor.p5js.org/mnk2970/sketches/H56XkUaWB
We changed the javascript code so that it would send our data to the server! This involved creating a new serial event function.
**UPDATE THIS SECTION**
CREATE A FEATHER SIMULATION IN THE SERVER CODE
Now that the feather simulation was complete and server communication was established, we moved the feather code to the server code to prepare for our sensor fusion!
It was pretty simple to copy and paste a lot of the code, but various tweaks had to be made in the code for it to run efficiently! The most notable adjustments had to be made as a result of ws.js not having the same libraries as p5js.
image description: A comic!
Panel 1: it's 2:00 AM ET
Moaw: 'ABS' doesn't exist in ws.js... I should search online for a reference!
Panel 2: "How to define abs" is typed into a search engine
Panel 3: The laptop is showing image results for defined abdominal muscles
Moaw: Why did I think this was a good search query?
The feather simulation had the following adjustments:
All code specific to the client was removed (image files, background, etc.)
Definitions were made for functions and variables that only existed in p5js (abs, draw function)
The code was made to run at 20 FPS to increase efficiency and minimize latency, but then was mapped to 30 FPS to match the p5 sketch. THIS WAS THE MOST PAINFUL PART
Added tweak function and dt (derivative of time) to help adjust the data for new frame-rate
Tweak + dt help emulate resolution, which relates to mapping (like the reduction of polygons in 3d modeling)
A helpful reference regarding the Chain Rule, which affected our dt functions! https://www.analyzemath.com/calculus/Integrals/rules-of-integrals-with-examples-and-solutions.html
Here's some of our code where we used the tweak and dt functions to adjust out code to fit the new frame rate! As you can see, the previous equations that utilized addition or subtraction were adjusted by 'dt' and the equations that utilized multiplication or division were tweaked by 'tweak'.
function tweak (value) { // this is to adjust the dampening, which is affected by ouor frame rate which was lowered for efficiency
return Math.pow (value, dt) // this tweaks the value, the code was originally written for 60 fps so this helps
// if you halve the framerate, you halve the number of things added together when its x+=v each frame
// so to get back to the orignal feel you just multiply by two
// x = v + v + v + v + v + v // orignal
// x = (v + v + v) * 2 // half frame rate fixed
// for multiplies each frame, you use an exponential function to get back to the orignal feel
// x = v * v * v * v * v * v // orignal
// x = (v * v * v)^2 = pow(v,2) // square to return to original
}
function update() {
x += vx*dt; // multiplying by 'dt' lessens dependance on frame rate
y += vy*dt;
vy += 1*dt // applies gravity
if (y < 0) {
vy += 2*dt;
} else if (y > height - 170) {
vy -= 2*dt;
if (abs(vy) < 2) {
y = height - 170
vy = 0 // when it reaches the bottom, this should make the feather stop partying
}
}
if (x < 0) {
vx += 2*dt;
} else if (x > width - 350) {
vx -= 2*dt; // lines 33-44 keep our feather in bounds
}
if (abs(vy) > 1.0) {
vx += random(-100, 100) / 1000 * abs(vy) * dt;
} else {
vx /= tweak(1.1)
}
vy /= tweak(1.01) // exponential dampening to determine the speed of how it falls
vx /= tweak(1.01) // exponential dampening for my x value
if (abs(vy) > 2) {
vy = vy > 0 ? 2 : -2
}This took a good bit of tweaking, but HOORAY!
CONVOLUTED CONVOLUTION AND SENSOR FUSION
Our initial goal was to have our feather's movement affected by a wave function that represented a separate value for synchrony, but that ended up not panning out.
We relied on a dot matrix to make an array for each client! We would combine all of the clients sensor data and then divide it by the number of clients. The combining of sensor data for uncertainty reduction is known as SENSOR FUSION!
We added some code at the top within our call function! This addition of code adds and removes clients from the array.
server.on('connection', function(c) { // when a client establishes a connection, it calls this function
clients[clients.length] = c // throw client into the end of the array
c.synch = 0 //an instance variable on the client that keeps the state of the sensor
c.on('close', function(event) {
for (i = 0; i < clients.length ; i++) {
if (clients[i] === c) { // finds the client in the array
clients.splice(i, 1) // removes the client by subtracting one from the total when they disconnect
break // this means we're done removing clients
}
}
})
c.on('message', function(message) { //it listens for messages from the client
c.synch = +message
console.log(`Received message => ${message}`) // when it receives message, it shows 'message'
})
})At the bottom of the code, we have our adjustment function! This snippet of code combines the data and divides it by the number of clients in the array.
We updated our 'adjustment' variable, which affected the velocity/force of the existing feather equation!
adjustment = 0
for (i = 0; i < clients.length ; i++) {
targetheight = (height - 170)*(1- clients[i].synch) // combined data
adjustment += (y - targetheight)/50 / clients.length // force is divided by number of clients
}
vy -= adjustment*dt
Now after tweaking the p5js code so that it shared a state with a server, this ended up workin- WAIT
OH NOOO!! WHAT IS HAPPENING????????
This error occurred because JSON wasn't able to send the array to our client. We had to go back to the server code and convert our array to a string so the code could read the data!
function send2client() {
position = [x,y]
message = JSON.stringify(position) // send back an array of position
for (i = 0; i < clients.length; i++) {
try {
clients[i].send(message)
} catch (error) {
console.log(error);
}
}
}WHEW OOKAY! NOW WE'RE READY TO PARTY PARTY!! We wanted all of our code to be neatly packaged and a helpful reference, so we did our best to put all of our documentation in our p5js file, which includes our Arduino and server code! We even included instructions in the Arduino to make calibration easier and a diagram for people to set up their breadboards so that they can join the party!
YOU CAN CHECK OUT THE P5JS SKETCH (and everything that comes with it) HERE!! https://editor.p5js.org/mnk2970/sketches/ylgyvcaia