<aside>
Resources
https://tonejs.github.io/docs/15.0.4/index.html
https://docs.ml5js.org/#/reference/bodypose
https://thecodingtrain.com/tracks/ml5js-beginners-guide
</aside>
idea and design of the rain praying instrument.
This instrument is inspired by the idea of “rain”. I want to create a cubical 3D environment, with 4 rain-inspired sounds each occupying one wall, and user can “pray” for the rain using different pose. Since I’m using WebGL and ml5 pose detection in another class, I want to incorporate this technique in the AV instrument. For the visual, I chose pattern accumulated by simple geometry, to mimic the nature of rain. “Falling” is the main motion in this work, with different gravity directions going on each wall.
First, I selected 4 sounds in freesound. My goal was to find sounds that representing different raining scenario, unique from each other but still forms a sense of harmony. It’s important that not more than one of them could have strong melody and composition, or else it could be a mess. Here are the sounds group I found:
Second, I started the coding of the pose detection. The original idea was to use the distance of specific body part (ex: hands) key points and the walls as trigger. However, since the BlazePose 3D ml5 detection model use its waist key points as (0,0,0) in the skeleton, the body part’s relative position toward the 3D environment is unstable.
Hence, I changed into using the relative position within different body parts, to determine if the user is doing the trigger pose. This is a much better way, and can produce stabler audio output.
starting scene of the instrument
image created during the instrument playing
Third, I did the rain visual with transparent red strokes. I really like this almost draft-looking visual. It felt like a world with all kind of possibility and creativity embedded. Also, I realized in the process that, thanks to the random nature of the rain visual, different visual output can be created during the instrument performance, which is something unexpected for me.
Here is the final result: