Embodied Interaction

Aalto University, Lecturer: Matti Niinimäki

Hand Gesture Assignment

We started by reviewing some famous hand gestures. As Italians are renowned for using their hands often, even in daily conversations, we got interested in gestures like pinched fingers, but with some changes. We realized that when we open the pinched-fingers pose, it conveys a sense of growth, bloom, and prosperity.

This could be a proper fit for a relaxing VR gardening experience. By using hand detection, players can not only put flowers in the scene but also feel their bloom and have a deeper gardening experience. Here are two example actions using the mentioned gesture.

Here you can see a demo of how it can be used in a 2D space. We also implemented a simple interface for users to select, scale, and create flowers.

Final Project Idea

Spectrum Mirror

Whenever I listen to any music, I feel some energies around me; they are bouncing around based on the tones and change my mood. Obviously, they are not visible! But what if we show them using new technologies to have an immersive musical experience?

Music-reactive particle effect by Keijiro Takahashi

From Keijiro Takahashi

The idea is simple: it's been a long time since we started visualizing music and sounds using their spectrum. Here, I'll bind that data to different VFX properties to achieve the mentioned effect. For the first demo, players will stand in front of a screen and see themselves as a visual effect. They can play any song from Spotify, listen to it, and also observe their VFX replica on the screen.
For the first demo, I'll put most of my time into creating appealing visuals and also binding music frequency to them in a valid manner. But this idea can go further, both in terms of visual and audio analysis.

Tools

The visual effects are made in the Unity game engine using its GPU-based particle system (VFX Graph).
Also, to use Azure Kinect data in Unity, I used a free Unity package from here.

First Spectrum Mirror tests with a depth camera

Music Features

My goal is that users can play any music they want from Spotify, and the VFX will change based on that music in real time.
To get audio spectrum data, I use the Keijiro LASP repository, which provides the FFT (Fast Fourier Transform). I have access to Low Pass, Band Pass, High Pass, and spectrum data. In addition, I use some custom scripts to smooth those values based on my needs.
Also, I managed to use Spotify4Unity, which uses SpotifyAPI-NET, to get some attributes of the playing music on the fly. Now, whenever a new track plays, I have access to its Tempo, Energy, Valence, and Danceability (range 0 to 1).

For the course demo, I mainly used low, band, and high pass data of the spectrum, including their average over the past few seconds. Also, the passes are processed in a custom script to make them smoother and calculate their recent average.

Debug UI showing low, band, and high pass spectrum data

For the VFX part, I use those three different passes as the inputs of three different noises and use the song's energy values as the Drag parameters on them. The effect has two predefined colors, which are set based on camera distance, and for each song, it changes its Hue value randomly. Also, colors move based on the Danceability property of the playing music.

Final Showcase

My main goal for this project was to connect technology to art in a novel way and also learn new tools and techniques. I had wanted to contribute to some sort of digital art project for a while, and this course was a great start. During the class, I got familiar with TouchDesigner and started to learn VFX Graph, which I had been looking to learn for a while. Also, grasping different sensors and overall possibilities in this area was very informative.

The current version of the project works as a digital mirror where users stand in front of it and observe the visuals based on the desired music. Also, as I only use depth detection, there is no limit on the number of users, and it detects the whole body, including users' clothes, hair, bags, etc., but in an abstract way. Fortunately, most of the users were happy with the audio visualization side of the project. Currently, it works well with kinds of music that have many ups and downs, but in general, it works well enough. In addition, in some playtesting, users were so involved in playing with the visuals and moving around that audio responsiveness was not their first concern.

I also had an idea of incorporating body (hand) detection into this system and showing players another level of immersion by detecting hand collision with particles or playing some specific momentary effects based on hands (clapping, etc.). Unfortunately, I faced a huge library conflict with my current depth package and couldn't resolve it in time, but I think it's something doable in the future.

This idea can be used and expanded in different directions, from having this mirror concept in digital art museums to implementing it in some dance and live shows with even more cameras and visuals. For now, I'll start searching and applying for support and grants to extend this project. I believe this mirror idea fits great with the depth detection system, and I would love to work with some technical composers and maybe art directors for the final polish.