Phantom Sync

Introduction

This piece is an interactive sound installation that explores how sound can be spatialized, embodied, and shared through movement. Situated in a room equipped with four corner speakers and real-time motion tracking, the installation invites up to four participants to co-create an evolving musical composition simply by moving through space. Each participant activates and controls one distinct audio layer from a larger composition, which follows their position across the room. The movement is tracked using a Kinect sensor, allowing their sound to dynamically pan between speakers, creating a sense that the audio is tethered to their body.

As more people enter the installation, new sonic elements are introduced building the piece from a sparse melody into a full, immersive soundscape. Participants are rewarded for their presence and persistence. The longer they stay within the space, the more their individual track transforms, subtly layering in additional rhythms, harmonies, or effects. This encourages not only exploration but also a type of playful occupation—lingering, listening, and adjusting one’s position to feel how the space responds sonically.

The system is intentionally ephemeral. Once all participants exit the room, the audio resets to its minimal state, erasing the previous sonic configuration and setting the environment for a fresh experience. This cyclical structure reinforces the piece’s core themes: sound as presence, space as instrument, and participation as performance.

The piece invites a rethinking of how we interact with sound positioning it not as background ambiance or linear composition, but as something active, relational, and constantly changing. Through movement, participants are not merely listeners but co-creators of a shared, spatial auditory experience.

Conceptual Framework

The piece build form the theme of nostalgic augmentation. A process of reimagining familiar, flat, two-dimensional auditory experiences and recontextualizing them into spatial, embodied, and interactive environments. Drawing from the soundscapes of classic arcade games, audio once bound to the screen and linear progression, the installation brings these lo-fi, loop-based compositions into a responsive 3D field. These sonic memories, deeply rooted in early digital culture, are given new life through movement, presence, and spatial transformation.

In traditional settings, sound is ambient or directional, but rarely tangible. The piece challenges that perception by giving sound a physical sense of positionality. Through real-time tracking and speaker-based spatialization, sound is no longer a passive element in the background. It becomes something to be followed, felt, and navigated. Participants become both listeners and instruments, agents through which sound travels and evolves. Transforming the act of listening into an embodied experience.

The installation uses this sense of spatial audio to foster play and exploration. There are no rules, no objectives just an open invitation to move, to stay, to listen, and to interact. The evolving audio composition rewards curiosity and sustained engagement, encouraging participants to linger and discover how their presence shapes the sonic environment. Over time, what begins as a simple gesture of movement becomes an act of uncovering and invention.

At its core, the piece is a framework for co-creation. Each visitor contributes a layer to the collective composition, meaning the work is never complete without its participants. The piece exists in flux, shaped by those who inhabit it. No two sessions are the same. Through this collaboration, the installation highlights the relational nature of sound, presence, and space. Sound becomes a shared material, and movement becomes a medium of dialogue.

By collapsing the boundary between listener and composer, and blending nostalgia with spatial experimentation, the piece transforms sound from something heard into something lived reclaiming the sonic from the screen and redistributing it across bodies.

Interaction Video

Process Images

Technical Documentation

Core Interaction: Real-time spatial audio mapping via participant movement

Max Participants: 4 simultaneous users

Maximum Room Size: 15 feet x 15 feet

Speaker Positioning: One full-range speaker in each corner, aimed slightly inward toward the center

Tracking Zone: Kinect is positioned approximately 5.5 feet high, tilted slightly downward

System Overview

The piece is built on a real-time tracking and spatial audio engine using Kinect v2 for motion capture and TouchDesigner as the primary environment for sound control, logic processing, and spatial mapping. The system dynamically maps participants’ positions (X and Z coordinates) in a room to spatialized audio outputs across a four-speaker setup, with each speaker positioned in a corner of the installation space.

Hardware Components

1. Kinect v2 or Azure Kinect Captures participants’ skeletal and positional data in real time.
2. PC with GPU-capable specs Required to run TouchDesigner with stable performance for both video and audio processing.
3. 4 Active Full-Range Speakers Positioned in each corner of the room to enable quadraphonic sound panning.
4. Audio Interface (4-channel output) Interfaces with TouchDesigner to route each audio channel to its respective speaker.
5. Zoom L-12 or equivalent Used as an ASIO-compatible interface for clean multi-channel output.

Interaction Logic

Player Identified:

This logic flow is designed to detect when a participant enters the space and use that data to control the playback of their assigned audio layer. The system tracks player presence via the Kinect’s id channel and processes it as follows:

• Player ID Selection: The Kinect CHOP outputs skeletal tracking data, including unique id values for each detected participant. The select CHOP isolates a specific player ID (e.g., p1/id).
• Presence Validation: The math and logic CHOPs evaluate the selected ID stream to determine presence. If the ID exists (i.e., a body is actively tracked), a binary signal of 1 is output. If not, it outputs 0. This effectively becomes a presence flag.
• Binary Normalization: The output is normalized and processed through a math CHOP to ensure a clean binary value (0 or 1), suitable for triggering downstream logic.
• Cue Triggering: The renamed output channel (start) is fed into a start_cue CHOP that directly controls the Cue Pulse of an Audio File In CHOP. When a player enters, the track begins or resumes. When all players the audio files are reset to the beginning of each track.

Real Time Panning

This flow forms the core logic behind the spatial movement of audio within the installation. The participant’s position is tracked using the Kinect’s tx (horizontal) and tz (depth) values, effectively mapping their location onto a 2D grid. This data is used to dynamically control the perceived position of sound within the room. A mirrored volume balancing approach similar to a crossfading matrix was used to pan audio between the four corner speakers. As a participant moves closer to one speaker, its volume increases while the opposite speaker’s volume proportionally decreases, creating a smooth spatial transition. These values are passed through math and limit CHOPs to normalize the output between 0 and 1, ensuring a clean, bounded range suitable for modulating speaker levels. The result is a fluid, position-responsive soundscape where each participant’s audio track follows their movement in real time, reinforcing the illusion of sound having a tangible presence in space.

This process was then re-created for each of the four participants.