Video 1: Experimental Clip 1

The first video emerged from a tightly curated process, what I'd call a focused dialogue between myself and the AI. I used softer instruments, ambient textures, and subtle key shifts within MusicFX DJ, maintaining a slow and intentional pace throughout the session. My approach was compositional I sculpted the sound environment first, carefully adjusting parameters such as mood, key, and BPM. The resulting audio was both lush and restrained, allowing space for the visuals to breathe.

The visuals, generated in Autolume and driven by OSC data from TouchDesigner, reflected this tone. Because the musical energy was low and consistent, the latent vectors moved gently through the GAN space.

Video 2: Experimental Clip 2

In contrast, the second video was driven by play and improvisation. I deliberately pushed MusicFX DJ toward more rhythmic, syncopated territory experimenting with faster BPMs, less predictable prompts, and unexpected genre blends. The music evolved in a more randomized, exploratory fashion, and I allowed the AI tool to take more liberties. Rather than controlling the structure tightly, I responded in real-time to its outputs.

This mode of interaction created a loop of adaptation. Autolume's visuals still reacting to the same OSC pipeline became more volatile. Sudden spikes in the audio data sent the GAN imagery spiraling into abstract bursts, distortions, and color inversions. I found myself adjusting the music mid-performance to balance or exaggerate what the visuals were doing. Sometimes, if Autolume generated an unexpected flash of red or geometric chaos, I would nudge the audio toward more percussive elements to maintain the energy or contrast it deliberately.

The result was a jam between me and the machine. While I initiated many of the changes, the AI's responses often caught me off-guard. This video embodied a more emergent authorship, where control shifted fluidly, and serendipity became a collaborator. The randomness didn't feel like chaos it felt like creative momentum.

The Process

The aim of this exploratory project was to engage with readily available AI tools on the web and investigate the interplay of agency between user and machine. Specifically, I was interested in how much control a user can exercise over the tool and, conversely, how much influence the tool exerts on the creative outcome. My direction focused on the domains of music creation and real-time visualization, asking: What if we could create a generative set of music alongside live visuals that react to user input? This inquiry serves as a lens into the collaborative tension between human intention and machine interpretation.

The two core tools chosen for this experiment were Google's MusicFX DJ and Autolume by Meta Creation Lab. They were selected for their contrasting structures of user control, input/output mechanisms, and creative behavior.

MusicFX DJ allows users to enter textual prompts describing the kind of music or sound they want, along with basic parameters such as BPM and key. The tool then synthesizes music by blending generated samples based on these prompts. This gives users an initial sense of control, but the final musical output is ultimately curated by the AI's interpretation. The system chooses what it believes best fits the user's request, which can make the results feel unpredictable or even resistant to precise control. Furthermore, the generative process is non-deterministic meaning the same prompt is unlikely to yield the same result twice.

Autolume works differently. It leverages pretrained latent diffusion models that users can manipulate directly. By navigating sliders that influence the latent space, users can prompt the model to generate visual outputs in real time. The tool produces surprising results new forms, colors, or transitions the user may not have consciously designed, yet still keeps the visuals within the aesthetic constraints of the trained model. Importantly, Autolume supports parameter reproducibility, a specific slider combination can be saved and revisited to regenerate a nearly identical output, giving the user more reliable control.

Why MusicFX? Why Autolume?

These two tools were selected for their differing affordances. MusicFX DJ is fast and accessible, offering an immediate route to music generation through high-level abstraction. It's ideal for spontaneous exploration, but sacrifices precision. Autolume, while more difficult to set up, allows for deeper engagement, its model-based visual generation opens a playground of visual aesthetics where user agency is both supported and occasionally subverted.

Together, they form a useful spectrum for exploring degrees of user control versus machine influence in generative work.

Technical Challenges and Setup

MusicFX DJ runs directly in the browser with no installation required. It is intuitive and straightforward, simply enter the site URL and begin generating.

Autolume required more technical effort. It involved setting up Python environments, installing dependencies, and using command-line tools like PowerShell. Once installed, however, it offers a graphical interface that simplifies ongoing use, making it accessible to non-coders for visual experimentation.

To connect the auditory world of MusicFX DJ with the visual world of Autolume, I used TouchDesigner as a real-time signal analysis and routing tool. MusicFX does not natively support audio-reactive output, but by routing the system audio into TouchDesigner, I was able to extract features from the music in real time, such as amplitude peaks, frequency bands, and rhythmic patterns.

These values were then mapped to control parameters in Autolume using OSC (Open Sound Control). Although Autolume doesn't explicitly state what each internal slider does, I experimented with sending OSC messages to adjust its UI sliders dynamically based on the incoming music data. This resulted in visuals that appeared to react to the music's energy, intensifying with loudness, shifting with tempo, or warping with bass frequencies, despite the tool not being designed for live audio interaction. This layer of interconnection introduced a semi-synchronous form of agency: music influenced visuals, but the translation was intentionally fuzzy, abstract, and partially machine-determined.

The Output and Agency

Through this hybrid setup, I was able to push the boundary of what agency means in a human–AI collaboration. MusicFX gave the impression of creative delegation sort of like a conductor in a choir where I prompted, and it generated. Autolume invited co-creation like a partner to paint togther with where I guided, and it hallucinated. The introduction of TouchDesigner and OSC allowed for an emergent behavior across systems, where none of the tools alone were explicitly designed for this kind of integration, but together, they enabled a dynamic, responsive audiovisual experience.

A central question remains, when AI tools begin to suggest, shape, or even dominate the outcome, what role does the human creator play? Is the result authored or co-authored? Are we still composing or are we curating what the machine produces? These questions are not only philosophical, but ethical and methodological as well, especially in spaces of art, design, and experience creation.

Toward Interactive Spatial Installations

As a final step in the experiment, I explored the potential of projection mapping the AI-generated visuals onto physical surfaces like walls, windows, or architectural elements. This phase was not just a display method, but an expansion of the work into a site-responsive, spatial experience.

The visuals created in Autolume now partially influenced by live music were projected onto semi-transparent surfaces like translucent window blinds and angled walls. The visual response, when situated in physical space, created a sense of immersion and interactivity even without explicit touch or motion tracking. This offered a glimpse into the potential of using AI-driven, audio-reactive visuals as ambient installations in public or gallery settings.

The next phase could involve combining this approach with body tracking (e.g., Kinect or cameras), allowing participants to influence either the music or visuals through spatial movement—creating a closed feedback loop between sound, space, body, and machine.

This project showed me that AI tools are not merely digital instruments they are creative partners with their own biases and inclinations. By combining MusicFX, Autolume, and TouchDesigner, I was able to orchestrate a real-time, generative loop of music, visuals, and spatial experience. Each tool had its own logic, and my role was often that of a conductor, guiding, responding, and adapting.

Next, I aim to create more tightly coupled systems that involve real-time participant feedback, body tracking, and richer AI models. The goal is to build an installation where the boundaries between user, space, and machine blur into a shared performance of curiosity, interpretation, and emergence.