If you’ve ever produced music or worked with audio, you know the feeling of "mixing fatigue." After looping the same four-bar chorus for six hours, your ears get shot. Your brain starts compensating for muddy bass, harsh cymbals, and clashing frequencies—and you completely lose acoustic objectivity. Traditionally, the only fix was to step away and hope for fresh perspective tomorrow. Over the past few months, we've been building a solution: **Gemini Audio Engineer** 🎛️🤖 ### 🧠 What is it? It’s a native Apple Audio Unit (AUv2) plugin for Logic Pro that places a multimodal AI audio engineer directly in your digital audio workstation (DAW). Think of it as having a seasoned mixing consultant and studio co-pilot sitting in the control room next to you. ### ⚙️ How does it work? 1. **It Listens:** It captures audio directly from your DAW timeline alongside high-resolution Mel-spectrograms. 2. **It Analyzes:** Powered by Google Gemini, it diagnoses frequency masking, dynamic buildup, and tonal balance issues (e.g., "The lead synth is cluttering 300 Hz and masking the vocal"). 3. **It Acts (No Magic Black Boxes):** Instead of slapping a static generic preset on your track, it formulates an exact mixing plan and programmatically sculpts a built-in pro DSP signal chain—adjusting an 8-band parametric EQ, VCA compressor, de-esser, analog tape saturation, and reverb with one click. 4. **You Stay in Control:** Every parameter the AI touches is exposed as standard automation nodes in Logic Pro, so you can tweak every knob or draw automation curves across your song. ### 🚀 What makes the engineering unique? • **Real-Time Safe:** A hybrid architecture combining a C++ JUCE 7 DSP core (zero latency, no audio dropouts) with an embedded SwiftUI interface. • **Mix Context Awareness:** Track plugins can communicate with the master bus, so the AI understands the *entire song* while sculpting only the *individual instrument*. • **Zero Overhead:** A custom macOS service that uses 0% CPU and 0 MB RAM until you open the plugin window in your DAW. We’re bridging the gap between cutting-edge multimodal AI and low-level audio DSP. Curious to hear from any producers, audio engineers, or AI builders in my network: How would you use a digital assistant in your creative workflow? #MusicProduction #AudioEngineering #LogicPro #GoogleGemini #ArtificialIntelligence #SoftwareEngineering #DSP
When we built the first Swift-based prototype of Gemini Audio Engineer, we proved something wild was possible: placing a multimodal AI model directly inside Logic Pro to diagnose audio and make mixing decisions.
But moving from a promising prototype to a zero-latency, production-grade audio plugin revealed tough real-world engineering realities:
❌ Real-time audio threads demand zero heap allocations and lock-free execution.
❌ Single-track plugins are inherently "deaf" to the rest of the song.
❌ Nobody wants a Python AI backend eating CPU and RAM in the background 24/7 while producing music.
Over the past few weeks, we completely re-architected the entire ecosystem. Here is everything we’ve engineered and shipped since that first Swift release:
---
⚡ 1. Migrated the Audio Core to Real-Time Safe C++ & JUCE 7
We moved the entire DSP engine from Swift to C++ using JUCE 7 and CMake (now an AUv2 component validated by Apple’s auval):
• 100% custom, zero-latency DSP core: 8-band parametric biquad EQ, VCA compressor with live Gain Reduction metering, analog-modeled saturation (Tape/Soft/Hard Clip), algorithmic Schroeder reverb, de-esser, and mid/side stereo imager.
• Atomic parameter smoothing (CurvePlayer) eliminates zipper noise and pops when the AI applies sudden mix changes.
• Lock-free, real-time safe audio thread with zero memory allocations during playback.
🎨 2. Native SwiftUI Embedded Directly inside the JUCE AU Window
You no longer need a separate helper window or companion app. We built a custom bridge embedding native SwiftUI (NSHostingView) directly inside the native JUCE Audio Unit editor:
• Fluid, hardware-accelerated UI with animated EQ curves and responsive meters.
• Bidirectional sync: Tweaking sliders in the Mix tab instantly updates the C++ DSP parameters, and when the AI applies a mix, all 32+ physical sliders animate to reflect the AI’s settings in real time.
🧠 3. The Holy Grail: Inter-Plugin Mix Context Awareness
The age-old problem in mixing: individual track plugins can’t hear the full mix, while master bus plugins can only EQ everything at once.
Using macOS shared App Group IPC, we built dual-stream context awareness:
• Drop an instance on the Master Bus as the "Master Broadcaster."
• Drop instances on individual instrument tracks (Vocals, Synth, Bass) ➔ they auto-link to the song's master reference.
• Google Gemini evaluates BOTH streams simultaneously. It detects frequency masking and mud across the entire song, but applies surgical DSP moves strictly to the active track stem (e.g., carving 2.5 kHz on a synth to unmask vocals without thinning out the rest of your track).
🔋 4. Zero-Overhead On-Demand Service (launchd PathState)
You no longer have to manually open a terminal or run virtual environments:
• 0 MB RAM & 0% CPU while Logic Pro is closed.
• A custom macOS LaunchAgent detects when the plugin GUI opens, auto-spins up the local FastAPI backend in ~1–2 seconds, and auto-shuts down cleanly the instant you close the window or quit your DAW.
🎛️ 5. Expanded to 5 Operating Modes
Beyond consultative auditing (Engineer), arrangement & MIDI generation (Producer), and automated hands-on mixing (Executor), we introduced:
• 🧭 Assistant Mode: Non-destructive planning & strategy passes.
• 🎬 Conductor Mode: Performance-aware, section-sensitive dynamic decisions.
🛡️ 6. Hardened Capture Engine & Robust Audio Fallbacks
• Rock-solid lock-free ring buffer tap directly in the DAW signal path.
• Eliminated watchdog timeouts and added graceful fallbacks to native AI audio understanding.
---
Big thanks to Anthony Galati (@adgalati) and Maui-B whose foundational open-source work inspired this monorepo journey.
AI in music shouldn't be a generic "one-click preset" black box. By bridging C++ audio engineering, native macOS UI, and multimodal LLMs, we're building a true digital co-pilot for the control room.
If you had a world-class mixing engineer sitting in your DAW 24/7, what’s the first track you’d have them listen to? 👇
#AudioTech #DSP #MusicProduction #LogicPro #JUCE #SwiftUI #GoogleGemini #Cpp #ArtificialIntelligence #AudioEngineering #IndieHacker
Comments