I just published an article breaking down Gemini Audio Engineer, my new open-source system that bridges native Apple Digital Audio Workstation (DAW) processing with artificial intelligence. Here is a quick look at what the article covers: Tackling "Mixing Fatigue": How traditional AI audio utilities act as offline black boxes, whereas Gemini Audio Engineer runs as a native AUv3 plugin directly inside Logic Pro to analyze and alter live audio chains in real time. Three Operational Modes: Engineer Mode: Provides objective, consultative acoustic audits and mel-spectrogram breakdowns (low-mid mud, high-end air, dynamic headroom) without touching your plugin settings. 🎹Producer Mode: Detects track key and BPM (via Chordino) to generate context-aware musical ideas and drag-and-drop .mid files directly onto DAW tracks. Executor Mode: Automatically configures a real-time 8-band parametric EQ, VCA compressor, de-esser, tape saturation, algorithmic reverb, and stereo widening engine with a single click. Iterative Refinement ("Re-Analyze"): How the AI captures post-DSP audio to critique its own processing choices and fine-tune its settings across multiple passes. DAW Automation Sync: Exposing 32 discrete AUParameter nodes directly to Logic Pro so creators can draw AI parameter automation curves directly on their timeline. Deep Stem Isolation: Utilizing Demucs background processing to isolate 4-stem audio tracks (vocals, drums, bass, keys) and transcribe audio to MIDI. Hybrid Privacy-First Architecture: Combining a zero-latency Swift/CoreAudio frontend with a local Python/FastAPI backend
Most AI utilities in music production function as offline black boxes. I wanted to build an intelligent assistant that operates inside Logic Pro as a native Apple Audio Unit (AUv3) component, interacting directly with real-time DSP audio threads and host automation lanes.
Meet Gemini Audio Engineer—a two-part open-source system bridging native C/Swift DSP execution with a Python/FastAPI AI engine.
SYSTEM ARCHITECTURE & AUDIO THREAD ENGINEERING
Zero-Latency AUv3 Frontend (GeminiEngineKit): • Audio Thread Real-Time Safety: The core signal loop (processChannelBuffer) is built using Apple's Accelerate framework (vDSP/vForce). To ensure glitch-free playback in DAWs like Logic Pro, the audio rendering loop avoids heap allocations, thread locks, and Objective-C dynamic messaging. • Ring Buffering & Async Capture: Incoming PCM audio passes through a lock-free circular buffer (RingBuffer.swift). Audio snippets (2–30s) are packaged into base64 WAV buffers and dispatched to the local backend asynchronously without blocking the DAW's render thread. • 32-Node AUParameter Sync: Every parameter controlled by the AI is registered with an AUParameterTree. This exposes 32 discrete AUParameter nodes directly to the host DAW, enabling timeline automation for AI-generated EQ gains, dynamic thresholds, and reverb wet/dry ratios. Local Python/FastAPI AI Engine (127.0.0.1:8000): • Multimodal Perception & Feature Extraction: The FastAPI server computes mel-spectrogram arrays using librosa and extracts key/BPM descriptor arrays via Chordino chromagram analysis. • Structured Payload Generation: The AI model (Google Gemini Pro/Flash or OpenAI GPT-4o Audio) evaluates audio features alongside optional user prompts and formulates a validated JSON parameter payload (DSPActionsPayload). • Asynchronous Heavy Processing: Long-running jobs—such as 4-stem audio separation using Demucs or audio-to-MIDI transcription via basic-pitch—run on background worker threads with a non-blocking job polling loop (JobPoller.swift).
CUSTOM SWIFT DSP SIGNAL CHAIN (DSP/)
• 8-Band Parametric Equalizer: Implemented with Direct Form II transposed biquad filters (BiquadFilter.swift), supporting peak, low/high shelving, and notch responses. • VCA Dynamic Compressor & De-Esser: Features logarithmic envelope followers and peak detection algorithms to drive smooth VCA gain reduction curves. The de-esser runs a dedicated high-frequency sidechain filter (5–10 kHz). • Harmonic Saturation: Multi-mode saturation engine including soft-clipping hyperbolic tangent (tanh) functions, analog tape emulation, and hard clipping. • Spatial & Reverb Processing: Mid/Side (M/S) matrix processing for phase-coherent stereo widening (1.0x to 1.10x+), paired with an algorithmic Schroeder/Freeverb implementation.
🔄 THE ITERATIVE REFINEMENT LOOP ("Re-Analyze")
When operating in Executor Mode, the system supports a closed-loop control cycle:
The AI captures unprocessed PCM audio and applies an initial DSP action payload. The producer hits Re-Analyze, causing the plugin to capture audio post-DSP. The AI evaluates its own acoustic corrections, fine-tuning parameter values across 2–3 passes until spectral balance and dynamic control are achieved.
🙏 OPEN-SOURCE ATTRIBUTION
Special thanks to Anthony Galati for authoring the original foundational architecture, and Maui-B for core contributions to the backend audio processing pipeline.
#Swift #DSP #AudioEngineering #CoreAudio #LogicPro #FastAPI #Python #MachineLearning #GoogleGemini #OpenSource #SoftwareArchitecture #AppleDeveloper
đź“‚ Open-Source Repositories & Links:
• Swift AUv3 Plugin Frontend: https://github.com/dschmidtadv/gemini-au-plugin
Star the repo on GitHub and let me know if you run into any setup issues! 🚀
Comments