From Silence to Symphony: How We Built an AI-Powered Audio Mixing Engine with FFmpeg 8 and Intelligent Agents

Building an AI-Powered Audio Mixing Engine

Professional meditation audio requires precise management of frequencies, dynamics, and timing. Human audio engineers rely on years of experience to balance voice tracks over background music.

We automated the creation of studio-quality meditation mantras. We mix voice narrations with background music. We built a system that learns and adapts with every successful mix.

The Architecture: Multi-Agent System

We built a multi-agent system using the Vizra ADK. The system orchestrates specialized components:

┌─────────────────────────────────────────────────────────────┐
│                 MantraGenerationWorkflow                    │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐    ┌──────────────────────────────┐   │
│  │ Content Finder  │ →  │  MantraGenerationAgent       │   │
│  │ (FindUnmixed)   │    │  (Gemini 2.5 Flash)          │   │
│  └─────────────────┘    └──────────────────────────────┘   │
│            ↓                           ↓                    │
│  ┌─────────────────────────────────────────────────────┐   │
│  │          IntelligentFFmpegMixerTool                 │   │
│  │  ┌───────────┐  ┌────────────┐  ┌────────────────┐  │   │
│  │  │ Gemini AI │→ │ Vector RAG │→ │ FFmpeg 8 Engine│  │   │
│  │  │ Parameter │  │  Learning  │  │   Execution    │  │   │
│  │  │ Generator │  │   System   │  │                │  │   │
│  │  └───────────┘  └────────────┘  └────────────────┘  │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

The MantraGenerationAgent

Our primary agent acts as an audio engineer. It coordinates the mixing process:

class MantraGenerationAgent extends BaseLlmAgent
{
    protected string $model = 'gemini-2.5-flash';
    protected ?float $temperature = 0.5;

    protected array $tools = [
        FindUnmixedContentTool::class,
        IntelligentFFmpegMixerTool::class,
        VectorMemoryTool::class,
    ];
}

The agent processes high-level instructions. It pairs voice and music content. It consults past successful mixes to adjust parameters based on time-of-day context.

The Intelligence Layer: Gemini and Audio Analysis

Audio Analysis

The system analyzes audio characteristics using FFmpeg probe commands before mixing:

private function analyzeAudio(string $path): array
{
    $command = sprintf(
        'ffprobe -v quiet -print_format json -show_format -show_streams %s',
        escapeshellarg($path)
    );

    return [
        'duration' => (float) ($info['format']['duration'] ?? 0),
        'bitrate' => (int) ($info['format']['bit_rate'] ?? 0) / 1000,
        'channels' => (int) ($info['streams'][0]['channels'] ?? 0),
        'sample_rate' => (int) ($info['streams'][0]['sample_rate'] ?? 44100),
    ];
}

Parameter Generation

We pass audio files and extracted metadata to Gemini 2.0 Flash. The model generates optimized FFmpeg parameters for the specific voice-music pairing:

$response = Prism::structured()
    ->using(Provider::Gemini, 'gemini-2.0-flash-exp')
    ->withPrompt($prompt, [$voiceAttachment, $musicAttachment])
    ->withSchema($mixingParametersSchema)
    ->asStructured();

The Core Mix: FFmpeg 8 Filter Chains

Filter Construction

The system constructs FFmpeg filter chains based on AI recommendations:

private function buildFilters(array $params): string
{
    $filters = [];

    // Voice processing: normalize, gain, convert to stereo
    $filters[] = '[0:a]loudnorm=I=-18:LRA=10:TP=-1.5,volume=1.5dB,aformat=channel_layouts=stereo[voice_norm]';

    // Voice delay
    $delayMs = $params['voice_delay_seconds'] * 1000;
    $filters[] = "[voice_norm]adelay={$delayMs}|{$delayMs}[voice_for_mix]";

    // Music processing: normalize, EQ, gain
    $filters[] = '[1:a]loudnorm=I=-19:LRA=9:TP=-2,' .
                 'highpass=f=80,lowpass=f=12000,' .
                 'equalizer=f=4000:width_type=h:width=800:g=-3,' .
                 'volume=-1dB[music_proc]';

    // The mix with intelligent weighting
    $filters[] = '[music_proc][voice_for_mix]amix=inputs=2:normalize=0:dropout_transition=2:weights=1 1[mix_combined]';

    // Final mastering: fade-in and loudness normalization
    $filters[] = '[mix_combined]afade=t=in:ss=0:d=0.12,loudnorm=I=-15:LRA=8:TP=-2[final]';

    return implode(';', $filters);
}

Time-Context Mixing

The system adapts parameters based on the listening context. Morning mixes use a 3-4 second voice delay with gentle ducking. Night mixes use a 6-8 second delay with strong ducking.

Vector Memory and RAG

Storing Successful Mixes

We use every successful mix as training data. We store the parameters and metadata:

private function storeSuccessfulMix(
    array $params,
    array $voiceInfo,
    array $musicInfo,
    AgentContext $context
): void {
    $description = sprintf(
        'Successful audio mix: Voice duration %.1fs, Music duration %.1fs. ' .
        'Ducking: %s threshold, %.2f ratio, %dms attack, %dms release.',
        $voiceInfo['duration'],
        $musicInfo['duration'],
        $params['ducking_threshold'],
        $params['ducking_ratio'],
        $params['ducking_attack'],
        $params['ducking_release']
    );

    $agent->vector()->addDocument([
        'content' => $description,
        'metadata' => [
            'parameters' => $params,
            'duration_ratio' => $voiceInfo['duration'] / max($musicInfo['duration'], 1),
            'timestamp' => now()->toIso8601String(),
        ],
        'namespace' => 'audio_mixing',
    ]);
}

Retrieval-Augmented Generation

The system queries its vector memory for similar successful mixes before processing new content:

$similarMixes = $agent->rag()->search([
    'query' => "meditation mantra mixing with {$voice->label} and {$music->label}",
    'namespace' => 'audio_mixing',
    'limit' => 3,
    'threshold' => 0.7,
]);

A morning mantra inherits parameters from past morning mixes. Gemini refines the final settings for the specific content.

Quality Assurance

We use LLM judges to assess mix quality and score the output.

Voice Clarity Assertion

class VoiceClarityAssertion extends BaseAssertion
{
    protected function getPrompt(
        string $input,
        string $output,
        ?string $expected = null
    ): string {
        return <<<PROMPT
        Evaluate voice clarity in this meditation mantra mix:

        **Clarity Criteria:**
        1. Voice Prominence (30 points)
        2. Intelligibility (30 points)
        3. Frequency Clarity (20 points)
        4. Mix Balance (20 points)

        Voice should be effortlessly intelligible and feel natural,
        as if narrator is speaking directly to listener with gentle
        ambient music in background.
        PROMPT;
    }
}

Parameter Guardrails

We enforce strict parameter bounds to ensure technical viability:

private const SAFE_VOICE_NORMALIZATION = 'loudnorm=I=-18:LRA=10:TP=-1.5:dual_mono=true:linear=true';
private const SAFE_DUCKING_RATIO = 3;
private const SAFE_DUCKING_THRESHOLD_DB = -22;
private const MIN_VOICE_DELAY_SECONDS = 2;
private const MAX_VOICE_DELAY_SECONDS = 20;

private function enforceParameterSafety(
    array $params,
    array $voiceInfo,
    array $musicInfo
): array {
    $availableIntro = $musicDuration - $voiceDuration - self::MUSIC_TAIL_BUFFER_SECONDS;
    $params['voice_delay_seconds'] = max(
        self::MIN_VOICE_DELAY_SECONDS,
        min(self::MAX_VOICE_DELAY_SECONDS, (int) round($availableIntro))
    );

    $params['ducking_ratio'] = max(3, min(12, $params['ducking_ratio']));
    $params['voice_gain_db'] = max(-2.0, min(3.0, $params['voice_gain_db']));
    $params['music_gain_db'] = max(-4.0, min(1.0, $params['music_gain_db']));

    return $params;
}

Results

We track system performance metrics:

• Processing requires 30-60 seconds per mantra. Manual mixing took 15-30 minutes.
• Automated quality evaluations pass at a 75% rate.
• Human engineers intervene on less than 5% of mixes.

We continue to refine real-time audio analysis and incorporate user feedback loops into the vector memory system.