Audio Streaming

Stream pcm16 audio from gpt-audio and write the chunks to a WAV file while printing the transcript.

audio_streaming.py
"""
Audio Streaming
=============================

Audio Streaming.
"""

import base64
import wave
from typing import Iterator

from agno.agent import Agent, RunOutputEvent
from agno.models.openai import OpenAIChat

# Audio Configuration
SAMPLE_RATE = 24000  # Hz (24kHz)
CHANNELS = 1  # Mono (Change to 2 if Stereo)
SAMPLE_WIDTH = 2  # Bytes (16 bits)

# Provide the agent with the audio file and audio configuration and get result as text + audio
# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------
agent = Agent(
    model=OpenAIChat(
        id="gpt-audio",
        modalities=["text", "audio"],
        audio={
            "voice": "alloy",
            "format": "pcm16",
        },  # Only pcm16 is supported with streaming
    ),
)

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    output_stream: Iterator[RunOutputEvent] = agent.run(
        "Tell me a 10 second story", stream=True
    )

    filename = "tmp/response_stream.wav"

    # Open the file once in append-binary mode
    with wave.open(str(filename), "wb") as wav_file:
        wav_file.setnchannels(CHANNELS)
        wav_file.setsampwidth(SAMPLE_WIDTH)
        wav_file.setframerate(SAMPLE_RATE)

        # Iterate over generated audio
        for response in output_stream:
            response_audio = response.response_audio  # type: ignore
            if response_audio:
                if response_audio.transcript:
                    print(response_audio.transcript, end="", flush=True)
                if response_audio.content:
                    try:
                        pcm_bytes = base64.b64decode(response_audio.content)
                        wav_file.writeframes(pcm_bytes)
                    except Exception as e:
                        print(f"Error decoding audio: {e}")
    print()

OpenAI schedules gpt-audio for shutdown on January 20, 2027. It remains available until that announced date. See the provider lifecycle notice and verify audio formats and streaming support when selecting a replacement.

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U agno openai

Export your OpenAI API key

export OPENAI_API_KEY="your_openai_api_key_here"

Create the output directory

Create the directory used for the WAV file:

python -c "from pathlib import Path; Path('tmp').mkdir(parents=True, exist_ok=True)"

Run the example

Save the code above as audio_streaming.py, then run:

python audio_streaming.py

Full source: cookbook/02_agents/12_multimodal/audio_streaming.py