OpenAI

OpenAITools allow an Agent to interact with OpenAI models for performing audio transcription, image generation, and text-to-speech.

The current toolkit's image_model defaults to the retired dall-e-3. Set image_model="gpt-image-2" for the current OpenAI image model.

Prerequisites

Before using OpenAITools, ensure you have the openai library installed and your OpenAI API key configured.

Install dependencies:
```
uv pip install agno -U openai
```
Set your API key: Obtain your API key from OpenAI and set it as an environment variable.
export OPENAI_API_KEY=xxx

Initialization

Import OpenAITools and add it to your Agent's tool list.

from agno.agent import Agent
from agno.tools.openai import OpenAITools

agent = Agent(
    name="OpenAI Agent",
    tools=[OpenAITools(image_model="gpt-image-2")],
    markdown=True,
)

Usage Examples

1. Transcribing Audio

The agent transcribes an audio file:

transcription_agent.py

from pathlib import Path
from agno.agent import Agent
from agno.tools.openai import OpenAITools
from agno.utils.media import download_file

audio_url = "https://agno-public.s3.amazonaws.com/demo_data/sample_conversation.wav"

local_audio_path = Path("tmp/sample_conversation.wav")
download_file(audio_url, local_audio_path)

agent = Agent(
    name="OpenAI Transcription Agent",
    tools=[OpenAITools(transcription_model="whisper-1", enable_image_generation=False)],
    markdown=True,
)

agent.print_response(f"Transcribe the audio file located at '{local_audio_path}'")

2. Generating Images

The agent generates an image from a text prompt:

image_generation_agent.py

import base64

from agno.agent import Agent
from agno.tools.openai import OpenAITools
from agno.utils.media import save_base64_data

agent = Agent(
    name="OpenAI Image Generation Agent",
    tools=[OpenAITools(image_model="gpt-image-2")],
    markdown=True,
)

response = agent.run("Generate a photorealistic image of a cozy coffee shop interior")

if response.images:
    image_base64 = base64.b64encode(response.images[0].content).decode("utf-8")
    save_base64_data(image_base64, "tmp/coffee_shop.png")

3. Generating Speech

The agent generates speech from text:

speech_synthesis_agent.py

import base64

from agno.agent import Agent
from agno.tools.openai import OpenAITools
from agno.utils.media import save_base64_data

agent = Agent(
    name="OpenAI Speech Agent",
    tools=[OpenAITools(
        text_to_speech_model="tts-1",
        text_to_speech_voice="alloy",
        text_to_speech_format="mp3",
        enable_image_generation=False
    )],
    markdown=True,
)

response = agent.run("Generate audio for the text: 'Hello, this is a synthesized voice example.'")
if response and response.audio:
    audio_base64 = base64.b64encode(response.audio[0].content).decode("utf-8")
    save_base64_data(audio_base64, "tmp/hello.mp3")

View more examples here.

Customization

You can customize the underlying OpenAI models used for transcription, image generation, and TTS:

OpenAITools(
    transcription_model="whisper-1",
    image_model="gpt-image-2",
    text_to_speech_model="tts-1-hd",
    text_to_speech_voice="nova",
    text_to_speech_format="wav"
)

Toolkit Params

Parameter	Type	Default	Description
`api_key`	`Optional[str]`	`None`	OpenAI API key. Uses OPENAI_API_KEY env var if not provided
`enable_transcription`	`bool`	`True`	Enable audio transcription functionality
`enable_image_generation`	`bool`	`True`	Enable image generation functionality
`enable_speech_generation`	`bool`	`True`	Enable speech generation functionality
`all`	`bool`	`False`	Enable all tools when set to True
`transcription_model`	`str`	`whisper-1`	Model to use for audio transcription
`text_to_speech_voice`	`str`	`alloy`	Voice to use for text-to-speech (alloy, echo, fable, onyx, nova, shimmer)
`text_to_speech_model`	`str`	`tts-1`	Model to use for text-to-speech (tts-1, tts-1-hd)
`text_to_speech_format`	`str`	`mp3`	Audio format for TTS output (mp3, opus, aac, flac, wav, pcm)
`image_model`	`str`	`dall-e-3`	Model to use for image generation
`image_quality`	`Optional[str]`	`None`	Quality setting for image generation
`image_size`	`Optional[str]`	`None`	Size setting for image generation
`image_style`	`Optional[str]`	`None`	Style setting for image generation (vivid, natural)

Toolkit Functions

The OpenAITools toolkit provides the following functions:

Function	Description
`transcribe_audio`	Transcribes audio from a local file path
`generate_image`	Generates images based on a text prompt
`generate_speech`	Synthesizes speech from text