TwelveLabs

Analyze videos with the TwelveLabs Pegasus model and generate multimodal text embeddings with Marengo.

TwelveLabsTools enable an Agent to understand and search video using TwelveLabs. The toolkit exposes three capabilities:

  • analyze_video: answers a natural-language question about a video using the Pegasus video understanding model.
  • embed_text: generates a multimodal embedding with the Marengo model that lives in the same latent space as TwelveLabs video, audio and image embeddings (useful for searching a video corpus by text).
  • embed_video: embeds a whole video into the same Marengo latent space (one vector per 2–10s segment). Video embedding is asynchronous, so the tool creates an embedding task, polls until it finishes (bounded by embed_timeout), and returns a compact summary of the segmentation (segment count, embedding dimensionality, and each segment's time offsets and scope) rather than the raw float vectors. Because it is long-running, embed_video is opt-in and disabled by default.

Prerequisites

You need to install the twelvelabs library and an API key which can be obtained from the TwelveLabs dashboard.

uv pip install agno twelvelabs openai

Set the TWELVELABS_API_KEY and OPENAI_API_KEY environment variables.

export TWELVELABS_API_KEY=****
export OPENAI_API_KEY=***

Example

The following agent will use TwelveLabs to answer a question about a video and to generate a text embedding.

cookbook/91_tools/twelvelabs_tools.py
from agno.agent import Agent
from agno.tools.twelvelabs import TwelveLabsTools

# Example 1: Enable all tools
agent = Agent(
    tools=[TwelveLabsTools(all=True)],
    markdown=True,
)

agent.print_response(
    "What is happening in this video? https://interactive-examples.mdn.mozilla.net/media/cc0-videos/flower.mp4",
)

# Example 2: Enable only text embedding (useful for embedding search queries
# against a TwelveLabs video index)
embedding_agent = Agent(
    tools=[
        TwelveLabsTools(
            enable_analyze_video=False,
            enable_embed_text=True,
        )
    ],
    markdown=True,
)

embedding_agent.print_response(
    "Embed the text 'a cat playing piano' and tell me how many dimensions it has."
)

# Example 3: Embed a whole video with Marengo (one vector per segment). This is
# asynchronous under the hood — the tool waits for the embedding task to finish.
video_embedding_agent = Agent(
    tools=[
        TwelveLabsTools(
            enable_analyze_video=False,
            enable_embed_text=False,
            enable_embed_video=True,
        )
    ],
    markdown=True,
)

video_embedding_agent.print_response(
    "Embed this video and tell me how many segments and dimensions it has: "
    "https://interactive-examples.mdn.mozilla.net/media/cc0-videos/flower.mp4"
)

Toolkit Params

ParameterTypeDefaultDescription
api_keyOptional[str]NoneThe TwelveLabs API key. Read from the TWELVELABS_API_KEY environment variable if not provided.
analyze_modelstrpegasus1.5The Pegasus model used for analyze_video.
embed_modelstrmarengo3.0The Marengo model used for embed_text and embed_video.
max_tokensint2048Maximum number of tokens for analyze_video responses.
embed_poll_intervalfloat5.0Seconds to wait between status checks while an embed_video task is processing.
embed_timeoutfloat300.0Maximum seconds to wait for an embed_video task to finish before giving up.
enable_analyze_videoboolTrueEnable the analyze_video functionality.
enable_embed_textboolTrueEnable the embed_text functionality.
enable_embed_videoboolFalseEnable the embed_video functionality. Opt-in because video embedding is long-running.
allboolFalseEnable all functionality.

Toolkit Functions

FunctionDescription
analyze_videoAnalyze a video and answer a natural-language question about it using the Pegasus model.
embed_textGenerate a multimodal (Marengo) embedding for a piece of text, which can be used to search a video corpus by text.
embed_videoEmbed a whole video into the Marengo latent space (one vector per 2–10s segment) and return a summary of the segmentation.

Developer Resources