Comparison

AutoGen Instantiation Performance Evaluation

Measure AutoGen AssistantAgent instantiation over 1,000 iterations per metric with PerformanceEval, using a gpt-5.6-luna OpenAIChatCompletionClient and a weather tool.

Demonstrates agent instantiation benchmarking with AutoGen.

This evaluation performs 2,010 callable invocations: 10 warm-ups, 1,000 runtime measurements, and 1,000 memory measurements. It measures construction, not model inference.

PerformanceEval runs the callable separately for each enabled metric: warm-ups first, then runtime measurements, then memory measurements. num_iterations applies to each metric, and the default is 10 additional warm-up calls. Model retries, tools, delegation, and memory extraction can add provider requests beyond the callable count.

Memory measurements use Python’s tracemalloc; they do not measure process RSS, GPU memory, database-server memory, or remote model memory. Record dependency versions, database state, and model settings when comparing results.

autogen_instantiation.py
"""
AutoGen Instantiation Performance Evaluation
============================================

Demonstrates agent instantiation benchmarking with AutoGen.
"""

from typing import Literal

from agno.eval.performance import PerformanceEval
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient


# ---------------------------------------------------------------------------
# Create Benchmark Tool
# ---------------------------------------------------------------------------
def get_weather(city: Literal["nyc", "sf"]):
    """Use this to get weather information."""
    if city == "nyc":
        return "It might be cloudy in nyc"
    elif city == "sf":
        return "It's always sunny in sf"
    else:
        raise AssertionError("Unknown city")


tools = [get_weather]


# ---------------------------------------------------------------------------
# Create Benchmark Function
# ---------------------------------------------------------------------------
def instantiate_agent():
    return AssistantAgent(
        name="assistant",
        model_client=OpenAIChatCompletionClient(
            model="gpt-5.6-luna",
            model_info={
                "vision": False,
                "function_calling": True,
                "json_output": False,
                "family": "gpt-5.6-luna",
                "structured_output": True,
            },
        ),
        tools=tools,
    )


# ---------------------------------------------------------------------------
# Create Evaluation
# ---------------------------------------------------------------------------
autogen_instantiation = PerformanceEval(func=instantiate_agent, num_iterations=1000)

# ---------------------------------------------------------------------------
# Run Evaluation
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    autogen_instantiation.run(print_results=True, print_summary=True)

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U agno "autogen-ext[openai]" autogen-agentchat

Export your API keys

export OPENAI_API_KEY="your_openai_api_key_here"

Run the example

Save the code above as autogen_instantiation.py, then run:

python autogen_instantiation.py

Full source: cookbook/09_evals/performance/comparison/autogen_instantiation.py

Async Function Performance Evaluation

Benchmark an async agent.arun call over 10 iterations per metric with PerformanceEval.arun, printing runtime and memory results.

CrewAI Instantiation Performance Evaluation

Measure CrewAI Agent instantiation over 1,000 iterations per metric with PerformanceEval, using a gpt-5.6-luna llm and a decorated weather tool.