Smolagents Instantiation Performance Evaluation

Measure runtime and memory across 1,000 Smolagents ToolCallingAgent constructions per metric with PerformanceEval.

This evaluation performs 2,010 callable invocations: 10 warm-ups, 1,000 runtime measurements, and 1,000 memory measurements. It measures construction, not model inference.

PerformanceEval runs the callable separately for each enabled metric: warm-ups first, then runtime measurements, then memory measurements. num_iterations applies to each metric, and the default is 10 additional warm-up calls. Model retries, tools, delegation, and memory extraction can add provider requests beyond the callable count.

Memory measurements use Python’s tracemalloc; they do not measure process RSS, GPU memory, database-server memory, or remote model memory. Record dependency versions, database state, and model settings when comparing results.

smolagents_instantiation.py
"""
Smolagents Instantiation Performance Evaluation
===============================================

Demonstrates agent instantiation benchmarking with Smolagents.
"""

from agno.eval.performance import PerformanceEval
from smolagents import InferenceClientModel, Tool, ToolCallingAgent


# ---------------------------------------------------------------------------
# Create Benchmark Tool
# ---------------------------------------------------------------------------
class WeatherTool(Tool):
    name = "weather_tool"
    description = """
    This is a tool that tells the weather"""
    inputs = {
        "city": {
            "type": "string",
            "description": "The city to look up",
        }
    }
    output_type = "string"

    def forward(self, city: str):
        """Use this to get weather information."""
        if city == "nyc":
            return "It might be cloudy in nyc"
        elif city == "sf":
            return "It's always sunny in sf"
        else:
            raise AssertionError("Unknown city")


# ---------------------------------------------------------------------------
# Create Benchmark Function
# ---------------------------------------------------------------------------
def instantiate_agent():
    return ToolCallingAgent(
        tools=[WeatherTool()],
        model=InferenceClientModel(model_id="meta-llama/Llama-3.3-70B-Instruct"),
    )


# ---------------------------------------------------------------------------
# Create Evaluation
# ---------------------------------------------------------------------------
smolagents_instantiation = PerformanceEval(func=instantiate_agent, num_iterations=1000)

# ---------------------------------------------------------------------------
# Run Evaluation
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    smolagents_instantiation.run(print_results=True, print_summary=True)

Run the Example

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Install dependencies

uv pip install -U agno smolagents

Run the example

Save the code above as smolagents_instantiation.py, then run:

python smolagents_instantiation.py

Full source: cookbook/09_evals/performance/comparison/smolagents_instantiation.py