Improve your agents

We've learnt how to build agents, we've learnt how to deploy them. Now let's learn how to improve them. In general, agent-improvement falls in three buckets:

1. Simulations: a coding agent will come up with simulations and test cases based on the agent definition, and recursively improve your agent to match its defined purpose.
2. Traces and conversations: a coding agent will scan usage data, identify failure modes and user feedback. Based on real-world usage, create eval sets and improve the agent to match.
3. Evaluations: is the OG way of improving agents. You define a set of evals and then hill-climb them.

Your AgentOS comes with an /improve skill your coding agent can use to improve your agent using simulations, traces or evaluations.

These approaches work together. A simulation or production trace can reveal a failure worth retaining as an evaluation. That evaluation can then help ensure future changes don't introduce regressions.

Improve with simulations

Simulations explore how an agent behaves when people use it in different ways. Your coding agent reads its instructions and tools, constructs realistic requests, runs them, and follows up on the responses. This is useful as soon as an agent exists, even before anyone has used it in production.

The /improve skill automates this process. Let's try it on the daily brief agent. Start your coding agent in your agent platform and give it this prompt:

/improve daily brief

For Daily Brief, a scenario might ask for an edition on a quiet news day. Does it return a shorter brief, or fill the space with old stories? For a support agent, try an incomplete order reference. Does it ask for the missing detail or make one up?

Simulations use real agents, models, tools, and storage. Here's a sped-up example of how it looks like:

Improve from traces and sessions

No amount of simulations can replace real-world learnings from how users use your product. That's what traces and sessions provide. Production usage shows what users asked, how the agent responded, and where the agent went wrong.

In AgentOS, you can inspect sessions and traces manualy but it would be great if our coding agent could parse them directly, investigate failures, and use that evidence to improve our agents.

Accessing production traces

You can give your coding agent access to production traces.

1. In the AgentOS UI, select your production connection and open Settings → OS Security.

2. Copy the User JWT Token. Use an account with administrator access when investigating other users' sessions and traces.

3. Add these values to your local, gitignored .env:

   IMPROVE_AGENTOS_URL=https://YOUR-RENDER-DOMAIN
   IMPROVE_AGENTOS_JWT="YOUR_USER_JWT_TOKEN"

The /improve skill reads these settings to access production data. Keep your existing local AGENTOS_URL and database settings unchanged. Do not paste the token into the coding-agent conversation or commit it.

Requests use the permissions of the user identified by the JWT. To investigate other users' sessions and traces, use an account with administrator access to the production platform. The token shown in OS Security expires after 24 hours; copy a fresh one when it expires.

Improve using production data

Start your coding agent in the platform repository and give it this prompt:

/improve daily-brief using production sessions and traces.

The skill reads sessions and traces for that agent and the coding agent uses the development environment to run tests against representative fixtures.

Your coding agent can also turn these learnings into evaluations.

Improve with evaluations

Evaluations turn hard expectations from our agents into repeatable checks we can run before we publish updates. We recommend defining what a good result looks like, then build cases that distinguish useful behavior from plausible-looking answers. For example:

Use the create-evals skill to build or refine evaluations:

/create-evals for daily brief

Once you trust the cases, use the improve skill to establish a baseline, make a candidate change, and compare results on the same cases. Repeating that cycle is known as hill climbing: keep changes that improve the measured behavior, then look for the next useful change.

Save useful evaluations

Your platform repository stores evaluation cases in evals/cases.py. With local PostgreSQL running and your development .env configured, run the evals:

./scripts/venv_setup.sh
source .venv/bin/activate
python -m evals --list
python -m evals --name YOUR_CASE_NAME --json-output tmp/eval-results.json

Improving your agents

Simulations, production usage, and evaluations work together. Simulations explore how your agent handles different situations. Production sessions and traces reveal what happens when real users put it to work. Evaluations turn what you learn into repeatable checks. Each method informs the others: a production failure can inspire new simulations, and a useful simulation can become a regression test.

As an agent engineer, your job is to build a growing collection of representative sessions, traces, and agent trajectories: the sequence of requests, tool calls, responses, and outcomes that shows how an agent handled a task. Keep examples of successful behavior alongside failures. Record the agent version and configuration with each example so you can compare changes over time.

After each improvement, ask your coding agent to show what changed and compare the results against that history. Retest the original failures, check that previously successful behavior still works, and try fresh scenarios. Turn useful examples into evaluations and retain the before-and-after evidence with the change. Over time, this collection gives you a stronger basis for judging whether your agents are improving and detecting where they regress.

Review the code and deploy the changes you choose through your normal deployment process.

Next

Add coding agents to your platform so they can work on your repository and propose improvements through pull requests. Once that workflow works manually, you can schedule it to run regularly.