As we build and maintain AI agents, the bottleneck for high quality is how well we understand the impact of the changes we make, and how easily we can iterate on prompts and configurations based on that understanding.
A prompt that performs well during launch can weaken as user behavior changes, tools evolve, policies shift, and models are updated. New edge cases appear in production. Tool descriptions that looked clear in development may become ambiguous under real traffic. This results in teams inspecting traces, adjusting prompts, rerunning a few examples, and redeploying. However, they often do this with limited evidence that the change actually improves the agent across the workflows that matter. They're throwing stuff at the wall, and often even lack the ability to see what sticks.
Amazon Bedrock AgentCore addresses that operational gap with new agent quality optimization capabilities. Currently offered in public preview, these capabilities for continuously improving agent performance through data-driven configuration changes. It uses agent traces to generate improvements and validate them with controlled experiments, and introduces three core capabilities: Recommendations, Configuration Bundles, and A/B testing.
Caylent evaluated AgentCore optimization capabilities through private-beta access before public preview, with very good results. These are useful additions to the AgentCore operating model, especially for teams already building production agents on AWS. The caveat is that it should not be treated as an automatic prompt optimizer, nor does it replace evaluation engineering. Its value is that it makes agent improvement more structured, testable, and operational, while automatically plugging in observability data so you can evaluate improvements.
Executive Takeaways
The agent quality optimization capabilities in AgentCore help teams move from intuition-led prompt edits to an evidence-backed improvement loop: observe behavior, evaluate performance, generate candidate changes, validate offline, test under live traffic, and promote only when the data supports it.
In Caylent’s private-beta evaluation, calibrated scoring showed meaningful improvement in two workloads, and a smaller, more mixed improvement in a third workload created by Caylent for the evaluation. Evaluators were calibrated against workload-specific gold cases, which is what we recommend before teams use them as promotion gates.
The main advantage we have observed with AgentCore is that it enforces a structured optimization loop, a practice Caylent has recommended to our customers since 2022. That loop includes real data, and this is where these new AgentCore capabilities shine: For agents running in AWS's stack (with runtime, Amazon CloudWatch, evaluations), that data is plugged into the loop with next-to-zero effort and is immediately usable.
What These Agent Optimization Capabilities Add
AgentCore connects several parts of AWS's agentic AI ecosystem into a repeatable improvement loop.
The first capability is Recommendations. Recommendations use AI to generate optimized agent configurations from real session traces. Teams point the service at agent traces, specify a target evaluator as the reward signal, and receive an optimized configuration. There are two types of recommendations: system-prompt recommendations, which are meant to improve the system prompt used in the agent's calls to LLMs, and tool-description recommendations, for the descriptions of the tools an agent has access to. Note that recommendations are generated by LLMs and should be reviewed and tested before they are applied.
The second capability is Configuration Bundles. A Configuration Bundle is a versioned, immutable snapshot of dynamic agent configuration, including system prompts, model IDs, tool descriptions, and other key-value pairs that the agent reads at runtime. Bundles decouple configuration from code, enabling teams to change how the agent responds without redeploying application code, provided the runtime is designed to read configuration dynamically.
The third capability is A/B testing. AgentCore A/B testing splits live production traffic between two variants through gateway in AgentCore. Assignment is sticky by runtime session ID, online evaluation scores each session, and the service reports per-evaluator metrics such as mean score, absolute and percent change, p-value, confidence interval, and significance flag. Variants can be different Configuration Bundle versions within the same runtime, or different Gateway targets that point to different runtime endpoints.
Together, those capabilities enable optimization within an operating loop rather than through individual edits.
From Manual Edits to Evidence-Backed Changes
Most teams already have some version of an agent improvement process. They inspect failures, adjust prompts or tool descriptions, run a test set, and deploy a new version. The limitation is often operational discipline.
Common failure modes include promoting a prompt because it fixed a few memorable failures, improving one workflow while silently regressing another, optimizing to a generic score that does not reflect business quality, and changing behavior through code deployments when configuration would be cleaner.
AgentCore addresses these gaps by making the path to improvement explicit:
- Recommendations generate candidate changes from observed traces.
- Configuration Bundles make prompt, tool-description, model, and runtime configuration changes versionable.
- Offline evaluation can test candidates before live exposure.
- Gateway A/B testing can compare variants under production traffic patterns.
AgentCore helps teams generate and validate candidates inside the AWS-native environment where the agent runs.
The Importance of Configuration Bundles
In many agent systems, prompts and tool descriptions live too close to application code. Changing a system prompt may require a new deployment. Testing two prompt variants may require separate runtime builds or custom routing. Rolling back behavior may require reverting code instead of reverting the configuration.
Configuration Bundles create a cleaner separation. A bundle version can hold the dynamic configuration that controls agent behavior. The deployed runtime can stay the same while different requests receive different configuration versions.
This pattern matters for three reasons. First, it makes experiments cleaner. When the change is configuration-only, a control and treatment can run through the same runtime with different bundle versions.
Second, it makes rollbacks cleaner. Because bundle versions are immutable, teams can reference a previous version if a candidate underperforms.
Third, it creates configuration lineage. Bundle versions form a history of behavior-changing configuration. That history is useful for understanding what changed and when. However, this should not be confused with full AWS CloudTrail audit coverage. At the time of writing, AWS documentation states that the preview APIs of these agent optimization capabilities in AgentCore do not support AWS CloudTrail and should not be used for workloads requiring an AWS CloudTrail audit trail until support is added.
Configuration Bundles are optional. You can validate changes by packaging them as a bundle version or by deploying a separate runtime endpoint. Using different endpoints as targets is appropriate when the change includes code changes, a framework upgrade, or an entirely different agent implementation. For changes that can be managed through configuration, we recommend decoupling them from the code and moving them into Configuration Bundles.
Pre-Requisites for the Agent Optimization Loop
Recommendations and A/B testing have the same agent requirements as AgentCore evaluations: an agent deployed on AgentCore runtime with observability enabled, or an agent built with a supported framework configured with AgentCore’s end-to-end traceability. The documented supported frameworks are Strands Agents and LangGraph with OpenTelemetry or OpenInference instrumentation. The prerequisites also include CloudWatch Transaction Search, telemetry in CloudWatch Logs, current SDK or CLI versions, and IAM permissions for the optimization features.
How Caylent Evaluated AgentCore Optimization Capabilities
Caylent evaluated these new optimization capabilities via private beta access before the public preview. Our goal was to answer a practical implementation question: Can this workflow generate useful candidate improvements, and what evaluation discipline is required before teams should promote those changes?
We tested across three generated evaluation workloads:
- A retail-support sample workload
- An independent, Caylent-generated retail-support holdout set
- A Caylent-created passenger-rail support workload
Each generated case defined expected facts, expected actions, prohibited claims, and prohibited actions. We scored paired baseline/control and treatment responses across three response replicates per condition. The mean score weighted fact coverage, action correctness, and prohibited-avoidance behavior. Confidence intervals were computed using paired bootstrap over case-level means.
These datasets were generated for the evaluation. They are not public benchmarks, and the results should not be read as universal claims about every agent or every domain.
The Results
Across all three workloads, the treatment improved the mean score. The retail-support sample showed the strongest result: the mean score improved from 0.856 to 0.962, and the primary pass rate improved from 71.4% to 92.2%. The independent retail-support holdout confirmed the same direction: mean score improved from 0.849 to 0.926, and primary pass rate improved from 64.6% to 78.8%.
The passenger-rail workload also showed a positive mean-score movement, from 0.715 to 0.759, but the result was more mixed. Primary pass-rate lift was inconclusive because the confidence interval crossed zero, and case-level results included both improvements and regressions.
Guille Ojeda
