Guille Ojeda
Caylent Catalysts™
Generative AI Strategy
Accelerate your generative AI initiatives with ideation sessions for use case prioritization, foundation model selection, and an assessment of your data landscape and organizational readiness.
Automating Prompt Iterations with Amazon Bedrock Advanced Prompt Optimization
Generative AI & LLMOps
Explore how Amazon Bedrock Advanced Prompt Optimization can automate evaluation-driven prompt refinement, and see what we learned from our hands-on experiment about model-specific improvements, prompt tradeoffs, and the validation needed before putting optimized prompts into production.
Prompt optimization is one of the most effective levers available to improve the performance of AI agents. At this stage, Large Language Model (LLM) Evaluations should be a well-known solution for measuring the outputs of models and agents and how your changes affect them, and you should have implemented your own evaluations (evals) for your agentic AI solutions. But even with an automated feedback cycle, iterating on a prompt to find a candidate improvement that scores better on your evals requires significant effort.
On May 14, 2026, AWS introduced Amazon Bedrock Advanced Prompt Optimization to turn that manual work into an automated, evaluation-driven job. You first give it a prompt template, samples, an evaluation method, and one or more target models. Bedrock then invokes each model, scores its responses, analyzes the failures, and returns a model-specific candidate prompt with the evidence produced during optimization.
At Caylent, we tested the feature with a bill-of-lading extraction workload: one prompt, 50 evaluation records, a 29-field JSON contract, and a custom Lambda evaluator across Amazon Nova 2 Lite, Anthropic Claude Sonnet 4.5, and Anthropic Claude Haiku 4.5. Every candidate prompt scored higher under our evaluator. Every candidate also became longer, and one field regressed across all three runs.
In this blog, we'll discuss how Amazon Bedrock Advanced Prompt Optimization works, and the structured-document extraction experiment we ran to test it.
Simple and Advanced Prompt Optimization
Amazon Bedrock provides two prompt-optimization workflows. The best option for you depends on whether you need a fast rewrite or an evaluation-backed comparison against a workload.
If you want to improve a prompt without changing the model, select a single target model for optimization. To migrate a prompt to another model, use the current Amazon Bedrock model as the baseline and select up to four Bedrock candidate models. The targets must be Bedrock models that produce text, although evaluation samples can include images and PDFs when the selected model supports them. For a direct comparison with a non-Bedrock baseline, run that model separately and score its outputs with the same Lambda evaluator used for the Bedrock target models, so you can compare results. The service workflow describes this external comparison path.
Evaluation Methods for Advanced Prompt Optimization
The evaluator determines which candidate is better, but your team still has to define what “better” means. The available methods encode different levels of precision and create different review requirements.
Each template uses one method, although templates in the same job may use different methods. AWS documents all four configurations.
For our experiment, we chose a custom Lambda evaluator because it gave us direct control over the extraction objective. Our function returned a 0 to 1 score per record. Fields that are Critical, High, Moderate, and Low received weights of 4, 3, 2, and 1, and type-aware comparisons normalized acceptable variations and awarded partial credit. A correct null earned half of that field's normal weight, while missed, invented, or incorrect values earned no field credit.
Missing or unexpected JSON keys reduced the score. Recoverable JSON wrapped in prose or code fences incurred a 5% penalty, and an unparseable response scored zero. Diagnostics identified weak, missed, and hallucinated fields. The overall custom score was the arithmetic mean of the 50 record scores. Note that evaluator clarity and independent tests are part of the optimization design because Amazon Bedrock Advanced Prompt Optimization uses the function source, docstrings, scores, and diagnostics to analyze failures.
What Advanced Prompt Optimization Can Rewrite
The prompt template is the unit we're optimizing. We've attached the original prompt template we used at the end of this article, along with the three candidates the optimization produced. You'll notice there are {{placeholder}} variables, which are used to insert sample-specific content. When using the prompt for inference, those variables are replaced with real values, which for optimization, are taken from the samples.
Here is a simplified version of the initial prompt template we used in our experiment:
<background>
<advpo:optimize>
You are an expert in bills of lading and can extract key information from them. A bill of lading is a legal document describing the type, quantity, and destination of goods being shipped.
</advpo:optimize>
</background>
<task>
<advpo:optimize>
Convert the unstructured OCR text into a JSON object using the attributes below. Return null when a field is absent or uncertain; do not invent values.
</advpo:optimize>
<attributes>
[
{
"Field": "Delivery Date",
"Description": "<advpo:optimize>The date when the shipment is expected to arrive at the destination. May also be referred to as Expected Delivery Date or ETA. Must be in MM/DD/YYYY format.</advpo:optimize>",
"Alias": "ETA, Estimated Delivery Date, Delivery ETA, Consignee Signature Date, Delivery Date, Expected Delivery",
"DataType": "Date",
"Format": "MM/DD/YYYY",
"Priority": "Moderate"
},
{
"Field": "POD",
"Description": "<advpo:optimize>Proof of Delivery indicator or flag. May appear as TRUE/FALSE, 0/1, or another indicator of delivery confirmation.</advpo:optimize>",
"Alias": "Proof of Delivery, POD, Delivery Proof, Delivered",
"DataType": "String",
"Format": "Boolean-like values (TRUE/FALSE, 0/1, Y/N)",
"Priority": "Critical"
},
{
"Field": "Quantity",
"Description": "<advpo:optimize>The total number of units in the shipment, rather than an individual line-item quantity.</advpo:optimize>",
"Alias": "Total Pallets, Total Units, Total, Total Pieces, Unit Count",
"DataType": "Numeric",
"Format": "Integer",
"Priority": "High"
},
{
"Field": "Quantity UOM",
"Description": "<advpo:optimize>The unit used to quantify the shipment, such as pieces, pallets, crates, or skids.</advpo:optimize>",
"Alias": "Pallets, Cases, Pieces, Boxes, Bales, PLT, CRATES, SKID, Units",
"DataType": "String",
"Format": "Unit descriptor, may be abbreviated",
"Priority": "High"
},
{
"Field": "Ship Date",
"Description": "<advpo:optimize>The date when the goods leave the origin. It may also appear as the Pickup Date, Dispatch Date, or simply Date. Must be in MM/DD/YYYY format.</advpo:optimize>",
"Alias": "Pickup Date, Order Date, Dispatch Date, Load Date, Ship Date, Date",
"DataType": "Date",
"Format": "MM/DD/YYYY",
"Priority": "Critical"
},
{
"Field": "Signature Count",
"Description": "<advpo:optimize>The total number of visible signatures, including driver, receiver, and other party signatures.</advpo:optimize>",
"Alias": "Signatures, Number of Signatures, Signature Count",
"DataType": "Numeric",
"Format": "Integer (0 or positive)",
"Priority": "Critical"
}
]
</attributes>
<advpo:optimize>
Extraction guidance: Return every field key listed above exactly once, even when its value is null. Prefer explicitly labeled values over inferred ones.
</advpo:optimize>
<advpo:optimize>
Ensure that all dates use MM/DD/YYYY. Return null if an attribute or its alias is not found. Do not add quantities or weights when the document does not provide a total.
</advpo:optimize>
Return only the JSON object, without comments or additional text.
</task>
<cache_marker>
<document_ocr_data>
{{documentText}}
</document_ocr_data>By default, the optimizer can edit the whole template, but you can use selective optimization to scope the rewrite to specific sections marked with <advpo:optimize>and <advpo:exclude> tags, as we did above. In our case, we tagged most extraction instructions and each field description. Field names, aliases, data types, formats, and priorities stayed outside the editable blocks.
With <advpo:optimize>, only tagged content can change. If a template contains only <advpo:exclude> blocks, the tags freeze that content and leave everything else editable. Placeholders aren't changed, and Amazon Bedrock automatically removes the tags from the generated candidates.
Notice that for each field we only tagged the value of Description with <advpo:optimize>. This allowed the optimizer to rewrite only the description while preserving the literal text of the field name, alias, data type, format, and priority. All three resulting prompt candidates preserved every field name, alias, data type, format, and priority.
We still had issues with some fields. POD was a Boolean-like field indicating whether the document contained evidence that the shipment had been received. It was separate from Signature Count, which recorded the number of filled signature positions. In one of our reference responses, for example, a document with one driver signature had Signature Count="1" but POD="0". This should have meant that signature does not mean confirmation that the consignee had received the shipment. Optimizing the prompt unintentionally changed that rule. The Sonnet candidate, for example, added several delivery-evidence rules and ended with:
IMPORTANT: If Signature Count is 1 or greater, POD must be '1'.
That shortcut conflicted with the reference described above. For the record with one driver signature and POD="0", the Sonnet candidate returned POD="1".
The tags protect the literal text outside of them, but not the semantics. Selective optimization reduces the parts you need to review, but edits made around protected text can still change how the model interprets fixed instructions.
How Advanced Prompt Optimization Worked in Caylent's Extraction Job
Our extraction job mapped to the workflow as follows:
- Define the template. Ours converted bill-of-lading OCR from
{{documentText}}into a 29-field JSON object. - Prepare samples. We supplied 50 OCR records with reference JSON.
referenceResponseis optional in the service schema, but we highly recommend including it since it's the field that tells the evaluator what the correct response should be for that input, allowing it to perform ground-truth comparison and produce better optimization results. - Choose an evaluation method. We selected a custom Lambda evaluator so we could encode the schema, partial-credit rules, business priorities, and output penalties directly.
- Select models and settings. Across two jobs, we targeted Nova 2 Lite, Claude Sonnet 4.5, and Claude Haiku 4.5.
- Run the loop. Amazon Bedrock rendered the records, invoked each model, scored the outputs, analyzed diagnostics, rewrote the editable text, and re-evaluated it.
- Inspect results. Each model received its own template, record-level scores, token and latency data, and cost estimates.
This is a simplified version of the inputs we used, with real values substituted with synthetic data:
CEMETERY CONSTRUCTION
BILL OF LADING
Ticket #: TKT-4182
Pickup date: 9/14/2025
Job #: JOB-782
Delivery date: 9/14/2025
Pickup location:
Blue Cedar Precast
1847 Cedar Ridge Road
Northbridge, TX 75001
Contact: Casey Rowan
Phone: 202-555-0147
Destination:
Evergreen Memorial Gardens
9200 Harbor Point Drive
Westhaven, TX 75002
Contact: Morgan Reed
Phone: 202-555-0182
Product/Quantity:
8
CAST STONE BASE
Load requirements:
Load: 8:00 am to 3:00 pm
Unload: 8:00 am to 3:00 pm
Truck #: TRK-284
Trailer #: TRL-719
Dispatching company: Northstar Freight Services
Driver company: Northstar Freight Services
Driver name: Jordan Hale
Driver signature: Jordan HaleThese are the fields from the response relevant to this example, with their values also substituted:
{
"Delivery Date": "09/14/2025",
"POD": "0",
"Quantity": 8,
"Quantity UOM": null,
"Ship Date": "09/14/2025",
"Signature Count": 1
}The explicitly labeled pickup and delivery dates both mapped to 09/14/2025. The reference counted the driver’s signature but did not treat it as confirmation that the consignee had received the shipment. Quantity Unit of Measure (UOM) remained null because the document supplied a quantity without a unit.
A different record contained two quantity levels in its freight table. OCR flattened the original columns into a sequence of lines. Reconstructed from the surrounding H/U, PACKAGE, and QTY/TYPE headers, the relevant row was:
Our reference response selected the handling-unit level:
{
"Quantity": 26,
"Quantity UOM": "PALLETS"
}Sonnet and Haiku originally returned 500 / Pieces, and they returned 26 / Pallets when invoked with their optimized candidate prompts. Nova still returned 500 / Pieces even with its optimized candidate prompt. The same example therefore produced different model-specific behavior even though the starting prompt, record, reference, and evaluator were the same across models.
The JSON Lines (.jsonl) file used to pass the inputs should contain one object per template and evaluation configuration. Current quotas allow 10 templates per job, 100 samples per template, and five target models.
The Evaluation Dataset Defines What the Optimizer Can Learn
The optimization set is an executable description of the workload. It needs routine inputs, difficult examples, and high-risk cases, without omitting behavior the candidate prompt must preserve throughout its optimization.
Pay close attention to the reference responses you provide, just as you would when writing tests for deterministic software. In our case, one OCR input appeared twice in our 50 records with conflicting references for Commodity, Destination Address, Shipment Number, and Signature Count. As a result, the jobs contained 49 unique OCR inputs, and their scores include that inconsistency. Before running a production job, deduplicate samples, validate reference responses, and test the evaluator against known outputs to ensure the results are reliable.
Because Amazon Bedrock evaluates and rewrites based on the supplied examples, the scores you get and the optimization decisions made from them depend heavily on the optimization set. In addition to this, you should reserve a holdout dataset of unseen inputs that you can use to verify that the optimizer isn't generating local optimizations that score better on the optimization set but worse on general behavior (a simple example of this would be if the optimized prompt contained hard rules specific to your examples). AWS recommends this comparison in its dataset guidance.
Our public experiment did not include a separate holdout because we were limited in the number of documents we could use for anything publicly shareable. Because of this, we can't establish that the improvements observed in the optimization dataset will extend to unseen bills of lading. We did have a holdout set when we performed manual prompt optimization for the customer this experiment is based on, and our improvements there did translate. The holdout set is what lets you verify performance before promoting the optimized version to production, and monitoring production traffic provides further confirmation.
Every Candidate Scored Higher Under Caylent's Evaluator
Keep in mind that the score is relative to the samples and evaluator, so it is not a universal accuracy measure. In this article, we use it for within-model comparisons between each model's original and optimized prompt. Advanced Prompt Optimization can also compare target models when they use the same samples, evaluator, and controlled model settings. Amazon’s result guide documents the output and normalization.
Methodology: 50 evaluation records, one workload-specific Lambda evaluator, and within-model comparisons using a shared starting prompt. These are Caylent custom evaluation scores, not generic accuracy rates.
Every candidate outscored its baseline on the optimization set. The custom score is the mean of the 50 record scores, while the token figures include the prompt and the OCR input for each record. Our metric did not penalize prompt length, and every candidate became larger and consequently more expensive to run (which is not a factor we optimized for in this experiment). Their exact cost and context-window impact depend on the model, document length, caching, and request volume, so actual values will only be known if the candidates are deployed to production and measured.
Amazon Bedrock Advanced Prompt Optimization Pricing
Advanced Prompt Optimization has no separate service fee. Target-model, optimizer, and optional LLM-judge inference use Bedrock Standard on-demand rates; Lambda evaluation is billed separately. For a Lambda template, AWS estimates target usage as 16 * N * (P + O) tokens and optimizer input as 101 * (3,700 + 0.35P) tokens, where N is the number of records, P is the input token count in the prompt template, and O is the expected target-model output length. AWS currently uses Claude Sonnet 4.6 as the optimizer.
The following is a hypothetical pricing calculation using the stated Standard rates, not the measured cost of Caylent's experiment. For N=50, P=1,000 prompt-template input tokens, and O=500 expected output tokens, those formulas estimate 800,000 target input tokens, 400,000 target output tokens, and 409,050 optimizer input tokens. Sonnet 5's input/output rates of $2/$10 per million tokens and Sonnet 4.6 optimizer rates of $3/$15, with an illustrative 100,000 optimizer output tokens, produce an estimated cost of $8.33 plus Lambda.
Optimizer output length is unknown until you run the optimizer, and LLM-as-a-judge or steering criteria add a separate judge-inference component. Actual cost of using the optimized prompt will only be known once it's deployed and measured in production, since while optimization datasets are representative of real inputs, production traffic may have cache hits, retries, guardrails, or other behaviors that are present but we don't optimize prompts for.
Composite Metrics And Field-Level Regressions
Amazon Bedrock Advanced Prompt Optimization receives one scalar score per record from your evaluators. Weighted sums allow gains in one dimension to offset losses in another, opening the possibility to regressions in some fields so long as the aggregate score is overall higher. To prevent this, behaviors that must pass independently need a hard-fail gate, a large penalty, or a separate approval criterion. This should be controlled in the Lambda evaluator, where separate metrics and gates can be defined for each score before aggregation.
One record from our experiment shows how the score combined an improvement in one field with a regression in another. The following redacted reconstruction from the flattened OCR includes the document date, its ship-from and ship-to blocks, two labeled delivery dates, and the relevant signature area:
07/31/2024
BILL OF LADING
Bill of Lading Number: [REDACTED]
SHIP FROM
[ORIGIN DETAILS REDACTED]
Early Delivery Date: 08/01/2024 00:00
Late Delivery Date: 08/01/2024 00:00
SHIP TO
[DESTINATION DETAILS REDACTED]
SIGNATURE AREA
SHIPPER SIGNATURE / DATE
CARRIER SIGNATURE / PICKUP DATE
[TWO FILLED ENTRIES; NAMES AND INTERLEAVED OCR MARKS REDACTED]
Property described above is received in good order, except as noted.The supplied reference response expected the following values:
{
"Delivery Date": "08/01/2024",
"POD": "1",
"Signature Count": 2
}The supplied reference labeled this combination of the filled signature area and receipt notation as POD="1".
For Sonnet, the relevant fields and total score changed as follows:
For the other 27 fields, the evaluator awarded the same credit before and after optimization. The optimized response lost the Delivery Date field, which we scored Moderate and valued at 2 weighted points, but gained the POD field, which we scored Critical and valued at 4. It also stopped wrapping the JSON in code fences, removing the 5% response-level penalty. The record score therefore rose even though the optimized response missed the Delivery Date field. This record illustrates the trade-off that can happen when each field is aggregated into a single score, and how some parts of the response might regress even as the overall score goes up and the optimized prompt is deemed better.
The regression on Delivery Date seems to come from how dates are interpreted. The differences in the prompts showed how all three candidates changed the date evidence they would accept. The original description defined the field as an expected arrival date:
- The date when the shipment is expected to arrive at the destination. May also be referred to as Expected Delivery Date or ETA.
Nova redefined it around confirmed physical delivery:
- The date when the shipment was physically delivered and acknowledged by the recipient — confirmed by a signed/stamped delivery receipt, receiver signature with date, or an explicit "Date Delivered" / "Delivery Date" notation.
It then added:
- Do NOT use "Requested Date", "Required Date", "Must Arrive By", "Promise Date", appointment dates, or scheduled dates — these are not Delivery Dates unless accompanied by an actual signature confirming receipt.
Sonnet preserved the original field description but added a separate rule later in the prompt:
- DELIVERY DATE: Extract this value ONLY if an explicit delivery date label is printed on the document [...] or a signed/stamped date in a receiver confirmation section.
Haiku changed the description itself:
- The date when the shipment actually arrived or is confirmed to have been delivered.
It also explicitly instructed the model not to use requested or estimated dates.
Because each candidate changed many instructions, the differences in the prompts don't prove that any one line caused the regression. It does establish, however, that Nova and Haiku changed the field's business meaning, while Sonnet kept the description intact but added narrower guidance elsewhere, which affected the meaning. Because Delivery Date meant expected arrival in our starting contract, Nova and Haiku's redefinitions are wrong, and Sonnet's narrower evidence rule still needs validation despite the higher aggregate score. There isn't an automatic solution; this is a potential failure mode that can be introduced during prompt optimization, and you should watch for it.
Further Differences in Model-Specific Candidate Prompts
Our optimization produced prompts that used different strategies to perform the extraction. Sonnet introduced a two-pass workflow:
- PASS 1 - DOCUMENT INVENTORY: Read the entire document carefully and create a mental inventory of every distinct labeled value you find...
- PASS 2 - FIELD MAPPING: Using your inventory from Pass 1, map each labeled value to exactly one output field...
Nova emphasized field precedence, role disambiguation, null handling, and signature checks. Haiku added extensive examples, repeated null rules, output reminders, and a final self-check.
Some additions fit the optimization records more closely than we would accept in an actual production prompt. One OCR record contained these two date fields:
REQUESTED DATE
09-AUG-2024 (MUST ARRIVE ON OR BEFORE THIS DATE)
SCHEDULED SHIP DATE
31-DEC-2030Haiku then generated this rule:
- If a candidate date is far in the future (e.g., year 2029 or later) or appears to be a system placeholder (e.g., 12/31/2030, 31-DEC-2030), do NOT use it as the Ship Date...
The candidate copied 31-DEC-2030 directly from the evaluation record and turned that example into a fixed cutoff. A fixed 2029 cutoff may handle this corpus well, and even work well in production right now, but it would introduce a new failure mode as that arbitrary date nears. Haiku also reused exact quantity and packaging examples, and Nova incorporated names, addresses, operational identifiers, and other corpus fragments. Using sample-derived content in prompts tends to lead to overfitting to the dataset used for optimization and introduces privacy risks if the dataset contains real data (which was not our case for this experiment). Testing candidate prompts against a holdout dataset allows you to prove whether these optimizations generalize to other documents or are simply overfit.
You should thoroughly review each candidate for altered definitions, contradictions, source-derived literals, weakened output requirements, and instructions that will age poorly. Sanitize record-derived content before reuse or publication.
Output formatting produced another model-specific result. The starting prompt already said:
- Just return the JSON object, do not include any other text or comments.
Sonnet expanded that into:
- OUTPUT FORMAT REQUIREMENT: Return ONLY a raw JSON object. Do NOT wrap your response in markdown code fences...
Nova and Haiku generated similarly emphatic versions. All 50 baseline responses per model required the evaluator to strip fences or surrounding text. After optimization, 38 Nova and 21 Sonnet responses were bare JSON, while all 50 Haiku responses still required cleanup. No baseline or optimized response was unparseable, but it's notable that similar instructions produced different behavior across models.
Conclusion: From an Optimized Prompt to a Production Decision
We estimated that manually optimizing the base prompt for our three target models would have taken us two to three days of prompt engineering. Preparing the examples, configuring the job, and completing this cycle using Amazon Bedrock Advanced Prompt Optimization took four hours or less. Advanced Prompt Optimization made the mechanical work of creating prompt variations and testing them with the evaluator easier, but our team still owned the data, evaluator, editable scope, review, and production decision.
A production workflow should carry that ownership through each stage:
- Define and test the objective. Specify the aggregate score and no-regression gates, then verify known good, partial, and bad outputs.
- Build separate datasets. Prepare representative optimization and held-out examples, then deduplicate and validate their references.
- Protect stable text and run the job. Set the editable scope, models, and inference settings explicitly.
- Review each candidate. Check business meaning, sensitive content, brittle rules, and prompt growth.
- Validate on unseen data. Freeze the prompts and compare them on held-out data with the same model settings.
- Make the adoption decision. Apply aggregate and field-level gates, then measure schema compliance, hallucinations, tokens, latency, and cost in the application.
Advanced Prompt Optimization makes prompt refinement into a structured, evaluation-backed process. For our experiment, it produced three higher-scoring, model-specific candidates in a shorter assisted cycle than it would have taken us to do that work manually. However, every prompt became longer, the Delivery Date field regressed across all three runs, and our generated rules needed domain and privacy review. The decision to promote these candidate prompts depends on business correctness, held-out field-level validation, output compliance, and acceptable runtime cost. That work is essential for organizations.
How Caylent Can Help
Amazon Bedrock Advanced Prompt Optimization can make prompt refinement faster and more systematic, but getting from a higher-scoring candidate to a production-ready prompt still requires the right evaluation strategy, domain expertise, and validation process. Caylent helps organizations design evaluation frameworks, optimize prompts for specific models and workloads, validate candidates against held-out data, and establish the guardrails needed to move optimized prompts into production with confidence. With deep expertise across AWS, generative AI, and agentic AI, Caylent helps teams turn experimentation into reliable, measurable AI solutions. Reach out to us today to get started.
Generative AI & LLMOps
Guille Ojeda
Guille Ojeda is a Principal Innovation Architect at Caylent, a speaker, author, and content creator. He has published 2 books, over 200 blog articles, and writes a free newsletter called Simple AWS with more than 45,000 subscribers. He's spoken at multiple AWS Summits and other events, and was recognized as AWS Builder of the Year in 2025.
View Guille's articlesAccelerate your GenAI initiatives
Leveraging our accelerators and technical experience
Browse GenAI Offerings
Related Blog Posts
Claude Fable 5.1: What Changed From Fable 5 and What to Validate
Explore what’s new in Claude Fable 5.1, how it compares to Fable 5, and what organizations should validate before migrating their workloads.
Generative AI & LLMOps
Agent Discovery at Runtime With AWS Agent Registry
Explore how AWS Agent Registry can reduce custom work and endpoint-management work while enabling governed publication and runtime discovery across growing agent ecosystems.
Generative AI & LLMOps
How We Enabled Our Workforce to Be Anthropic Certified
Explore the lessons we learned from preparing our workforce to get certified on Claude through Anthropic's certification program and the study habits that made the biggest difference.
Generative AI & LLMOps