Generative AI (GenAI) creates new opportunities for automated benchmarking by adding output variability and model cost dimensions to traditional performance metrics. In this blog, we share a framework for monitoring alignment and drift across several Large Language Models (LLMs) hosted on Amazon Bedrock.
New LLMs are being released on a daily/weekly basis, and practitioners need to evaluate their performance to identify the most suitable ones for specific tasks. For that, different benchmark/evaluation tools and datasets are developed depending on the task the LLM is designed to tackle, whether it is coding, chatbot, summarization, instruction following ability, etc. Some of the popular benchmark datasets and metrics are the HumanEval dataset (code generation benchmark), the MT Bench (instruction-following ability benchmark), the MBPP (python code generation benchmark), and the Chatbot Arena (chatbot assistant benchmark).
Assessing LLM performance through automated benchmarking poses a considerable challenge due to the open-ended nature of the tasks, making it highly challenging to develop a program for automated evaluation of response quality. Hence, Human-in-the-loop (HITL), where human evaluators provide feedback on the quality of the text generated by the LLMs for different tasks, proves to be a viable solution. Amazon Comprehend's LLM Trust and Safety features offer an automated approach that can complement human review. AWS also launched the “Model Evaluation on Amazon Bedrock'' feature in re:Invent 2023 which is still in preview mode to create and view model evaluation jobs. It provides three options to evaluate LLMs: 1) to automatically evaluate a single model using recommended metrics for built-in task types, text summarization, question and answer, text classification, and open-ended text generation, 2) to evaluate up to 2 models using a work team of your choice to provide feedback, or 3) to customize the number of models to evaluate using a work-team designated by AWS. If you choose the human evaluation options you can also define a custom task on top of the built-in tasks.
In this post, we will walk you through an example of a HITL framework to evaluate the performance of LLMs available on Bedrock on your own generated datasets and specific tasks. You may find the code in this github repository.
From a high-level perspective, our solution consists of the following sections:
We will be using Bedrock as the model repository for LLM and Amazon DynamoDB as a prompt catalog to store our datasets/prompts to be used later in the process. To orchestrate calling Bedrock APIs, parse the response and record the desired output, we use AWS Step Functions. Finally, we will use Streamlit to build a quick UI to collect user feedback. Please refer to the following diagram demonstrating how the components are interconnected to build our LLM benchmarking solution.
For our example, we have included the following LLMs from Bedrock:
AI21:
We will extend this list as new LLMs become available on Bedrock.
We need to store all the prompts for benchmarking LLMs, and DynamoDB is a great option to catalog and version prompts for repeatability. For our example, we have leveraged the following categories/datasets adapted from here. This includes knowledge, code, instruct, creativity, and reflexion. Below is a snapshot of the DynamoDB table that includes our curated prompts.
We use Step Function to orchestrate several AWS Lambda functions in parallel by calling Bedrock APIs for each selected LLM and storing the responses in the DynamoDB table. Below, you can find the state machine language and resulting graph after a successful run.
We have developed a Lambda function template and provided LLM-specific prompt configuration and parameters to build 4 Lambda functions, one for each LLM.
The outputs of these Lambda functions capture the LLM response, token size (if the output token size is part of the model response otherwise number of characters), latency and model parameters. The results are recorded in another DynamoDB table called “bedrock_benchmark”. The Step Function is scheduled to trigger the Lambda functions daily. If the latest response from an LLM in any category differs from its previous response, then a record is added to this table to detect and track drift.
Below you can see the Lambda function code.
We had to time the invocation latency and calculate output token size in the code, but while writing this blog post, Bedrock invocation metrics including InputTokenCount, OutputTokenCount, and InvocationLatency were added to CloudWatch Metrics and as part of the response as well. Thanks to AWS's continuous improvement, one can update the above code to retrieve the metrics and simplify the code.
As you can see in screenshots below, each LLM on Bedrock has a different input configuration, which is provided to the corresponding Lambda function as environment variables. You may tweak parameters influencing the LLM response, including temperature, top P, max token count, and top K. The naming convention across models might be different, which has made the Lambda function lengthy to accommodate all variations. If you implement this workflow for a real application, modify the parameters to fit your requirements.
We used Streamlit to build a simple but effective UI to present the cases where the response for any LLM in any category is different from its previous one and to collect feedback from a human evaluator. The user can choose between 0 to indicate “INCORRECT”, 1 for “CORRECT” response, and 2 for “EXCELLENT” to guide improvements.
In our Streamlit app, it first fetches all of the curated prompts as well as the most recent LLM response and their previous different response. Then it iterates through them by presenting the identified cases to the user, collecting user feedback accordingly and finally storing the feedback in the DynamoDB table. We deployed our Streamlit app in SageMaker Studio using a Terminal instance, and here is how it looks with an example.
If you keep running this workflow and collect user feedback accordingly, you can create informative radar charts to demonstrate each LLM's capabilities in different tasks quickly. You can also monitor the trend to observe the stability of a certain model over time.
Going beyond the examples shared here, Caylent is extending this framework to include response performance and cost so that the price/performance of alternative models can also be compared to evaluate return on investment. We'll also be testing Model Evaluation on Amazon Bedrock during preview to take advantage of the new service capabilities. As LLMs continue to evolve rapidly, we recommend benchmarking regularly to ensure you're using the most appropriate model for your needs.
In this quick guide we presented a simple solution where you can build your curated dataset and the human-in-the-loop workflow using AWS services and Streamlit to benchmark LLMs available on Bedrock to choose the best option for your specific tasks.
Is your company trying to figure out where to go with generative AI? Consider finding a partner who can help you get there. At Caylent, we have a full suite of generative AI offerings. Starting with our Generative AI Strategy Caylent Catalyst, we can start the ideation process and guide you through the art of the possible for your business. Using these new ideas, we can implement our Generative AI Knowledge Base Caylent Catalyst to build a quick, out-of-the-box solution integrated with your company's data to enable powerful search capabilities using natural language queries. Caylent’s AWS Generative AI Proof of Value Caylent Catalyst can help you build an AI roadmap for your company and demonstrate how generative AI will play a part. As part of these Catalysts, our teams will help you understand your custom roadmap for generative AI and how Caylent can help lead the way. For companies ready to take their generative AI initiatives beyond the scope of Caylent's prebuilt offerings, we can tailor an engagement exactly to your requirements.
Ali Arabi is a Senior Machine Learning Architect at Caylent with extensive experience in solving business problems by building and operationalizing end-to-end cloud-based Machine Learning and Deep Learning solutions and pipelines using Amazon SageMaker AI. He holds an MBA and MSc Data Science & Analytics degree and is AWS Certified Machine Learning professional.
View Ali's articles
Randall Hunt, Chief Technology Officer at Caylent, is a technology leader, investor, and hands-on-keyboard coder based in Los Angeles, CA. Previously, Randall led software and developer relations teams at Facebook, SpaceX, AWS, MongoDB, and NASA. Randall spends most of his time listening to customers, building demos, writing blog posts, and mentoring junior engineers. Python and C++ are his favorite programming languages, but he begrudgingly admits that Javascript rules the world. Outside of work, Randall loves to read science fiction, advise startups, travel, and ski.
View Randall's articlesLeveraging our accelerators and technical experience
Browse GenAI Offerings
Discover how Amazon Q Developer is redefining developer productivity - featuring a real-world migration of a .NET Framework application to .NET 8 that transforms weeks of manual effort into just hours with AI-powered automation.
Explore how Amazon Bedrock AgentCore and the Agent Marketplace are industrializing, standardizing, and commoditizing the underlying agent infrastructure, helping organizations eliminate the operational toil and risk that have slowed the adoption of agentic systems.
Explore how agentic AI architectures can address the complexity, uncertainty, and personalization needs of modern healthcare by mirroring medical team dynamics, enabling dynamic reasoning, mitigating bias, and delivering more context-aware and trustworthy medical insights.
Ali Arabi
Randall Hunt
{
"Comment": "Bedrock Models Benchmark",
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"End": true,
"Branches": [
{
"StartAt": "Amazon Models",
"States": {
"Amazon Models": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": "${AmazonFunctionArn}"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException",
"Lambda.TooManyRequestsException"
],
"IntervalSeconds": 1,
"MaxAttempts": 3,
"BackoffRate": 2
}
],
"End": true
}
}
},
{
"StartAt": "Anthropic Models",
"States": {
"Anthropic Models": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": "${AnthropicFunctionArn}"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException",
"Lambda.TooManyRequestsException"
],
"IntervalSeconds": 1,
"MaxAttempts": 3,
"BackoffRate": 2
}
],
"End": true
}
}
},
{
"StartAt": "Cohere Models",
"States": {
"Cohere Models": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": "${CohereFunctionArn}"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException",
"Lambda.TooManyRequestsException"
],
"IntervalSeconds": 1,
"MaxAttempts": 3,
"BackoffRate": 2
}
],
"End": true
}
}
},
{
"StartAt": "AI21 Models",
"States": {
"AI21 Models": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": "${Ai21FunctionArn}"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException",
"Lambda.TooManyRequestsException"
],
"IntervalSeconds": 1,
"MaxAttempts": 3,
"BackoffRate": 2
}
],
"End": true
}
}
}
]
}
}
}
import boto3
from boto3.dynamodb.conditions import Attr
import json
import pandas as pd
import time
import hashlib
from datetime import date
import os
import ast
bedrock = boto3.client("bedrock")
bedrock_runtime = boto3.client("bedrock-runtime")
ddb = boto3.resource('dynamodb')
dynamodb = boto3.client('dynamodb')
response_s = dynamodb.scan(
TableName=os.environ['prompt_catalog'])
table = ddb.Table(os.environ['benchmark_table'])
model_shape = json.loads(os.environ['model_shape'])
model_shape = ast.literal_eval(str(model_shape))
supported_models = os.environ['supported_models']
supported_models = ast.literal_eval(supported_models)
today_date = date.today()
def computeMD5hash(my_string):
m = hashlib.md5()
m.update(my_string.encode('utf-8'))
return m.hexdigest()
def get_prompt_result(model_id,body):
body = json.dumps(body)
print(body)
start_time = time.time()
resp = bedrock_runtime.invoke_model(modelId=model_id, body=body)
end_time = time.time()
elapsed_time = end_time - start_time
return json.loads(resp['body'].read()), elapsed_time
def put_item_anthropic(model, response, today_date, resp, elapsed_time, model_shape):
table.put_item(
Item={
"model_prompt_id" : model + "_" + response['id'].get('S') ,
"date" : str(today_date),
"token_size" : len(resp["completion"].replace(" ","")),
"output" : resp["completion"],
"output_hash" : computeMD5hash(resp["completion"]),
"prompt_model_id" : response['id'].get('S') + "_" + model,
"latency": str(elapsed_time),
"model_config": str(model_shape)
})
def put_item_amazon(model, response, today_date, resp, elapsed_time, model_shape):
table.put_item(
Item={
"model_prompt_id" : model + "_" + response['id'].get('S'),
"date" : str(today_date),
"token_size" : resp["results"][0]["tokenCount"],
"output" : resp["results"][0]["outputText"],
"output_hash" : computeMD5hash(resp["results"][0]["outputText"]),
"prompt_model_id" : response['id'].get('S') + "_" + model,
"latency": str(elapsed_time),
"model_config": str(model_shape)
})
def put_item_ai21(model, response, today_date, resp, elapsed_time, model_shape):
table.put_item(
Item={
"model_prompt_id" : model + "_" + response['id'].get('S') ,
"date" : str(today_date),
"token_size" : len(resp["completions"][0]["data"]["tokens"]),
"output" : resp["completions"][0]["data"]["text"],
"output_hash" : computeMD5hash(resp["completions"][0]["data"]["text"]),
"prompt_model_id" : response['id'].get('S') + "_" + model,
"latency": str(elapsed_time),
"model_config": str(model_shape)
})
def put_item_cohere(model, response, today_date, resp, elapsed_time, model_shape):
table.put_item(
Item={
"model_prompt_id" : model + "_" + response['id'].get('S') ,
"date" : str(today_date),
"token_size" : len(resp["generations"][0]["text"].replace(" ","")),
"output" : resp["generations"][0]["text"],
"output_hash" : computeMD5hash(resp["generations"][0]["text"]),
"prompt_model_id" : response['id'].get('S') + "_" + model,
"latency": str(elapsed_time),
"model_config": str(model_shape)
})
def try_prompts():
for response in response_s["Items"]:
for model in supported_models:
print(f"\tModel: {model}")
body = model_shape.copy()
if model.startswith("anthropic"):
body["prompt"] = f"\n\nHuman:{response['prompt'].get('S')}\n\nAssistant:"
elif model.startswith("amazon.titan"):
body["inputText"] = response['prompt'].get('S')
elif model.startswith("ai21"):
body["prompt"] = response['prompt'].get('S')
elif model.startswith("cohere"):
body["prompt"] = response['prompt'].get('S')
resp, elapsed_time = get_prompt_result(model, body)
data_old = []
scan_kwargs = {
"FilterExpression": Attr('model_prompt_id').eq(f"{model}_{response['id'].get('S')}")
}
done = False
start_key = None
while not done:
if start_key:
scan_kwargs["ExclusiveStartKey"] = start_key
response_old = table.scan(**scan_kwargs)
data_old.extend(response_old.get("Items", []))
start_key = response_old.get("LastEvaluatedKey", None)
done = start_key is None
if model.startswith("anthropic"):
if len(data_old) == 0:
put_item_anthropic(model, response, today_date, resp, elapsed_time, model_shape)
elif computeMD5hash(resp["completion"]) != data_old[-1]['output_hash']:
put_item_anthropic(model, response, today_date, resp, elapsed_time, model_shape)
elif model.startswith("amazon.titan"):
if len(data_old) == 0:
put_item_amazon(model, response, today_date, resp, elapsed_time, model_shape)
elif computeMD5hash(resp["results"][0]["outputText"]) != data_old[-1]['output_hash']:
put_item_amazon(model, response, today_date, resp, elapsed_time, model_shape)
elif model.startswith("ai21"):
if len(data_old) == 0:
put_item_ai21(model, response, today_date, resp, elapsed_time, model_shape)
elif computeMD5hash(resp["completions"][0]["data"]["text"]) != data_old[-1]['output_hash']:
put_item_ai21(model, response, today_date, resp, elapsed_time, model_shape)
elif model.startswith("cohere"):
if len(data_old) == 0:
put_item_cohere(model, response, today_date, resp, elapsed_time, model_shape)
elif computeMD5hash(resp["generations"][0]["text"]) != data_old[-1]['output_hash']:
put_item_cohere(model, response, today_date, resp, elapsed_time, model_shape)
def lambda_handler(event, context):
try_prompts()
return {
'statusCode': 200,
'body': json.dumps('Done!')
}