Evaluating LLM Performance: A Benchmarking Framework on Amazon Bedrock

New LLMs are being released on a daily/weekly basis, and practitioners need to evaluate their performance to identify the most suitable ones for specific tasks. For that, different benchmark/evaluation tools and datasets are developed depending on the task the LLM is designed to tackle, whether it is coding, chatbot, summarization, instruction following ability, etc. Some of the popular benchmark datasets and metrics are the HumanEval dataset (code generation benchmark), the MT Bench (instruction-following ability benchmark), the MBPP (python code generation benchmark), and the Chatbot Arena (chatbot assistant benchmark).

Assessing LLM performance through automated benchmarking poses a considerable challenge due to the open-ended nature of the tasks, making it highly challenging to develop a program for automated evaluation of response quality. Hence, Human-in-the-loop (HITL), where human evaluators provide feedback on the quality of the text generated by the LLMs for different tasks, proves to be a viable solution. Amazon Comprehend's LLM Trust and Safety features offer an automated approach that can complement human review. AWS also launched the “Model Evaluation on Amazon Bedrock'' feature in re:Invent 2023 which is still in preview mode to create and view model evaluation jobs. It provides three options to evaluate LLMs: 1) to automatically evaluate a single model using recommended metrics for built-in task types, text summarization, question and answer, text classification, and open-ended text generation, 2) to evaluate up to 2 models using a work team of your choice to provide feedback, or 3) to customize the number of models to evaluate using a work-team designated by AWS. If you choose the human evaluation options you can also define a custom task on top of the built-in tasks.

In this post, we will walk you through an example of a HITL framework to evaluate the performance of LLMs available on Bedrock on your own generated datasets and specific tasks. You may find the code in this github repository.

Overall Solution Architecture:

From a high-level perspective, our solution consists of the following sections:

• Model Repository
• Prompt Repository
• Prompt/Response Workflow Orchestration
• UI to collect human evaluator feedback

We will be using Bedrock as the model repository for LLM and Amazon DynamoDB as a prompt catalog to store our datasets/prompts to be used later in the process. To orchestrate calling Bedrock APIs, parse the response and record the desired output, we use AWS Step Functions. Finally, we will use Streamlit to build a quick UI to collect user feedback. Please refer to the following diagram demonstrating how the components are interconnected to build our LLM benchmarking solution.