A Comprehensive Guide to LLM Evaluations

Deploying Large Language Models (LLM) into production without a robust evaluation framework is like deploying software without tests: simply a bad idea. Yet many organizations struggle with LLM evaluations because it's a bit different from traditional software testing paradigms.

Traditional software operates deterministically: given the same input, it produces the same output every time. In contrast, LLMs are non-deterministic. They generate outputs from a probability distribution across an effectively infinite space of possible responses. When we're expecting deterministic outputs, an LLM's probabilistic behavior might be a risk, but it'll still be tested in the same way as deterministic code. Testing becomes challenging when we're using the LLM's non-determinism as a feature, fully intending on generating different outputs from the same inputs. In traditional testing, we're used to a finite set of expected outputs, which we test for with a simple equality comparison: actual_output == expected_output. Now the set of acceptable outputs is effectively infinite. How do we test for that?

How Can I Test An LLM?

The difference isn't technical, and thus can't be solved with a few lines of code. We need to take a step back, and redefine how we decide whether an output is good or not. It's no longer sufficient for an output to be identical to a golden answer, quality now encompasses factual accuracy, contextual relevance, coherence across long interactions, safety from harmful content, and alignment with ethical norms.

Evaluation approaches can be organized along several key dimensions:

• Reference-based evaluation compares LLM output against predefined ground truth answers, making it suitable for tasks with relatively deterministic correct answers. The comparison doesn't have to be an equality test, you can compare the technical depth of a blog article for example. They key point is that you're comparing with a golden answer or golden set of answers, called ground truth.
• Reference-free evaluation assesses intrinsic text qualities without comparison to specific references, essential for open-ended tasks where quality is judged on criteria like coherence, relevance, or adherence to guidelines.
• Automated evaluation relies on algorithmic metrics such as presence of certain words, often used when assessing the safety of an output. These are much faster and cheaper to calculate than more nuanced metrics that rely on semantics.
• Human evaluation involves human judges assessing outputs based on guidelines. They're slow and expensive, but are still the gold standard for capturing nuanced, subjective qualities that automated systems miss.
• Model-based evaluation bridges these approaches by using LLMs to assess quality, the context-aware understanding of human evaluators with the repeatability of automated evaluations.

Programmatic Evaluation Metrics

Programmatic evaluation methods are scalable, repeatable, and cost-effective, which means even if they don't cover everything you might want to evaluate, most of the time, it's a good idea to use them as either a starting point or a complement.

Foundational Lexical Metrics

The earliest programmatic metrics operate on lexical overlap principles, quantifying similarity between model-generated text and human-written references by counting shared word sequences.

BLEU (Bilingual Evaluation Understudy), originally developed for machine translation, measures candidate text quality by calculating n-gram precision relative to reference translations. BLEU's fundamental limitation is its inability to handle lexical variation. It penalizes synonym use or alternative phrasing even when meaning is preserved, leading to poor correlation with human judgment for semantically diverse tasks.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) takes a recall-oriented approach designed for summarization evaluation. It measures how many reference summary n-grams appear in the model-generated summary. ROUGE shares BLEU's core weakness: reliance on surface-level word overlap means it cannot capture semantic meaning and may reward summaries that are lexically similar but factually incorrect or incoherent.

METEOR (Metric for Evaluation of Translation with Explicit Ordering) performs word alignment between candidate and reference texts, giving credit not only for exact matches, but also for synonyms, stems, and paraphrases using resources like WordNet. METEOR calculates both precision and recall based on unigram matches, combines them via harmonic mean, and introduces a fragmentation penalty to reward fluency. While it improves correlation with human assessment, it still struggles with deep semantic and contextual nuances.

Embedding-Based Metrics

The primary problem of lexical metrics is their inability to understand that different words can convey identical meaning. This drove development of semantic evaluation methods leveraging pre-trained language models.

BERTScore pioneered this direction by using models like BERT to generate contextual embeddings for each token in candidate and reference texts. It computes cosine similarity between each candidate token and the most similar reference token, then aggregates these scores to calculate embedding-based precision (how well candidate tokens are supported by reference), recall (how well reference tokens are captured by candidate), and F1 score (the harmonic mean of precision and recall). This approach recognizes paraphrases and synonyms as high-quality matches, significantly improving correlation with human quality judgments.

LLM-as-a-Judge

As large language models have become more sophisticated, their applications have shifted toward open-ended, creative, and conversational tasks where single ground truth references often don't exist. This shift rendered traditional reference-based metrics inadequate, and while human evaluation is still an option, it isn't scalable. The field's response: using highly capable LLMs as automated, scalable proxies for human evaluators. This approach is called "LLM-as-a-Judge".

Core Mechanisms and Rationale

The core idea is leveraging state-of-the-art foundation models' contextual understanding capabilities to automate nuanced qualitative assessment. The main goal is to evaluate outputs where quality is subjective and multi-faceted, allowing us to automate the assessment of attributes like helpfulness, coherence, creativity, or persona adherence.

There are three primary ways to perform these assessments:

Pairwise Comparison presents the judge LLM with a prompt and two corresponding outputs (typically from different models, model versions, or different versions of your app). It determines which response is better according to given criteria, or declares a tie. This is considered more reliable than absolute scoring because it forces relative choice.

Direct Scoring evaluates a single output against a detailed rubric, instructing the model to assign scores on a Likert scale (commonly 1 to 5) for specific quality dimensions. For example, it might rate an answer's factuality, clarity, and conciseness separately. This provides more granular feedback than pairwise comparison but can be more susceptible to inconsistency.

Reference-free Evaluation assesses intrinsic text properties without needing golden answers. You can configure the judge to evaluate whether a chatbot response is polite, if an answer is factually consistent with provided source documents, or if generated code adheres to specific style guidelines.

Engineering Effective Judge Prompts

The quality, fidelity and repeatability of your LLM-as-a-Judge evaluations depends significantly on the quality and precision of the evaluation prompt. Crafting effective judge prompts is a form of meta-prompt engineering aimed at eliciting consistent, unbiased, accurate judgments.

Our experiences at Caylent creating evaluations for multiple customers have led us to the following best practices:

Clear Criteria and Rubrics: Your prompt must provide explicit, unambiguous, well-defined rubrics. Vague instructions like "Is this a good answer?" produce unreliable results. Instead, you should break criteria down into specific, verifiable components. Don't just ask the LLM to "rate helpfulness," also define what helpfulness is, and give examples.

Chain-of-Thought (CoT) Prompting: Instructing the judge to "think step-by-step" before delivering verdicts improves reliability and transparency. This technique forces models to articulate reasoning, breaking complex judgments into simpler evaluation sequences. Additionally, you should instruct the Judge to output its rationale for the score.

Structured Output Formats: Instructing the judge to return assessments in structured formats like JSON or XML make it easy to parse scores, rationales and feedback, allowing programmatic analysis and aggregation across large datasets.

Example-based Calibration: The judge's alignment with human standards significantly improves if you provide few-shot examples of high-quality and low-quality evaluations within the prompt itself. Seeing concrete examples helps the model calibrate its internal standards and produce judgments more consistent with human expert assessments.

Here's an example of a Judge prompt: