When using data or model parallelism features in SageMaker AI it is recommended to use instances with high-throughput, low-latency.
Evaluation
If you use LLMs for traditional ML problems such as sentiment analysis or classification in general, you can still use traditional ML performance metrics such as accuracy, precision, and recall. However, to evaluate the LLM's performance, depending on the task you use it for, we may use different metrics. Some of the well known ones are:
- BLEU for machine translation, which compares the precision of n-grams between reference input and generated output
- METEOR for machine translation which is based on the harmonic mean of unigram precision and recall, with recall weighted higher than precision.
- ROUGE for summarization, which considers precision, recall, and F1-Score over a sequence.
- Perplexity for text generation, which measures how well the trained model has learned the distribution in text.
- BERTScore for text generation (embedding similarity)
- CIDEr measures the similarity between a generated caption for an image and the reference captions.
- SPICE evaluates the caption generated for an image by placing importance on capturing details about objects, attributes, relationships (semantic propositions)
- Using larger models with a bigger number of parameters to evaluate responses generated by smaller language models
- Universal benchmarks for a range of tasks: machine translation, summarization, natural language generation, question answering, and more. Some of the famous benchmarks are Pile, GLUE, SuperGLUE, MMLU, LAMBADA, and Big Bench Collaboration Benchmarks,
- Code specific benchmarks: HumanEval, and Mostly Basic Python Programming (MBPP)
- Human in the loop (HITL), where human evaluators provide feedback on the quality of the generated text by the model
Hosting Options
Traditional hosting solutions used for smaller models do not provide the required optimization functionality to host LLM models with optimal inference latency and/or throughput. With SageMaker AI you can use large model inference container images with Deep Java Library (DJL) Serving. SageMaker AI supports model parallelism and inference optimization libraries, including:
- DeepSpeed is an open-source inference optimization library
- Hugging Face Accelerate is an open-source model parallel inference library
- FasterTransformer is an Nvidia open source library for efficiently running transformer-based neural network inference
Additionally, you can host LLMs on SageMaker AI using Hugging Face LLM Deep Learning Container (DLC), enabling high-performance text generation through tensor parallelism, dynamic batching, and model quantization.
Monitoring
You should closely monitor prompts, generated response, and RAG performance in production along with data quality, model quality, infrastructure utilization, endpoint latency and throughput.
It is also important to compare their capabilities to a baseline over time for any drift or eventual degradation of quality in the generated response.
Monitoring metrics
Some of the proposed metrics to monitor prompts/responses are context, relevance to prompt, existence of pattern (repetitiveness), repeatability, readability, token size, jailbreaking, injection, refusal, sentiment, toxicity, and response hallucination. For RAG it is recommended to monitor chunk size, produced embeddings, embedding speed, and semantic search results. All of these aspects need to be monitored over time to facilitate timely remediation through retraining.
Testing strategies
It is also possible to implement a Shadow Testing or AB Testing pipeline to compare different prompts, embeddings, and/or LLMs over time to choose the best one for different use cases. SageMaker AI provides these features out of the box and you can also combine it with other open source tools such as MLflow to get the best of all of them.
SageMaker A2I
In many cases where automated monitoring is not possible or relevant benchmark is not available for your specific domain you can use the human-in-the-loop capabilities of SageMaker AI through SageMaker Augmented AI (A2I) to implement human review of LLM predictions in production.
Ali Arabi
