Amazon SageMaker AI enables developers to deploy power machine-learning models. Learn the various options and which endpoint deployment type is best suited for your business.
Learn how AWS technologies help healthcare providers, payers, and healthtech organizations enhance patient care, streamline operations, and secure data with advanced cloud solutions.
Learn how Amazon SageMaker AI accelerates computer vision projects by providing powerful tools for rapid experimentation and seamless scaling from development to production.
Chatbots often fall short, with 48% of users reporting they fail to solve issues. A chatbot's effectiveness depends on the data it can access, making data pre-processing essential, and success starts with understanding your use cases to ensure the right data is available.
Ali Arabi is a Senior Machine Learning Architect at Caylent with extensive experience in solving business problems by building and operationalizing end-to-end cloud-based Machine Learning and Deep Learning solutions and pipelines using Amazon SageMaker AI. He holds an MBA and MSc Data Science & Analytics degree and is AWS Certified Machine Learning professional.
View Ali's articlesAmazon SageMaker AI Inference Endpoints are a powerful tool to deploy your machine learning models in the cloud and make predictions on new data at scale. However, it can be challenging to understand which deployment options to pick. This article aims to provide an overview of the various options and help you decide which endpoint deployment type is best for your use case.
Depending on the business use case, inference workload, required latency, and cost factors, to name a few, one can choose from one of the options below:
Serverless inference is a fully managed inference endpoint, suitable for workloads with intermittent or infrequent traffic patterns, with built-in high-availability and fault tolerance capabilities. There is no need to select instance types, provision capacity, or set scaling policies as the service automatically provisions and scales up or down the compute capacity based on the volume of inference requests.
Two things to remember: the maximum request payload for serverless inference endpoints is 4 MB with a 60-second timeout. The main configuration for serverless inference includes memory size selection, up to 6GB. A suitable workload for these endpoints is form processing for a bank’s mortgage department or chatbots.
Amazon SageMaker AI fully manages these endpoint types, making them suitable for workloads with high throughput and low latency requirements. The main configuration for real-time inference includes setting the autoscaling policy, compute resource selection, and the deployment mode. The deployment modes include single model, multi-model, and inference pipeline.
The maximum request payload is 6 MB with a 60-second timeout. However, real-time inference endpoints can leverage many different instance types with up to eight NVIDIA A100 Tensor Core GPUs, 100Gbps networking, 96 vCPUs, 1.1TB instance memory, and 8TB of NVMe storage. When configuring the endpoint, customers can use production variants to deploy different model versions with A/B testing or shadow deployment strategies. An example of a typical workload is personalized recommendations for users on an e-commerce website.
If your request payload is large (up to 1GB), involves long-running processes (up to 15 mins), and latency is not a concern, then asynchronous inference is the best option for you. In this scenario, an internal queue system handles inference requests asynchronously.
Unlike serverless and real-time inference endpoints, the system places the request and the response for asynchronous inference in an S3 bucket. Typical workloads suitable for asynchronous endpoints are computer vision or NLP problems where the payloads can be large, such as videos or documents.
In some cases, the persistent compute resource is not necessary, and the application needs to make inferences against a large dataset that can be scheduled as an Ad-hoc job. For this scenario, you can use long-running Batch Transform jobs to handle large payloads using a batch strategy (mini-batches of up to 100 MB each). Similar to the asynchronous inference endpoints, the system places the request and the response in an S3 bucket.
Batch Transform inference endpoints can also be a viable option to test different models by implementing multiple transform jobs per model. An example of a typical workload is propensity modeling for user conversion to inform the correct treatment and offer.
The table below summarizes the four options and can be used to inform the best model hosting option on Amazon SageMaker AI.
Beyond understanding the available Amazon SageMaker AI hosting options, there are also some strategies and capabilities that we can leverage to pick the right balance between latency and cost. We’ll review a few of them below.
Overall, ML application latency consists of overhead latency and model inference latency. AWS PrivateLink deployments make it possible to reduce overhead latency and improve security by keeping all the inference traffic within your VPC and by using the endpoint deployed in the AZ closest to the origin inference traffic to process the invocations. By default, Amazon Sagemaker AI endpoints deploy in 2 different AZs.
If you need to host a deep learning model for inferencing and are on a budget, you can leverage Amazon SageMaker AI Elastic Inference (EI) which costs less than a full GPU instance but with comparable high throughput and low latency. You must use an EI-enabled version of TensorFlow, PyTorch, MXNet, or other frameworks using ONNX. One important consideration, Elastic Inference doesn’t support all operations across all ML frameworks. Make sure you test your model with EI.
If the inference workflow is composed of several complicated steps, including preprocessing, transformation, model selection, and post-processing, then Triton can maximize throughput while also providing ultra-low inference latency. With Triton Inference Server integrated with Amazon SageMaker AI supporting ML frameworks such as TensorFlow, PyTorch, XGBoost, NVIDIA TensorRT, and other frameworks using ONNX, users can choose among NVIDIA GPUs, CPUs, and AWS Inferentia for the compute resource.
Another cost-effective hosting option is a multi-model endpoint. Multi-model inference endpoints allow customers to improve endpoint utilization by hosting multiple models on the same serving container and sharing the memory across models. Models can be added or removed from the real-time multi-model inference endpoint. This is a good option for hosting similar models in size and latency behind a single endpoint. This option works with Auto Scaling and AWS PrivateLink.
It is always a good idea to test different configurations for the inference endpoint for the best performance at the lowest cost for your particular use case. Users can now leverage the Amazon SageMaker AI Inference recommender capability to load test the real-time endpoint with different configurations based on the specified ML Framework, ML Domain, and ML Task right in a SageMaker AI notebook, which will eliminate weeks of manual testing and tuning time. Users can then choose the best configuration options based on the reported metrics for latency, price, performance, and throughput.
As an AWS Generative AI competency partner, we have significant experience in helping customers lead the AI revolution, leveraging AI to enhance operational efficiency, improve internal and external communications, and offer highly personalized experiences for their customers.
In this blog post, we’ve reviewed the different possible ML inference endpoints on Amazon SageMaker and their typical use cases. If your team needs this expertise to help you deploy Machine Learning models on Amazon SageMaker or AWS at scale, consider engaging with Caylent through our MLOps pods.
Fig. 1: ML Model Hosting options on AWS
Fig. 2: Serverless Inference (Image created by the author)
Fig. 3: Real-time Inference (Image created by the author)
Fig. 4: Asynchronous Inference (Image created by the author)
Fig. 5: Batch Transform inference (Image created by the author)
Ali Arabi
