In this part, we will mainly focus on the considerations when deploying a Large Language Model to production, various libraries that support open-source LLM optimizations and tools/resources linking to model comparisons.
Considerations for Production Deployment
Scalability
Ensure that the infrastructure supporting LLM deployment is scalable to handle varying workloads and traffic demands. This includes considerations for horizontal and vertical scaling, load balancing, and resource allocation. This also depends on choosing the right Inference option.
Prompting vs. finetuning vs. RAG
If we get good quality results from a pre-trained model with prompting then we can stick to prompting.
If we need LLM to work with internal or fresh data then we can choose the RAG framework.
If we have enough data and other methods do not provide good results, we can fine-tune the model.
Along with this, we should also consider the cost, time, accuracy, data recency, ease of implementation and interpretability
Performance Optimization & Resource Efficiency
Optimize LLM inference performance to meet latency and throughput requirements. Minimize resource consumption, including compute, memory, and storage, to optimize cost-effectiveness. This may involve Hardware and software optimization techniques such as quantization, distillation etc mentioned in part 1 of this blog.
Deployment Flexibility
Design deployment architectures that offer flexibility in deployment options, including on-premises, cloud, and hybrid deployments. This allows for adaptation to changing business requirements and infrastructure preferences.
Security and Compliance:
Address security concerns such as data privacy, and compliance with regulatory requirements. Implement encryption, access controls, and auditing mechanisms to protect sensitive data and ensure compliance with industry standards. Implementing in-house deployment and using open-source llm may also help.
Evaluation
Evaluating LLMs can be challenging. There are many evaluation metrics for specific NLP tasks such as the Bleu score for translation, the Rouge score for summarization. However, these are based on precision and recall and there might be cases where the Rouge and bleu score might still not be able to capture the intricacies in the text.
Integration with Libraries and Cloud Platforms
vLLM:
vLLM is a fast and easy-to-use library for LLM inference and serving.
Supports paged attention, continuous batching, quantization and other software optimization techniques.
Works with Hugging Face models.
MLC LLM :
Machine Learning Compilation for Large Language Models (MLC LLM) is a high-performance universal deployment solution that allows native deployment of any large language models with native APIs with compiler acceleration
support many open-source models
Text Generation Inference
Text Generation Inference (TGI) is a toolkit for deploying and serving Large Language Models (LLMs). TGI enables high-performance text generation for the most popular open-source LLMs
Supports paged attention, flash attention, continuous batching, quantization and other software optimization techniques.
works out of the box to serve optimized models for all modern models. They can be found in this list.
LightLLM:
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance
Supports token attention, flash attention, dynamic batching and other software optimization techniques.
Comparison Tools for Evaluation
There are many tools to compare the latency, throughput, memory and performance of different models on various hardware. Below are few of them:
Open LLM Leaderboard
The 🤗 Open LLM Leaderboard aims to track, rank and evaluate open LLMs and chatbots.
LLM-Perf Leaderboard
The 🤗 LLM-Perf Leaderboard 🏋️ aims to benchmark the performance (latency, throughput, memory & energy) of Large Language Models (LLMs) with different hardwares, backends and optimizations
Martian's provider leaderboard
Martian's provider leaderboard collects metrics daily and tracks them over time to evaluate the performance of LLM inference providers on common LLMs Aviary Leaderboard
Aviary Leaderboard
Aviary Explorer allows you to take a single prompt and send it to a number of open-source LLMs hosted by Anyscale. It also gives you estimates of the cost as well as the latency.
Conclusion
In this blog, we explored various LLM performance optimization methods, scalability, and resource efficiency techniques. We also covered different libraries that integrate various open-source LLMs and different software and hardware optimization techniques to speed up inference. We also covered comparison tools for different models.