Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 21 hours
Course Outline
Introduction to Scaling Ollama
- Ollama’s architecture and key scaling considerations
- Common bottlenecks in multi-user deployments
- Best practices for preparing infrastructure
Resource Allocation and GPU Optimisation
- Strategies for efficient CPU/GPU utilisation
- Considerations for memory and bandwidth
- Container-level resource constraints
Deployment with Containers and Kubernetes
- Containerising Ollama using Docker
- Running Ollama within Kubernetes clusters
- Load balancing and service discovery
Autoscaling and Batching
- Designing autoscaling policies for Ollama
- Batch inference techniques to optimise throughput
- Trade-offs between latency and throughput
Latency Optimisation
- Profiling inference performance
- Caching strategies and model warm-up
- Reducing I/O and communication overhead
Monitoring and Observability
- Integrating Prometheus for metrics
- Building dashboards with Grafana
- Alerting and incident response for Ollama infrastructure
Cost Management and Scaling Strategies
- Cost-aware GPU allocation
- Considerations for cloud vs. on-prem deployment
- Strategies for sustainable scaling
Summary and Next Steps
Requirements
- Practical experience in Linux system administration
- Strong understanding of containerisation and orchestration
- Familiarity with deploying machine learning models
Target Audience
- DevOps Engineers
- ML Infrastructure Teams
- Site Reliability Engineers
