Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Options
- Trade-offs between Hosted APIs and self-hosting
- On-premises, edge, and air-gapped patterns
- Single model vs. model portfolio strategies
- Cost comparison per decision
Hardware Sizing and Capacity
- Memory footprint based on parameter count
- CPU-only vs. GPU acceleration
- Concurrency and throughput planning
- Allowing headroom for task length growth
Serving Stacks
- vLLM and SGLang options
- llama.cpp and GGUF execution
- Ollama for local development
- Selecting stacks based on decision workload characteristics
Quantization Without Losing Calibration
- Precision formats and memory savings
- Measuring quality and calibration loss
- Per-tensor and importance-aware variants
- Acceptance testing after quantization
Latency Engineering
- Median vs. tail latency
- Latency growth with context length
- Warm starts and model caching
- Batching multiple decision questions
Edge and Constrained Environments
- Running small decision models on CPU
- Apple Silicon and embedded deployment
- Offline operation and update strategies
- Air-gapped weight distribution
Containerization and Scaling
- Image design for model serving
- Volume and cache management
- Horizontal scaling and routing
- Resource limits and GPU sharing
Observability
- Request, latency, and error metrics
- Confidence distribution monitoring
- Tracking decision outcomes and escalations
- Alerting on calibration drift
Versioning and Rollout
- Model and adapter versioning
- Shadow and canary deployment
- Pinned model files for audit purposes
- Rollback procedures
Security and Multi-Tenancy
- Authentication and rate limiting
- Tenant isolation for prompts and logs
- Network isolation and egress control
- Audit logging of decisions
Requirements
- Experience in Linux administration and containerization.
- Familiarity with model inference servers and GPU management.
- Basic knowledge of REST APIs, load balancing, and monitoring.
Target Audience
- MLOps and platform engineers managing inference infrastructure.
- SRE teams supporting AI services.
- AI engineers deploying decision models on-premises.
- Architects designing air-gapped or regulated AI deployments.
