Databricks Migration Workshop: From Stored Procedures to Lakehouse (5-Day Intensive) Training Course

Introduction, Objectives, and Migration Strategy

• Defining course goals, aligning with participant profiles, and establishing success metrics
• Exploring high-level migration approaches and assessing associated risks
• Initial setup of workspaces, repositories, and lab datasets

Day 1 — Migration Fundamentals and Architecture

• Understanding Lakehouse concepts, an overview of Delta Lake, and Databricks architecture
• Distinguishing between SMP and MPP systems and their implications for migration
• Overview of Medallion (Bronze→Silver→Gold) design principles and Unity Catalog

Day 1 Lab — Translating a Stored Procedure

• Practical exercise: migrating a sample stored procedure to a Databricks notebook
• Mapping temporary tables and cursors to DataFrame transformations
• Validating outputs and comparing results with the original procedure

Day 2 — Advanced Delta Lake & Incremental Loading

• Exploring ACID transactions, commit logs, versioning, and time travel features
• Utilising Auto Loader, MERGE INTO patterns, upserts, and schema evolution
• Applying OPTIMIZE, VACUUM, Z-ORDER, partitioning, and storage tuning strategies

Day 2 Lab — Incremental Ingestion & Optimization

• Implementing Auto Loader ingestion pipelines and MERGE workflows
• Executing OPTIMIZE, Z-ORDER, and VACUUM operations while validating results
• Evaluating improvements in read and write performance

Day 3 — SQL in Databricks, Performance & Debugging

• Leveraging analytical SQL features such as window functions, higher-order functions, and JSON/array manipulation
• Interpreting Spark UI, DAGs, shuffles, stages, and tasks to diagnose bottlenecks
• Applying query tuning techniques including broadcast joins, hints, caching, and spill reduction

Day 3 Lab — SQL Refactoring & Performance Tuning

• Refactoring complex SQL processes into optimised Spark SQL queries
• Using Spark UI traces to identify and resolve data skew and shuffle issues
• Benchmarking performance before and after optimisation and documenting the tuning process

Day 4 — Tactical PySpark: Replacing Procedural Logic

• Understanding the Spark execution model: driver, executors, lazy evaluation, and partitioning strategies
• Converting loops and cursors into vectorised DataFrame operations
• Implementing modularisation, UDFs/pandas UDFs, widgets, and reusable libraries

Day 4 Lab — Refactoring Procedural Scripts

• Refactoring procedural ETL scripts into modular PySpark notebooks
• Introducing parametrisation, unit-style testing, and reusable functions
• Conducting code reviews and applying best-practice checklists

Day 5 — Orchestration, End-to-End Pipeline & Best Practices

• Designing Databricks Workflows: job structures, task dependencies, triggers, and error handling
• Architecting incremental Medallion pipelines with quality rules and schema validation
• Integrating with Git (GitHub/Azure DevOps), implementing CI, and establishing testing strategies for PySpark

Day 5 Lab — Build a Complete End-to-End Pipeline

• Assembling a Bronze→Silver→Gold pipeline orchestrated via Workflows
• Implementing comprehensive logging, auditing, retries, and automated validations
• Executing the full pipeline, validating outputs, and preparing deployment documentation

Operationalization, Governance, and Production Readiness

• Best practices for Unity Catalog governance, data lineage, and access controls
• Managing costs, cluster sizing, autoscaling, and job concurrency patterns
• Creating deployment checklists, rollback strategies, and operational runbooks

Final Review, Knowledge Transfer, and Next Steps

• Participant presentations showcasing migration outcomes and key lessons learned
• Conducting gap analysis, recommending follow-up actions, and handing over training materials
• Providing references, suggesting further learning paths, and outlining support options