Apache Airflow: Building and Managing Data Pipelines Training Course

Apache Airflow serves as an open-source framework designed for creating, scheduling, and tracking workflow processes. Leveraging Airflow 2.x, organizations can orchestrate data pipelines on a large scale, seamlessly integrate with cloud-based services, and guarantee strong observability and security within production settings.

This live, instructor-led session—available either online or on-site—is tailored for data and DevOps professionals ranging from beginner to intermediate levels who aim to utilize Apache Airflow for constructing, automating, and expanding end-to-end data pipelines.

Upon completion of this program, participants will be equipped to:

Deploy and configure Apache Airflow across various environments.
Create, schedule, and monitor DAGs utilizing the TaskFlow API.
Connect Airflow with diverse data sources, cloud platforms, and ML workflows.
Oversee ETL pipelines while ensuring data integrity and reliability.
Enhance the security and scalability of Airflow using executors, queues, and Kubernetes.
Implement observability standards and best practices for production deployments.

Course Structure

Engaging lectures paired with open discussions.
Extensive exercises and practical drills.
Real-time implementation within a live-lab setting.

Customization Availability

To request a tailored training program based on this course, please reach out to us for arrangements.

This course is available as onsite live training in Nigeria or online live training.

Thank you for sending your enquiry! One of our team members will contact you shortly.

Thank you for sending your booking! One of our team members will contact you shortly.

Course Outline

Introduction to Apache Airflow

Understanding the concept of workflow orchestration
Primary features and advantages of Apache Airflow
Enhancements in Airflow 2.x and an overview of its ecosystem

Architecture and Fundamental Concepts

Scheduler, web server, and worker process components
Understanding DAGs, tasks, and operators
Executors and backend options (Local, Celery, Kubernetes)

Deployment and Initial Setup

Installing Airflow in local and cloud-based environments
Configuring Airflow with various executor types
Establishing metadata databases and service connections

Using the Airflow UI and Command Line Interface

Exploring the Airflow web-based interface
Monitoring DAG executions, tasks, and log outputs
Utilizing the Airflow CLI for administrative tasks

Creating and Managing DAGs

Building DAGs using the TaskFlow API
Applying operators, sensors, and hooks
Managing task dependencies and scheduling intervals

Connecting Airflow to Data and Cloud Services

Linking databases, APIs, and message queues
Executing ETL pipelines via Airflow
Cloud integrations: AWS, GCP, and Azure operators

Monitoring and Observability

Task logging and real-time performance tracking
Tracking metrics using Prometheus and Grafana
Setting up alerting and notifications via email or Slack

Securing Apache Airflow

Implementing Role-Based Access Control (RBAC)
Authentication using LDAP, OAuth, and SSO
Managing secrets with Vault and cloud-based secret stores

Scaling Apache Airflow

Managing parallelism, concurrency, and task queues
Utilizing CeleryExecutor and KubernetesExecutor
Deploying Airflow on Kubernetes using Helm

Production Best Practices

Version control and CI/CD integration for DAGs
Testing strategies and debugging DAGs
Ensuring reliability and performance at scale

Troubleshooting and Performance Optimization

Diagnosing failed DAGs and tasks
Enhancing DAG execution performance
Identifying common pitfalls and strategies to prevent them

Conclusion and Future Steps

Requirements

Proficiency in Python programming
Basic understanding of data engineering or DevOps principles
General knowledge of ETL processes or workflow orchestration

Target Audience

Data scientists
Data engineers
DevOps and infrastructure specialists
Software developers

21 Hours

Need help picking the right course?
southafrica@nobleprog.co.za or +27 (0)10 005 5793

Testimonials (7)

The instructor adapted the training to the participants’ level and responded to all questions. He was very communicative, and it was easy to interact with him. I really appreciated the format of the training, which included many practical exercises. Overall, it was a very engaging and well-organized session.

Jacek Chlopik - ZAKLAD UBEZPIECZEN SPOLECZNYCH