Effective Data Orchestration with Airflow
This course provides an introduction to Airflow, starting with foundations and moving through advanced features such as sensors and third party operators. Throughout the course you'll build standard ingestion, model training, and data warehouse-based pipelines, and work with both the Taskflow API and the Astro SDK. This course will cover features through Airflow 2.5, the latest release covering a wide range of new improvements.
.jpeg&w=256&q=75)
Course taught by expert instructors
Henry Weller
Data Engineer at MongoDB
Henry is an experienced system builder working on making data infrastructure useful in various domains. He is currently a Data Engineer at MongoDB, and has held previous roles at Kevala, where he built a modern data stack orchestrated in Airflow, and Osaro, where he wrote control software for autonomous robotic systems.
.jpeg&w=750&q=75)
Mike Shwe
Senior Director of Product Management at Astronomer
Mike Shwe is an experienced product leader currently serving as Senior Director of Product Management at Astronomer, a driving force behind Apache Airflow. Mike was previously a Product Manager and Technical Program Manager at Google, Director of Data Engineering at Metaweb Technologies, and VP Products at M-Factor.
The course
Learn and apply skills with real-world projects.
Software engineers and data scientists new to the field of data engineering who are interested in learning how to build modern data pipelines.
Data engineers looking to level up their ETL skills, and adopt a general framework for their data use cases.
Ability to write Python and work with documented libraries
Familiarity with web applications, Docker basics, and the command line
Nice to have: Familiarity with Kubernetes, working within cloud compute environments such as AWS/GCS
Try these prep courses first
- Learn
- DAGs
- Operators, Tasks and Task Groups
- Intro to Taskflow API
- Astro CLI
- Airflow Architecture
- Airflow UI
Create a basic deployment of a cluster and your first useful DAG for processing timeseries energy data using the Taskflow API.
- Learn
- Sensors
- XComs
- Custom XCom Backends
- Dynamic Task Mapping
Build an energy price prediction ML pipeline and use XComs and dynamic task mapping to select the best model and save it to GCS.
- Learn
- Hooks and Dedicated Operators
- Data Warehouse Transformations
- Large Volume Workflows
- Control Plane vs Data Plane
Use dedicated BigQuery Operators to create external tables and views from data, perform SQL transformations and joins across data sources.
- Learn
- Astro SDK
- Data Warehouse Transformations Revisited
- Linking Pipelines Together Using Datasets
- Making Airflow more accessible to data analysts and SQL experts
Write an Astro SDK DAG that tightly couples Airflow tasks with data assets using File and Table abstractions.
Real-world projects
Work on projects that bring your learning to life.
Made to be directly applicable in your work.
Live access to experts
Sessions and Q&As with our expert instructors, along with real-world projects.
Network & community
Core reviews a study groups. Share experiences and learn alongside a global network of professionals.
Support & accountability
We have a system in place to make sure you complete the course, and to help nudge you along the way.
Course success stories
Learn together and share experiences with other industry professionals
As a product management professional with 20+ years of experience, I was nervous and excited to enroll in a Masters level course. Henry, a consummate teacher, played a tremendous role in helping me navigate a tough curriculum and be successful in my endeavor to learn. On many occasions, Henry helped clarify foundational elements needed for the course, reviewed specific lecture material, taught concepts from supplementary reading material suggested in the course and even helped me learn other reference material that would reinforce the learning.
Henry uses Airflow to implement data pipeline that handle large volumes of data ingestion, he shared his wealth of practical Airflow knowledge with colleagues to help them build modern data pipeline that are future proof. Henry is very knowledgeable on leveraging different Airflow operators that help improve productivity and performance.
