Astronomer - Apache Airflow 3 Fundamentals DAG Authoring Study Guide

8: Reliability and optimization - sensors, deferrable operators, callbacks, deadlines, concurrency, best practices

This chapter closes the authoring track by covering the reliability and optimization surface an Apache Airflow 3 DAG author has to reach for once a pipeline moves from a laptop into a shared cluster. It walks through the two waiting mechanisms (classic sensors and deferrable operators), the callback and Deadline Alert surface, the three-tier concurrency model, per-task priority weighting and mapped-task throttles, and the parse-time best practices that keep a scheduler and a triggerer responsive at scale.