Airflow 2.4 schedules DAGs on dataset updates, not only time

The release post announces Apache Airflow 2.4.0 with over 650 user-facing commits, including 46 features, 39 improvements, and 52 bug fixes. The headline change is data-aware scheduling (AIP-48): tasks declare Dataset outlets by URI, and other DAGs list those datasets in their schedule so a DAG run starts when the producing task finishes.

For data and ML pipeline teams, that shifts orchestration from clock-based triggers and cross-DAG sensors toward explicit data dependencies. Smaller producer and consumer DAGs can chain into one workflow without waiting on ExternalTaskSensor or TriggerDagRunOperator patterns the post says datasets can often replace with faster scheduling.

Datasets here are shared identifiers of meaning across DAG boundaries—an abstract dataset concept, not yet a read/write API. That URI-level contract is a step toward governed, system-readable links between pipelines, with the authors noting later 2.x releases will expand the foundation.