fix(scheduler): set end_date on tasks skipped by dagrun timeout#63250
fix(scheduler): set end_date on tasks skipped by dagrun timeout#63250YoannAbriel wants to merge 1 commit intoapache:mainfrom
Conversation
|
Did you actually check if it works? Or is that change just PR-ed without actually running some real tests? Can you show screenshots and logs from the execution? |
|
explanation: This is a vital part of airflow in order to modify scheduler you need to know exactly what you are doing and actually test it. |
|
Fair — unit tests only so far. Will set up a local environment with a DAG that hits the timeout path and share logs/screenshots. |
|
Reproduced and verified on Airflow 3.1.8 (standalone, SequentialExecutor). Test DAG with Before fix (unpatched 3.1.8):
After fix:
Two lines in |
2b79864 to
64899cc
Compare
06edc79 to
6ebc21a
Compare
24902f4 to
c497cda
Compare
9e738f5 to
d817d2a
Compare
74c3ea4 to
df2246a
Compare
When a DAG run times out via dagrun_timeout, unfinished tasks are marked as SKIPPED but end_date was not set. This caused task duration to keep increasing in the UI even though the task was already skipped. Set end_date to the current time when marking tasks as SKIPPED during DAG run timeout handling. Closes: apache#58536
df2246a to
8d5248c
Compare
Problem
When a DAG run times out via
dagrun_timeout, unfinished task instances are markedSKIPPEDbut theirend_dateis never set. The UI computes duration asnow - start_date, so skipped tasks show a continuously increasing duration — confusing and incorrect.Root Cause
In
SchedulerJobRunner._schedule_dag_run, the timeout handler setstask_instance.state = TaskInstanceState.SKIPPEDbut doesn't touchend_date. Without anend_date, the UI falls back to computing live duration.Fix
Set
end_date = timezone.utcnow()alongside the state change for all tasks skipped by dagrun timeout. Added a test that creates a timed-out DAG run with running tasks and verifiesend_dateis populated after the scheduler processes it.Closes: #58536
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code following the guidelines