Add ruff rules to catch deprecated Airflow imports for Airflow 3.1

[amoghrajesh]

Body

Description

We need to implement ruff rules to catch and/or auto-fix deprecated imports in Airflow for Airflow 3.1. These rules should help users migrate from old import paths to new ones.

Implementation Checklist

airflow.utils module

• airflow.utils.setup_teardown.BaseSetupTeardownContext → airflow.sdk.definitions._internal.setup_teardown.BaseSetupTeardownContext
• airflow.utils.setup_teardown.SetupTeardownContext → airflow.sdk.definitions._internal.setup_teardown.SetupTeardownContext
• airflow.utils.xcom.XCOM_RETURN_KEY → airflow.models.xcom.XCOM_RETURN_KEY
• airflow.utils.task_group.TaskGroup → airflow.sdk.TaskGroup
• airflow.utils.task_group.get_task_group_children_getter → airflow.sdk.definitions.taskgroup.get_task_group_children_getter
• airflow.utils.task_group.task_group_to_dict → airflow.sdk.definitions.taskgroup.task_group_to_dict
• airflow.utils.timezone.* → airflow.sdk.timezone
• airflow.utils.decorators.remove_task_decorator → airflow.sdk.definitions._internal.decorators.remove_task_decorator
• airflow.utils.decorators.fixup_decorator_warning_stack → airflow.sdk.definitions._internal.decorators.fixup_decorator_warning_stack

airflow.sensors module

• airflow.sensors.base.BaseSensorOperator → airflow.sdk.bases.sensor.BaseSensorOperator
• airflow.sensors.base.PokeReturnValue → airflow.sdk.bases.sensor.PokeReturnValue
• airflow.sensors.base.poke_mode_only → airflow.sdk.bases.sensor.poke_mode_only
• airflow.sensors.python.PythonSensor → airflow.providers.standard.sensors.python.PythonSensor
• airflow.sensors.bash.BashSensor → airflow.providers.standard.sensors.bash.BashSensor
• airflow.sensors.date_time.DateTimeSensor → airflow.providers.standard.sensors.date_time.DateTimeSensor
• airflow.sensors.date_time.DateTimeSensorAsync → airflow.providers.standard.sensors.date_time.DateTimeSensorAsync

airflow.secrets module

• airflow.secrets.cache.SecretCache → airflow.sdk.execution_time.cache.SecretCache

airflow.notifications module

• airflow.notifications.basenotifier.BaseNotifier → airflow.sdk.bases.notifier.BaseNotifier

airflow.models module

• airflow.models.abstractoperator.AbstractOperator → airflow.sdk.definitions._internal.abstractoperator.AbstractOperator
• airflow.models.abstractoperator.NotMapped → airflow.sdk.definitions._internal.abstractoperator.NotMapped
• airflow.models.abstractoperator.TaskStateChangeCallback → airflow.sdk.definitions._internal.abstractoperator.TaskStateChangeCallback
• airflow.models.param.Param → airflow.sdk.definitions.param.Param
• airflow.models.param.ParamsDict → airflow.sdk.definitions.param.ParamsDict
• airflow.models.baseoperator.BaseOperator → airflow.sdk.bases.operator.BaseOperator
• airflow.models.baseoperator.chain → airflow.sdk.bases.operator.chain

airflow.macros module

• airflow.macros.* → airflow.sdk.execution_time.macros

airflow.io module

• airflow.io.get_fs → airflow.sdk.io.get_fs
• airflow.io.has_fs → airflow.sdk.io.has_fs
• airflow.io.attach → airflow.sdk.io.attach
• airflow.io.Properties → airflow.sdk.io.Properties
• airflow.io.path.ObjectStoragePath → airflow.sdk.ObjectStoragePath

airflow.hooks module

• airflow.hooks.filesystem.FSHook → airflow.providers.standard.hooks.filesystem.FSHook
• airflow.hooks.base.BaseHook → airflow.sdk.bases.hook.BaseHook

airflow.decorators module

• airflow.decorators.dag → airflow.sdk.dag
• airflow.decorators.task → airflow.sdk.task
• airflow.decorators.task_group → airflow.sdk.task_group
• airflow.decorators.base.DecoratedOperator → airflow.sdk.bases.decorator.DecoratedOperator

Notes

• Some imports may have already been done
• We can use this as a knowledge base to develop assertion dags that import and validate old vs new paths for better user onboarding
• Consider phasing the implementation if the scope is too large

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[amoghrajesh]

CC @phanikumv / @vatsrahul1001

[aayushbisen]

Hey @amoghrajesh can I work on this task?

[amoghrajesh]

cc @sunank200 for some guidance on this

[amoghrajesh]

@aayushbisen yeah you can get involved in this effort! Wait till we create a formal plan for this.

[Lee-W]

Hey @amoghrajesh , I tried to list down a few things we already have with a few questions.

Also, AIR311 and AIR312 are the ones that suggest users to update, but won't break. Will the user still be able to use things like airflow.sensors.date_time.DateTimeSensor? If so, I think we'll need to create a new error code (e.g., AIR313 required to update after Airflow 3.1)

For the AIR301, we don't need to do anything. They won't work for 3.0 anyway.

There are some missing ones, probably due to missing newsfragments back to the time we list down the rules.

---

airflow.utils module

• airflow.utils.timezone.* → airflow.sdk.timezone (would be better to actaully list down everything)

airflow.sensors module

• airflow.sensors.base.BaseSensorOperator → airflow.sdk.bases.sensor.BaseSensorOperator (AIR301)
• airflow.sensors.date_time.DateTimeSensor → airflow.providers.standard.sensors.date_time.DateTimeSensor (AIR312)
• airflow.sensors.date_time.DateTimeSensorAsync → airflow.providers.standard.sensors.date_time.DateTimeSensorAsync (AIR312)

airflow.secrets module

• airflow.secrets.cache.SecretCache → airflow.sdk.execution_time.cache.SecretCache (AIR301) (should it be airflow.sdk.SecretCache instead?)

airflow.notifications module

• airflow.notifications.basenotifier.BaseNotifier → airflow.sdk.bases.notifier.BaseNotifier (AIR301)

airflow.models module

• airflow.models.abstractoperator.AbstractOperator → airflow.sdk.definitions._internal.abstractoperator.AbstractOperator (should these internal still be used?)
• airflow.models.abstractoperator.NotMapped → airflow.sdk.definitions._internal.abstractoperator.NotMapped
• airflow.models.baseoperator.chain → airflow.sdk.bases.operator.chain (AIR301) (should it be airflow.sdk.chain instead)

airflow.macros module

• airflow.macros.* → airflow.sdk.execution_time.macros (would be better to actually list down everything)

airflow.io module

• airflow.io.attach → airflow.sdk.io.attach (do you mean airflow.io.store.attach?) (AIR312)
• airflow.io.path.ObjectStoragePath → airflow.sdk.ObjectStoragePath (AIR312)

airflow.hooks module

• airflow.hooks.filesystem.FSHook → airflow.providers.standard.hooks.filesystem.FSHook (AIR311)

airflow.decorators module

• airflow.decorators.dag → airflow.sdk.dag (AIR311)
• airflow.decorators.task → airflow.sdk.task (AIR311)
• airflow.decorators.task_group → airflow.sdk.task_group (AIR311)

---

The following are these ones missed in AIR311 or AIR312 and I'll create a PR tomorrow

• airflow.sensors.base.BaseSensorOperator → airflow.sdk.bases.sensor.BaseSensorOperator
• airflow.sensors.base.PokeReturnValue → airflow.sdk.bases.sensor.PokeReturnValue
• airflow.sensors.base.poke_mode_only → airflow.sdk.bases.sensor.poke_mode_only
• airflow.sensors.python.PythonSensor → airflow.providers.standard.sensors.python.PythonSensor
• airflow.sensors.bash.BashSensor → airflow.providers.standard.sensors.bash.BashSensor
• airflow.decorators.base.DecoratedOperator → airflow.sdk.bases.decorator.DecoratedOperator
• airflow.models.param.Param → airflow.sdk.definitions.param.Param

[phanikumv]

Discussed with @Lee-W , this could be done post 3.1.0

[amoghrajesh]

Sure can!

[sjyangkevin]

I am recently working on astral-sh/ruff#22046 with @Lee-W , and I am wondering if this task is something I could also work on.