Reference for Database Migrations

.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Reference for Database Migrations '''''''''''''''''''''''''''''''''

Here's the list of all the Database Migrations that are executed via when you run airflow db migrate.

.. warning::

Those migration details are mostly used here to make the users aware when and what kind of migrations will be executed during migrations between specific Airflow versions. The intention here is that the "DB conscious" users might perform an analysis on the migrations and draw conclusions about the impact of the migrations on their Airflow database. Those users might also want to take a look at the :doc:database-erd-ref document to understand how the internal DB of Airflow structure looks like. However, you should be aware that the structure is internal and you should not access the DB directly to retrieve or modify any data - you should use the :doc:REST API <stable-rest-api-ref> to do that instead.

.. This table is automatically updated by prek hook by scripts/ci/prek/migration_reference.py .. All table elements are scraped from migration files .. Beginning of auto-generated table

+-------------------------+------------------+-------------------+--------------------------------------------------------------+ | Revision ID | Revises ID | Airflow Version | Description | +=========================+==================+===================+==============================================================+ | 1d6611b6ab7c (head) | 888b59e02a5b | 3.2.0 | Add bundle_name to callback table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 888b59e02a5b | 6222ce48e289 | 3.2.0 | Fix migration file ORM inconsistencies. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 6222ce48e289 | 134de42d3cb0 | 3.2.0 | Add partition fields to DagModel. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 134de42d3cb0 | e42d9fcd10d9 | 3.2.0 | Add partition_key to backfill_dag_run. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | e42d9fcd10d9 | f8c9d7e6b5a4 | 3.2.0 | Add allowed_run_types to dag. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | f8c9d7e6b5a4 | 53ff648b8a26 | 3.2.0 | Standardize UUID column format for non-PostgreSQL databases. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 53ff648b8a26 | a5a3e5eb9b8d | 3.2.0 | Add revoked_token table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | a5a3e5eb9b8d | 55297ae24532 | 3.2.0 | Make external_executor_id TEXT to allow for longer | | | | | external_executor_ids. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 55297ae24532 | e79fc784f145 | 3.2.0 | Add required fields to enable UI integrations for the | | | | | Deadline Alerts feature. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | e79fc784f145 | 0b112f49112d | 3.2.0 | add timetable_type to dag table for filtering. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 0b112f49112d | c47f2e1ab9d4 | 3.2.0 | Add exceeds max runs flag to dag model. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | c47f2e1ab9d4 | edc4f85a4619 | 3.2.0 | Add queue column to trigger table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | edc4f85a4619 | b12d4f98a91e | 3.2.0 | Enforce the new NOT NULL expectations for log.event | | | | | and dag.is_stale. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | b12d4f98a91e | 665854ef0536 | 3.2.0 | Drop id column from team table and make name the | | | | | primary key. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 665854ef0536 | e812941398f4 | 3.2.0 | Update ORM for asset partitioning. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | e812941398f4 | b87d2135fa50 | 3.2.0 | Replace deadline's inline callback fields with foreign key | | | | | to callback table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | b87d2135fa50 | 69ddce9a7247 | 3.2.0 | Restructure callback table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 69ddce9a7247 | 5cc8117e9285 | 3.2.0 | Add fail_fast column to dag table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 5cc8117e9285 | 1b2c3d4e5f6g | 3.2.0 | Add Human In the Loop Detail History table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 1b2c3d4e5f6g | ab6dc0c82d0e | 3.2.0 | Add length to dag_bundle_team.dag_bundle_name. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | ab6dc0c82d0e | 15d84ca19038 | 3.2.0 | Change serialized_dag data column to JSONB for | | | | | PostgreSQL. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 15d84ca19038 | 509b94a1042d | 3.2.0 | replace asset_trigger table with asset_watcher. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 509b94a1042d | 82dbd68e6171 | 3.1.8 | Change signed_url_template from VARCHAR(200) to TEXT. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 82dbd68e6171 | cc92b33c6709 | 3.1.8 | Add composite index (ti_id, id DESC) to task_reschedule. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | cc92b33c6709 | eaf332f43c7c | 3.1.0 | Add backward compatibility for serialized DAG format v3 to | | | | | v2. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | eaf332f43c7c | a3c7f2b18d4e | 3.1.0 | add last_parse_duration to dag model. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | a3c7f2b18d4e | 7582ea3f3dd5 | 3.1.0 | Add tables to store teams and associations with dag bundles. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 7582ea3f3dd5 | a169942745c2 | 3.1.0 | Make bundle_name not nullable. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | a169942745c2 | 808787349f22 | 3.1.0 | Remove dag_id from Deadline. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 808787349f22 | 3bda03debd04 | 3.1.0 | Modify deadline's callback schema. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 3bda03debd04 | f56f68b9e02f | 3.1.0 | Add url template and template params to DagBundleModel. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | f56f68b9e02f | 09fa89ba1710 | 3.1.0 | Add callback_state to deadline. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 09fa89ba1710 | 40f7c30a228b | 3.1.0 | Add trigger_id to deadline. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 40f7c30a228b | ffdb0566c7c0 | 3.1.0 | Add Human In the Loop Detail table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | ffdb0566c7c0 | 66a7743fe20e | 3.1.0 | Add dag_favorite table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 66a7743fe20e | 583e80dfcef4 | 3.1.0 | Add triggering user to dag_run. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 583e80dfcef4 | 3ac9e5732b1f | 3.1.0 | Add task_inlet_asset_reference table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 3ac9e5732b1f | 0242ac120002 | 3.1.0 | Change the on-delete behaviour of | | | | | task_instance.dag_version_id foreign key constraint to | | | | | RESTRICT. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 0242ac120002 | dfee8bd5d574 | 3.1.0 | Rename Deadline column in the Deadline table from deadline | | | | | to deadline_time and change its type from DateTime to UTC | | | | | DateTime. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | dfee8bd5d574 | fe199e1abd77 | 3.1.0 | Add Deadline to Dag. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | fe199e1abd77 | 29ce7909c52b | 3.0.3 | Delete import errors. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 29ce7909c52b | 959e216a3abb | 3.0.0 | Change TI table to have unique UUID id/pk per attempt. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 959e216a3abb | 0e9519b56710 | 3.0.0 | Rename is_active to is_stale column in dag | | | | | table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 0e9519b56710 | ec62e120484d | 3.0.0 | Rename run_type from 'dataset_triggered' to | | | | | 'asset_triggered' in dag_run table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | ec62e120484d | be2cc2f742cf | 3.0.0 | Add new otel span fields. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | be2cc2f742cf | d469d27e2a64 | 3.0.0 | Support bundles in DagPriorityParsingRequest. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | d469d27e2a64 | 16f7f5ee874e | 3.0.0 | Use ti_id as FK to TaskReschedule. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 16f7f5ee874e | cf87489a35df | 3.0.0 | Remove dag.default_view column. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | cf87489a35df | 7645189f3479 | 3.0.0 | Use TI.id as primary key to TaskInstanceNote. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 7645189f3479 | e00344393f31 | 3.0.0 | Add try_id to TI and TIH. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | e00344393f31 | 6a9e7a527a88 | 3.0.0 | remove external_trigger field. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 6a9e7a527a88 | 33b04e4bfa19 | 3.0.0 | Add DagRun run_after. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 33b04e4bfa19 | 8ea135928435 | 3.0.0 | add new task_instance field scheduled_dttm. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 8ea135928435 | e39a26ac59f6 | 3.0.0 | Add relative fileloc column. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | e39a26ac59f6 | 38770795785f | 3.0.0 | remove pickled data from dagrun table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 38770795785f | 5c9c0231baa2 | 3.0.0 | Add asset reference models. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 5c9c0231baa2 | 237cef8dfea1 | 3.0.0 | Remove processor_subdir. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 237cef8dfea1 | 038dc8bc6284 | 3.0.0 | Add deadline alerts table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 038dc8bc6284 | e229247a6cb1 | 3.0.0 | update trigger_timeout column in task_instance table to UTC. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | e229247a6cb1 | eed27faa34e3 | 3.0.0 | Add DagBundleModel. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | eed27faa34e3 | 9fc3fc5de720 | 3.0.0 | Remove pickled data from xcom table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 9fc3fc5de720 | 2b47dc6bc8df | 3.0.0 | Add references between assets and triggers. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 2b47dc6bc8df | d03e4a635aa3 | 3.0.0 | add dag versioning. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | d03e4a635aa3 | d8cd3297971e | 3.0.0 | Drop DAG pickling. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | d8cd3297971e | 5f57a45b8433 | 3.0.0 | Add last_heartbeat_at directly to TI. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 5f57a45b8433 | 486ac7936b78 | 3.0.0 | Drop task_fail table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 486ac7936b78 | d59cbbef95eb | 3.0.0 | remove scheduler_lock column. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | d59cbbef95eb | 05234396c6fc | 3.0.0 | Add UUID primary key to task_instance table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 05234396c6fc | 3a8972ecb8f9 | 3.0.0 | Rename dataset as asset. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 3a8972ecb8f9 | fb2d4922cd79 | 3.0.0 | Add exception_reason and logical_date to BackfillDagRun. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | fb2d4922cd79 | 5a5d66100783 | 3.0.0 | Tweak AssetAliasModel to match AssetModel after AIP-76. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 5a5d66100783 | c3389cd7793f | 3.0.0 | Add AssetActive to track orphaning instead of a flag. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | c3389cd7793f | 0d9e73a75ee4 | 3.0.0 | Add backfill to dag run model. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 0d9e73a75ee4 | 44eabb1904b4 | 3.0.0 | Add name and group fields to DatasetModel. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 44eabb1904b4 | 16cbcb1c8c36 | 3.0.0 | Update dag_run_note.user_id and task_instance_note.user_id | | | | | columns to String. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 16cbcb1c8c36 | 522625f6d606 | 3.0.0 | Remove redundant index. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 522625f6d606 | 1cdc775ca98f | 3.0.0 | Add tables for backfill. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 1cdc775ca98f | a2c32e6c7729 | 3.0.0 | Rename execution_date to logical_date. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | a2c32e6c7729 | 0bfc26bc256e | 3.0.0 | Add triggered_by field to DagRun. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 0bfc26bc256e | d0f1c55954fa | 3.0.0 | Rename DagModel schedule_interval to timetable_summary. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | d0f1c55954fa | 044f740568ec | 3.0.0 | Remove SubDAGs: is_subdag & root_dag_id columns from | | | | | DAG table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 044f740568ec | 5f2621c13b39 | 3.0.0 | Drop ab_user.id foreign key. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 5f2621c13b39 | 22ed7efa9da2 | 2.10.3 | Rename dag_schedule_dataset_alias_reference constraint | | | | | names. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 22ed7efa9da2 | 8684e37832e6 | 2.10.0 | Add dag_schedule_dataset_alias_reference table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 8684e37832e6 | 41b3bc7c0272 | 2.10.0 | Add dataset_alias_dataset association table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 41b3bc7c0272 | ec3471c1e067 | 2.10.0 | Add try_number to audit log. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | ec3471c1e067 | 05e19f3176be | 2.10.0 | Add dataset_alias_dataset_event. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 05e19f3176be | d482b7261ff9 | 2.10.0 | Add dataset_alias. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | d482b7261ff9 | c4602ba06b4b | 2.10.0 | Add task_instance_history. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | c4602ba06b4b | 677fdbb7fc54 | 2.10.0 | Added DagPriorityParsingRequest table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 677fdbb7fc54 | 0fd0c178cbe8 | 2.10.0 | add new executor field to db. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 0fd0c178cbe8 | 686269002441 | 2.10.0 | Add indexes on dag_id column in referencing tables. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 686269002441 | bff083ad727d | 2.9.2 | Fix inconsistency between ORM and migration files. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | bff083ad727d | 1949afb29106 | 2.9.2 | Remove idx_last_scheduling_decision index on | | | | | last_scheduling_decision in dag_run table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 1949afb29106 | ee1467d4aa35 | 2.9.0 | update trigger kwargs type and encrypt. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | ee1467d4aa35 | b4078ac230a1 | 2.9.0 | add display name for dag and task instance. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | b4078ac230a1 | 8e1c784a4fc7 | 2.9.0 | Change value column type to longblob in xcom table for | | | | | mysql. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 8e1c784a4fc7 | ab34f260b71c | 2.9.0 | Adding max_consecutive_failed_dag_runs column to dag_model | | | | | table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | ab34f260b71c | d75389605139 | 2.9.0 | add dataset_expression in DagModel. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | d75389605139 | 1fd565369930 | 2.9.0 | Add run_id to (Audit) log table and increase event name | | | | | length. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 1fd565369930 | 88344c1d9134 | 2.9.0 | Add rendered_map_index to TaskInstance. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 88344c1d9134 | 10b52ebd31f7 | 2.8.1 | Drop unused TI index. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 10b52ebd31f7 | bd5dfbe21f88 | 2.8.0 | Add processor_subdir to ImportError. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | bd5dfbe21f88 | f7bf2a57d0a6 | 2.8.0 | Make connection login/password TEXT. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | f7bf2a57d0a6 | 375a816bbbf4 | 2.8.0 | Add owner_display_name to (Audit) Log table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 375a816bbbf4 | 405de8318b3a | 2.8.0 | add new field 'clear_number' to dagrun. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 405de8318b3a | 788397e78828 | 2.7.0 | add include_deferred column to pool. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 788397e78828 | 937cbd173ca1 | 2.7.0 | Add custom_operator_name column. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 937cbd173ca1 | 4bc4d934e2bc | 2.7.0 | Add index to task_instance table. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | 4bc4d934e2bc (base) | None | 2.6.2 | Create initial database state from Airflow v2.6.2. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+

.. End of auto-generated table

.. spelling:word-list:: branchpoint mergepoint