docs/platform/connector-development/connector-breaking-changes.md
Whenever possible, changes to connectors should be implemented in a non-breaking way. This allows users to upgrade to the latest version of a connector without additional action required on their part. Assume that every breaking change creates friction for users and should be avoided except when absolutely necessary.
When it is not possible to make changes in a non-breaking manner, additional breaking change requirements include:
breakingChanges entry in the releases section of the metadata.yaml file{connector-name}-migrations.md file.@airbytehq/breaking-change-reviewers GitHub team, which is automatically requested for review on migration guide changes (see CODEOWNERS). PRs with breaking changes must also be labeled with breaking-change.Source Example: fix!: schema change).A breaking change is any change that requires users to take action before they can continue to sync data. The following are examples of breaking changes:
The addition of new config options, new streams, or new stream properties should generally not be considered a breaking change. However, there is one important edge case to consider:
High-Volume Streams: Adding a new high-volume stream (one that produces 2x or more the volume of other streams) can break existing pipelines by overwhelming destination capacity or significantly increasing sync times. When adding high-volume streams to existing connectors:
suggestedStreams metadata.suggestedStreams set, add one that excludes the new high-volume stream.In some cases, breaking changes can be avoided through automated migrations:
Config Migrations - For low-code connectors, config migrations can automatically transform old configurations to new formats, avoiding the need for users to manually reconfigure. See the low-code config migration model for technical reference.
State Migrations - For low-code connectors, state migrations can automatically transform old state formats to new formats, avoiding the need for a full refresh. See the low-code state migration model for technical reference.
When considering spec or state changes, evaluate whether a migration can eliminate the need for a breaking change before proceeding with a major version bump.
When changing data types in schemas, consider type compatibility to determine whether the change is truly breaking:
Widening that is breaking downstream: Changes like float to str or datetime to str widen the type but may break downstream processes that expect numeric or date values.
Widening that is non-breaking: Changes like int to bigint or int to double widen the type in a compatible way that preserves data semantics.
Non-compatible type changes: Changes between fundamentally incompatible types (e.g., datetime to float or vice versa) are breaking and should be avoided if at all possible, as they change the semantic meaning of the data.
When in doubt, treat type changes as breaking unless you can verify that all downstream use cases will handle the new type correctly.
Some legitimate breaking changes may not impact all users of the connector. For example, a change to the schema of a specific stream only impacts users who are syncing that stream.
The breaking change metadata allows you to specify narrowed scopes, and specifically which streams are affected by a breaking change. If a user is not using the affected streams and therefore are not affected by a breaking change, they will not see any in-app messaging or emails about the change.
To scope a breaking change to specific streams, add the scopedImpact property to your metadata.yaml entry:
releases:
breakingChanges:
2.0.0:
message: "This version changes the cursor for the `users` stream. After upgrading, please reset the stream."
upgradeDeadline: "2024-03-01"
scopedImpact:
- scopeType: stream
impactedScopes: ["users"]
In this example, only users syncing the users stream are affected. Users syncing other streams can safely ignore this breaking change.
For the full schema and additional scope types, see the scopedImpact documentation.
Your migration guide must be created as a separate file at docs/integrations/{sources|destinations}/{connector-name}-migrations.md. The guide should be detailed and user-focused, addressing the following for each breaking change version:
Your migration guide can be as long as necessary and may include images, code snippets, SQL examples, and compatibility tables to help users understand and execute the migration.
Review these examples to understand the expected format and level of detail:
It's desirable for a migration guide to instruct your reader how to plan for, execute, and clean up after an upgrade. This information is applicable to most upgrades for most connectors, and you shouldn't normally need to document it. A reusable content snippet exists at docusaurus/static/_migration_guides_upgrade_guide.md. It contains generic upgrade information shared by every connector, and you can import it into your migration guide seamlessly.
This avoids duplicating content, increases the likelihood that documentation remains up-to-date, and makes it easier to author your migration guide. The only migration content you should author in a bespoke fashion should focus on the specifics of this connector and connector version.
Import the reusable content into your doc as a React component.
import MigrationGuide from '@site/static/_migration_guides_upgrade_guide.md';
Display it.
<MigrationGuide />
import MigrationGuide from '@site/static/_migration_guides_upgrade_guide.md';
# Asana Migration Guide
## Upgrading to 1.0.0
Here are the details of this breaking change that are specific to Asana.
## Connector upgrade guide
<MigrationGuide />
When adding a breakingChanges entry to your connector's metadata.yaml file, you must provide two critical fields:
The message field is a short summary shown in-app to all users of the connector. This message should:
campaigns stream"The platform uses this message to send automated emails to affected users, so clear communication is critical.
The upgradeDeadline field specifies the date by which users should upgrade (format: YYYY-MM-DD). When setting this deadline:
Minimum timelines:
Rationale: The deadline should provide enough time for users to review the migration guide, test in staging environments, and execute the migration steps.
Exception: Immediate upstream breakage: In the case of immediate upstream breaking changes, such as an already-removed upstream API endpoint, the deadline can be present-day or even in the past - with the rationale that users' connections are already broken without the fix and therefor need the upgrade applied immediately.
Automated notifications: The platform automatically emails users when a breaking change is released and sends reminders as the deadline approaches.
Significant breaking changes such as those required by a lift-and-shift or full connector rewrite should use a "-gen2" suffix and establish a new canonical connector ID. This approach provides several benefits:
This pattern is particularly appropriate when:
Example: Migration from legacy JSON-only "raw" tables to normalized typed columns in destination tables. These historic changes were so significant that they would break all downstream SQL transformations and BI dashboards. A "gen-2" approach in these cases gives users the ability to run both "Gen 1" and "Gen 2" in parallel, migrating only after they have had a chance to adapt their code to the new data models.
The following improvements to breaking change management are under consideration for future implementation:
breakingChanges metadata format