Migrations: Observability

Work in progress

As of Depot 9.4.0, the migrations process is undergoing changes. The following features are implemented but additional observability and alerting features may be added in the future in order to facilitate diagnosis and recovery of migration failures.

The deployment of a package within a dataset goes through this state machine:

When the new version of the Depot package is registered, the entity creator lambda (snowflake or aurora) computes the Migration Plan from the previously deployed version. It sets the desiredVersion value of the runnning dataset to the new hash, stores the Migration Plan. Then:

• in case the dataset is set to migration mode synchronous, the lambda proceeds with the execution of the migration plan.
• in case the dataset is set to migration mode asynchronous, the lambda sends a message to the event bus, which triggers the execution of the migration plan in a separate process.

In both cases:

1. the attemptedVersion value is set to the new hash
2. the migration plan is executed
3. if successful, the deployedVersion value is set to the new hash

Whenever the desiredVersion, attemptedVersion or deployedVersion hashes are updated, a message is sent over EventBridge to the default event bus (or a custom event bus if configured), matching the following sample:

{
  "version": "0",
  "id": "dbc9acee-3189-2153-c22a-e0e893cf6634",
  "detail-type": "migration-lifecycle",
  "source": "tech.stage.depot",
  "account": "058264526050",
  "time": "2025-05-16T08:13:22Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "type": "lifecycle",
    "environmentId": "e2e404529091ed2",
    "datasetId": "9afc0c483ed3",
    "datasetAlias": "WsDevDogSrc",
    "environmentUri": "s3://sdp-bootstrap-058264526050-eu-west-1/sdp-e2e404529091ed2/datasets/9afc0c483ed3/a684a99ef01fa1fd5369705b887b1c6585e2c18bcb870c57718fdb46cb0647d2.json",
    "datasetVersionHash": "859fa99173740d4f85a379a32c99cb9f26839d3c20022de3af54b0e9c4f012bb",
    "event": "deployed"
  }
}

The event may contain one of four values:

event	description
desired	the dataset's desiredVersion has been updated: it is about to be migrated. The logical version of the dataset has already been updated within Depot.
attempted	the dataset's attemptedVersion has been updated: the migration process has started. The database state may be in the "before", "after" state, or anything in-between.
deployed	the dataset's deployedVersion has been updated: the migration process has completed successfully. The database state is now in the "after" state.
failed	The dataset migration process has failed and that failure has been detected. The database state is now in an unspecified state in between the "before" and "after" state. Manual recovery action may be required.

note

the "package version" used by the Depot migration engine does not use the package version at all. It is a hash of the contents of the package ("namespace") together with the dataset configuration. It is not monotonically increasing; it is only ever compared for equality.

The detail part conforms to the MigrationLifecycleEvent interface.

packages/depot-docs/docs/deployments-cdk/depot-cdk/documentation/classes/API.md packages/depot-docs/docs/deployments-cdk/depot-cdk/dataset/migrations/observability.md

Dataset migration state

It is possible to inspect the state of a dataset using the Depot CLI:

depot get-dataset --awsprofile depot-dev --env e2e404529091ed2 --id bdc659741d5a|jq '{id,alias,version,synced,status,desiredVersion,attemptedVersion,deployedVersion}'

The response looks like this

{
  "id": "bdc659741d5a",
  "alias": "WsDevPgDogSrc",
  "version": 1,
  "synced": "2025-05-12T11:51:06.589Z",
  "status": "RUNNING",
  "desiredVersion": null,
  "attemptedVersion": null,
  "deployedVersion": null
}

As a rule of thumb:

• desiredVersion, attemptedVersion and deployedVersion are all set to the same value: the dataset is in a stable state (no pending migration).
• desiredVersion is set to a value, attemptedVersion and deployedVersion to another: a migration has been scheduled but execution has not yet commenced.
• desiredVersion and attemptedVersion are set to a value, deployedVersion to another: a migration is in progress or a migration has failed.

Lifecycle eventbridge messages

Synchronous migrations

Note: the "desired version changed" lifecycle event isn't sent, but the dataset state is updated with desired = attempted = ${newHash} in a single PATCH operation.

Asynchronous migrations

Quick and dirty observability (until we have better)

You can route the Migration Lifecycle messages to your team alert channel by pasting the following cdk snippet in your service "data" stack:

new eb.Rule(this, `MigrationLifecycleRule`, {
  targets: [
    new targets.SnsTopic(alertTopic, {
      message: eb.RuleTargetInput.fromMultilineText(
        `Migration lifecycle event for dataset ${eb.EventField.fromPath('$.detail.datasetId')} (${eb.EventField.fromPath('$.detail.datasetAlias')}
             lifecycle event: ${eb.EventField.fromPath('$.detail.event')} -> version hash ${eb.EventField.fromPath('$.detail.versionHash')}
             in environment ${props.depotEnvironmentId}`,
      ),
    }),
  ],
  ruleName: `migration-lifecycle-alert`,
  description: `Notify (alert) rule for migration lifecycle events`,
  eventPattern: {
    source: ['tech.stage.depot'],
    detailType: ['migration-lifecycle'],
    detail: {
      // can also scope on datasetAlias and/or id
      event: ['desired', 'attempted', 'deployed', 'failed'],
    },
  },
});