Skip to main content

Backup and Restore

Overview and configuration​

Depot relies on features of the primary storage for data backup, allowing cost-effective point-in-time backup and restore. This guide shows how to enable backups and how to switch an environment over to restored data.

DynamoDB​

Supporting backups

To use the backup feature, your DynamoDB location must enable point-in-time recovery as follows:

const ddbLocation = Location.DynamoDB(stack, "MyDDBLocation", {
  name: "my-ddb-location",
  environment: myDepotEnv,
  pointInTimeRecovery: true
});

Aurora​

Backups are enabled by default for 7 days only. Backups are taken during a pre-defined backup window, once daily. There is currently no further granularity than this.

Be aware of​

warning

Backups are not currently supported for environments that use Snowflake as a primary storage location.

warning

Currently, to perform x-account or x-region backups, AWS Backups must be used to manually configure a schedule targeting the sdp-<env id>-Object, sdp-<env id>-History and sdp-<env id>-Index tables. Scheduling these tasks will be exposed through CDK in a later release, but for manual configuration see: https://eu-west-1.console.aws.amazon.com/backup/home

Restore process​

Automated backup/restore removed

The EMR Serverless based backup and restore step functions, and the depot backup-dataset / depot restore-dataset CLI commands that drove them, have been removed along with the depot-spark platform module. Restoring data into a new store is now done with the primary storage's own tooling (DynamoDB point-in-time recovery, AWS Backup, or an Aurora snapshot). The Depot-side steps around that copy (creating a store, pausing datasets, switching the active store, resuming) still apply and are described below. YAML representations of the API commands are provided alongside the CLI commands.

Create a new store for the environment​

The restore process starts by creating a new (empty) store, with the new user-supplied storeId. This is possible with the CLI command:

depot create-store --newId=abc123
id: abc123
schema: Store

The new store id becomes part of the path in S3 storage. For example, if multiple restores have taken place with store ids restore01 and restore02, the S3 location would look like:

  • s3://{bucket-name}/gold/{env-id}/{dataset-id}/... (having an initial store id for consistency is planned in a future)
  • s3://{bucket-name}/gold/{env-id}/restore01/{dataset-id}/...
  • s3://{bucket-name}/gold/{env-id}/restore02/{dataset-id}/...
note

When restoring from a back-up of an environment that had previously been restored (and therefore has multiple stores), the back-up path should include the appropriate store id - e.g. s3://{bucket-name}/gold/{env-id}/restore02/.

Pause data updates to the dataset​

A dataset can be in one of the following states: running, pausing or paused.

To prevent updates to a dataset while it’s being restored, the process first pauses updates the dataset. This can be achieved using the following CLI command:

patch-dataset --id=abc123 --data='{"status": "PAUSING"}'
id: abc123
schema: Dataset
status: PAUSING

Data can still be read from the dataset while the dataset is paused.

The dataset will enter a pausing state until all in-progress writes have completed (after sufficient time has passed since last write), at which point the status will change to paused. Use the list-dataset command to check progress.

Behaviour while pausing/paused​

Rest and GraphQL API

Any clients attempting to update a data set through the Rest or GraphQL APIs will receive a 503 Service Unavailable error response.

Data Transactions

All data transactions will run to completion to ensure data is consistent, but no new transactions are be processed. New transactions will remain queued and will be processed once the environment has been restarted.

Load data into the new store​

Populate the new store's partition in the primary storage location from your backup using the storage's own restore tooling. Depot no longer ships a step for this.

Switch to the new datastore​

The environment can now be switched to use the newly migrated store using the CLI or PatchEnvironment API method:

depot patch-environment --data='{"store": {"id": "store02"}}'
id: env123
schema: Environment
store:
  id: store02

All requests to the environment will now use the new store.

note

Remember to restore all object datasets before switching the store, because all of them will use the new store. If a dataset does not have data in the new store, it will appear empty.

Resume data updates to the dataset​

Re-enable updates to a dataset after switching the store by running the following CLI command:

patch-dataset --id=abc123 --data='{"status": "RUNNING"}'
id: abc123
schema: Dataset
status: RUNNING

You can now make read and write request to the new store through the API.

At this point, data is being written to the new store's partition in all storage locations, and the restore is complete.

Disaster recovery

Whilst the old mechanism of constantly streaming data to S3 is in theory better in a true disaster recovery situation since any point in time can be recovered without creating a backup, it is inferior to the PITR mechanism provided by DynamoDB natively. In particular, it is very difficult to be sure that all data has been reliably replicated to S3, since a bug could be introduced in our handling lambda that renders the backup data invalid.

The proposed structure for a disaster-resilient deployment is as follows:

The delivery of this configuration as an option will be in a later release.