Skip to main content

Location

Overview​

A Location defines where data is stored for your Depot environment Datasets. Supported location types include DynamoDB, Snowflake, Aurora, Cognito, ElasticSearch, Glue, and S3Tables. See the API documentation for exact Location types. Depot CDK API.

note

Location.Iceberg is deprecated — use Location.S3Tables instead.

Add a Location using one of the Location sub-types. For example to define a DynamoDB storage Location, you'll use the Depot CDK Location.DynamoDB Construct.

new Location.DynamoDB(this, "DDBLocation", {
    environment: depotEnvironment,
    name: "ddb-location",
    capacityMode: DynamoDBLocationCapacityMode.ONDEMAND
});

Location modes​

Explicit locations​

Locations can be set directly for Datasets, or a Composite Location can be used to designate 'tiered' storage location behaviour.

For e.g.:

const myDDBLocation = new Location.DynamoDB(this, "DynamoDBLocation", {
    name: "my-ddb-location",
    environment: myEnvironment,
    capacityMode: DynamoDBLocationCapacityMode.ONDEMAND
});

const myESLocation = new Location.ElasticSearch(this, "ElasticsearchLocation", {
    name: "my-es-location",
    environment: myEnvironment
});

const myCompositeLocation = new Location.Composite(this, "CompositeLocation", {
    name: "my-composite-location",
    environment: myEnvironment,
    primaryLocation: myDDBLocation,
    replicaLocation: myESLocation,
    replicationMode: ReplicationMode.IMMEDIATE
});

const myDataset = new Dataset(this, "MyDataset", {
    name: "my-dataset",
    environment: myEnvironment,
    package: myPackage,
    location: myCompositeLocation
});

Under this location model, the ordering of primary/replica can be controlled on a per-dataset basis. The following general rules apply:

When writing, the primaryLocation of the root composite location is written first, then any changes are written to the replicaLocation location, taking into account the replicationMode:

  • ReplicationMode.IMMEDIATE: Changes are written either synchronously (before return to the caller), if the request specifies consisitency:ATOMIC_ALL or using a short-term SQS -> SNS queue.
  • ReplicationMode.ASYNC: Changes are written using a batched process appropriate to the target replica storage (e.g. Spark or Snowflake transactions)

When querying, the primaryLocation of the root composite location is queried first, then fallback occurs through replica locations (which may themselves be composite locations).

Note: At present there is no equivalent setting for special cognito handling of the User schema, this will be implemented in a future release using schema-specific location mapping.

Location Specific Configuration​

Snowflake​

Snowflake locations can designate a default Executor to be used as a fallback for a Dataset if no specific Dataset Purposes have been assigned. Example:

const sfkExecutor = new Executor.Snowflake(this, "Executor", {
    environment: depotEnvironment,
    name: "snowflake-executor",
    credentials: {
        credentialsSecretArn: "arn:aws:secretsmanager:eu-west-1:123456789012:secret:secret-12345",
        accountId: "AB12345",
        region: "eu-west-1"
    },
    warehouse: SnowflakeWarehouse.existing("EXISTING_WAREHOUSE")
});

const sfkLocation = new Location.Snowflake(this, "SnowflakeLocation", {
    environment: depotEnvironment,
    name: "snowflake-location",
    createEntities: true,
    readable: true,
    writable: true,
    executor: sfkExecutor
});