Skip to main content

S3 Tables location

S3 Tables is AWS's managed Apache Iceberg table storage built directly into S3. Use this location type to store Depot Dataset data in an S3 Tables bucket.

Work in progress

Full S3 Tables support — including automatic bucket provisioning and namespace configuration — is being implemented in a future release. You can register the location type now, but no operational functionality (transactions, queries) is available yet.

Configuration properties​

PropertyRequiredDefaultDescription
nameyes—A name or alias for this location. Must be unique within the environment.
tableBucketNamenosdp-${envId}-dataOverride the default table bucket name. Strongly discouraged — AWS allows only 10 table buckets per region per account. Use only if you must point at a pre-existing bucket or share a bucket across environments.
tableBucketArnno—ARN of an externally-provisioned S3 Tables bucket. When set, Depot does not create or manage the bucket lifecycle — it only creates its own Catalog Integration and Linked Catalog Database against the external bucket. Mutually exclusive with tableBucketName.
removalPolicynoRETAINRemoval policy for the bucket that backs this location. One of DESTROY, RETAIN, SNAPSHOT. Use DESTROY for non-production.
icebergVersionno2Iceberg table format version to use for tables created under this location. Accepts 2 or 3. Use 3 only if all consumers (Snowflake, Athena, etc.) support Iceberg v3 in your environment — consult the Iceberg compatibility matrix to verify reader support before opting in. Per-table override: snowflake.iceberg.icebergVersion schema extension.
deleteModenoMERGE_ON_READRow-level delete strategy for Iceberg tables. MERGE_ON_READ writes delete files and lets readers reconcile. COPY_ON_WRITE rewrites data files on every delete or update. Per-table override: snowflake.iceberg.deleteMode schema extension.

Example​

import { Location } from '@stage-tech/depot-cdk';

new Location.S3Tables(this, 'MyDataLake', {
  environment,
  name: 'my-data-lake',
  // icebergVersion: 3,
  // deleteMode: 'COPY_ON_WRITE',
});

External bucket mode​

Use tableBucketArn to point Depot at a bucket provisioned and managed outside of Depot (e.g. shared infrastructure or a cross-team bucket).

In external mode Depot:

  • does not create or delete the AWS::S3Tables::TableBucket resource
  • creates its own Snowflake Catalog Integration and Linked Catalog Database against the external bucket
  • does not manage the bucket or namespace lifecycle
new Location.S3Tables(this, 'ExternalLake', {
  environment,
  name: 'external-lake',
  tableBucketArn: 'arn:aws:s3tables:eu-west-1:111122223333:bucket/shared-bucket',
});

Dataset-level namespace management is configured separately on the dataset's s3tables extension. Set namespaceManagement: 'EXTERNAL' there if the namespace itself is also externally managed.

Cross-account external buckets​

When tableBucketArn belongs to a different AWS account, Depot cannot automate Lake Formation setup — the owning account must configure the following manually before Snowflake queries will succeed:

  1. Account-level prerequisites (once per account): register arn:aws:s3tables:{region}:{account}:bucket/* as a Lake Formation data location (WithFederation: true) using a role that trusts lakeformation.amazonaws.com with sts:AssumeRole. Create a Glue federated catalog (s3tablescatalog) pointing to that wildcard ARN.

  2. Catalog-level grant (once per bucket): grant ALL on the default Glue catalog and on the S3 Tables sub-catalog ({ownerAccountId}:s3tablescatalog/{bucketName}) to the Snowflake IAM role.

  3. Namespace-level grant (once per dataset): grant ALL on Database { CatalogId: "{ownerAccountId}:s3tablescatalog/{bucketName}", Name: "{namespaceName}" } to the Snowflake IAM role.

The Snowflake IAM role must also have lakeformation:GetDataAccess in its IAM policy. Cross-account LF grants may additionally require AWS Resource Access Manager (RAM) sharing — consult the AWS Lake Formation cross-account documentation.

Iceberg location deprecation​

The Location.Iceberg construct is deprecated. Use Location.S3Tables instead — S3 Tables is AWS's successor offering managed Iceberg table storage with no operational overhead.