Iceberg interop case studies

An S3 Tables dataset is Apache Iceberg on disk, so engines other than Snowflake can read and write it directly. Every case below has the same shape, and the same asymmetry.

Where the data moves

Both sides read and write the same files. Nothing is copied, and no data passes through Depot.

Only one edge lags: Snowflake reads through catalog metadata it caches, refreshed on a poll cycle (refreshIntervalSeconds, default 3600). So:

• Depot → external is immediate. The files are on disk the moment Snowflake commits.
• External → Depot is not, until the cache turns over or a REFRESH transaction forces it.

Two more things apply to every case. Write to a record schema — it carries none of Depot's metadata columns — and read it back through a query schema wrapping it, since a record has no direct read path; see Reading a record. And grant the engine access to the table bucket, either with directReadPrincipals / directWritePrincipals on the dataset or with a DatasetPrincipalPermission carrying purposes from the consumer's own stack.

Writes go through S3 Tables, not Glue

A dataset is reachable through the S3 Tables endpoints and through the Glue Iceberg REST catalog. Both read; only the S3 Tables ones write. A commit over Glue is an UpdateTable, and Glue requires glue:UpdateTable in the caller's own identity policy — something Depot cannot grant on a role it does not own, and which a Lake Formation grant does not stand in for. Point writers at S3 Tables, where the table-bucket policy Depot writes is enough on its own.

Athena ↔ Snowflake

Athena reaches the bucket through the federated catalog, addressed as s3tablescatalog/<bucket>. Names are the physical ones — lowercase, underscores at camelCase boundaries — so petstore.Reading.runId is reading.run_id.

Write in Depot, read in Athena. No step in between.

mutation { createPetstoreObservation(data: { sensor: "sensor-a", runId: "run-42" }) { id } }

SELECT sensor, run_id FROM "s3tablescatalog/my-bucket"."my_namespace"."observation"
WHERE run_id = 'run-42'

Write in Athena, read in Depot. Needs the refresh.

INSERT INTO "s3tablescatalog/my-bucket"."my_namespace"."reading" (sensor, run_id)
VALUES ('sensor-a', 'run-42')

{ "actions": [ { "operation": "REFRESH", "target": "petstore.Reading" } ] }

Then read petstore.Reading through its query schema. Skip the REFRESH and the row simply is not there yet.

DuckDB ↔ Snowflake

DuckDB reads and writes a Depot dataset on an S3 Tables location directly, by attaching the dataset's table bucket as an Iceberg catalog.

Run DuckDB next to the bucket

DuckDB downloads every file it scans to the machine it runs on. Never point it at production-grade data from a laptop. The furthest it may run from the data is a cloud shell in the table bucket's own region.

Ask the Storage Locator for the dataset, then attach the route that matches what you do:

Route kind	ATTACH	Reads	Writes
S3_TABLES_REST	ATTACH '<warehouse>' AS s3t (TYPE iceberg, ENDPOINT_TYPE s3_tables)	yes	yes
GLUE_REST	ATTACH '<warehouse>' AS glue_cat (TYPE iceberg, ENDPOINT_TYPE 'glue')	yes, under Lake Formation	no

<warehouse> is the route's warehouse: the table bucket's ARN for S3_TABLES_REST, <account>:s3tablescatalog/<bucket> for GLUE_REST. The table is "<catalog>"."<table.namespace>"."<table.name>", and its columns carry physical names; the response's schema.propertyColumn maps each Depot property to its column.

INSTALL aws; INSTALL httpfs; INSTALL iceberg;
LOAD aws; LOAD httpfs; LOAD iceberg;

CREATE SECRET (TYPE s3, PROVIDER credential_chain, REGION 'eu-west-1');
ATTACH 'arn:aws:s3tables:eu-west-1:111122223333:bucket/my-bucket'
  AS s3t (TYPE iceberg, ENDPOINT_TYPE s3_tables);

With temporary credentials, give the secret KEY_ID, SECRET and SESSION_TOKEN explicitly instead of using the credential chain.

Write in Depot, read in DuckDB. As above, immediate.

SELECT sensor, run_id FROM s3t.my_namespace.observation WHERE run_id = 'run-42';

Write in DuckDB, read in Depot. Same refresh, same reason.

INSERT INTO s3t.my_namespace.reading (sensor, run_id) VALUES ('sensor-a', 'run-42');

{ "actions": [ { "operation": "REFRESH", "target": "petstore.Reading" } ] }

Spark

Covered by DPT-3695, which extends the Locator client in salt-repertoire to write record schemas.