Depot Spark Connector
The Depot Spark Connector gives Spark SQL-native, read-only access to Depot-controlled datasets, whichever store they live in — Aurora Postgres, Snowflake, or S3 Tables/Iceberg — through one Spark catalog. Datasets across stores can be joined in a single SQL statement; Iceberg-backed datasets also support time travel.
Before you start: the connector relies on the Storage Locator
Every read goes through the Storage Locator, so two things have to be true before a job can read anything:
- The target Depot environment has the Locator enabled (
admin.enabled+locator.enabled). - The job's own identity holds
lambda:InvokeFunctionon it, plus one authentication permission per store it actually reads:rds-db:connectfor Aurora, the deploy-time S3 Tables / Lake Formation grant for Iceberg, and read access to the credential secret for Snowflake. The Locator resolves metadata; it never hands out authentication — see the Locator page for what that means precisely.
Setup and registration
Add a dependency on the Depot Spark Connector to your project, then mount one Depot environment per catalog you want to read through:
import tech.stage.depot.spark.implicits._
val depotE2e = spark.mountDepot("depot:aws:<accountId>:eu-west-1:<depotEnvironmentId>").as("depot_e2e")
mountDepot(uri) parses the same depot:<partition>:<accountId>:<region>:<environmentId> shape
used elsewhere in these docs, and returns a builder that has registered nothing yet. Commit it with
either .as("name") — a real catalog name, and the one to prefer — or an explicit .anonymous() for
handle-only code. .as(...) is strongly advocated: an anonymous mount's generated name
(__depot_0, __depot_1, …) is a valid identifier but leaks into query plans and error messages and
is not stable across runs.
Keep catalog/alias names to letters, digits and underscore — depot_e2e, sphinx_master — since a
dot or hyphen forces backticks on every reference's catalog segment, on top of the ones dotted dataset
ids and schema names already need.
The raw underlying mechanism — registering spark.sql.catalog.<name> = tech.stage.depot.DepotCatalog
plus its partition/accountId/region/environmentId properties by hand with spark.conf.set —
still works and is what mountDepot does for you. Reach for it directly only if you have a reason to;
mountDepot is the documented path.
Reading a dataset
val items = depotE2e.read("WsDevPgDogSrc", "DogSource")
items.filter(col("testId").startsWith("AUR2SFK-")).show()
The handle backticks dataset ids and schema names for you, which matters because both commonly contain
dots (ax.replica, copyright.ice.data.pump.work.service.WorkMerged) that would otherwise split
under the SQL identifier parser. The raw, fully-qualified form also works and resolves identically:
spark.read.table("depot_e2e.`WsDevPgDogSrc`.`DogSource`")
The connector resolves whether the dataset lives in Aurora, Snowflake or S3 Tables and reads it with the appropriate driver or catalog underneath — nothing about the call differs from the caller's point of view. Column names come back converted from the underlying database's naming convention into the one declared in the original Depot schema.
Per-read options that only affect how the store is read — not which rows or columns — still work
with .option(...), e.g. fetchsize for Postgres/Snowflake JDBC reads:
spark.read.option("fetchsize", "100000").table(depotE2e.table("WsDevPgDogSrc", "DogSource"))
Expansion of ref types and of transients is not supported. Join the data in Spark to reconstruct your data graphs as appropriate.
How nested properties appear
A Depot property can be nested — store.id, owner.name — but Spark's identifier parser splits on
dots, so the connector has to pick a representation. It is a mount option rather than a per-read one,
because it decides the shape of the table:
spark.mountDepot(uri).withOptions(nesting = Flat("$")).as("depot_e2e") // the default
spark.mountDepot(uri).withOptions(nesting = Structs).as("depot_e2e")
| mode | store.id surfaces as | notes |
|---|---|---|
Flat(sep) | store$id | the default; sep defaults to $ |
Structs | store, a real StructType | reached as store.id |
Camel | storeId | flat, camelCased |
The separator has a session-level fallback, spark.depot.nesting.separator, as does idCompanions
below (spark.depot.nesting.idCompanions). The spark. prefix is mandatory rather than decorative:
Spark discards any key not starting with it when reading --conf, spark-defaults.conf or an EMR job
configuration, so a bare depot.nesting.separator would be silently ignored — precisely in the cases a
session-level default exists to serve.
Companion columns for refs. idCompanions is orthogonal to the mode. With it on, a ref property
also surfaces a camelCase companion alongside whatever the mode produced, so a filter can address the
raw column directly:
| the ref's flattened subfields | emits |
|---|---|
exactly {id} | <name>Id |
exactly {id, version} | <name>Id and <name>Version |
It triggers on ref-ness, not on a name heuristic, so a struct that merely happens to carry an id field
gains nothing. The connector default is off, since new consumers should not inherit surprise
columns. Under Camel the companion is already the mode's own output for the <name>.id case, so
combining the two adds nothing rather than emitting a duplicate column.
Modes are projections, so handles can be re-derived cheaply. nesting and idCompanions apply
above the scan, so changing one needs no new catalog, no new table and no re-resolution:
val flat = spark.mountDepot(uri).as("depot_e2e")
val nested = flat.withOptions(nesting = Structs) // same catalog, same table, same envelope
withOptions on an existing mount returns a derived handle sharing the underlying registration —
one Locator resolution, one envelope cache, one table identity — which is also why Spark can reuse the
scan when two modes appear in the same query. Options that fix the connection or the table's schema
(dbUser, authenticator, executorId) are settled at mount time and cannot be varied this way. See
Caching and catalog instances.
Reading the history table
A schema's history (append log of every change) is a sibling table under a history namespace level,
not an option — pass history = true:
depotE2e.read("WsDevPgDogSrc", "DogSource", history = true)
// raw form: spark.read.table("depot_e2e.`WsDevPgDogSrc`.history.`DogSource`")
This is a genuinely different table, not a flag on the current one — Depot deploys history as its own table, and the connector models it that way.
Time travel (Iceberg-backed datasets)
For a dataset backed by S3 Tables, the connector exposes Iceberg's snapshot history through standard Spark SQL, no extensions required:
SELECT * FROM depot_e2e.`WsS3TablesDefault`.`petstore.Pet` VERSION AS OF 8043...
SELECT * FROM depot_e2e.`WsS3TablesDefault`.`petstore.Pet` TIMESTAMP AS OF '2026-08-01 00:00:00'
Discover a snapshot id from the table's own metadata rather than hardcoding one:
SELECT * FROM depot_e2e.`WsS3TablesDefault`.`petstore.Pet`.snapshots ORDER BY committed_at DESC
VERSION AS OF / TIMESTAMP AS OF are not a substitute for the history table above: time travel is
a snapshot of the current table at a point in time; history is an append log with its own columns and
its own retention. Both are available, for different questions. Aurora- and Snowflake-backed datasets do
not support time travel today.
Joining across stores
Because every dataset is a table in the same catalog, joining data that lives in different stores is
ordinary SQL — no separate spark.read calls or DataFrame joins required:
SELECT a.*, b.*
FROM depot_e2e.`WsDevPgDogSrc`.`DogSource` a
JOIN depot_e2e.`WsDevDogSrc`.`petstore.Pet` b ON a.id = b.owner_id
Using a name in raw SQL: bind
val view = depotE2e.bind("dogs", "WsDevPgDogSrc", "DogSource")
spark.sql(s"SELECT * FROM $view WHERE test_id LIKE 'AUR2SFK-%'")
bind throws if the view name is already taken; use rebind for a deliberate replacement. A bound
view freezes its resolved envelope at bind time — it does not honour expiresAt on later queries —
so this is for interactive and SQL convenience, not long-running jobs. See
Caching and catalog instances for what that implies.
How each store is read
The connector presents one table interface, but the scan underneath it differs per store:
| store | scan path |
|---|---|
| Aurora | DSv2 ScanBuilder → V1Scan → JDBC relation |
| Snowflake | DSv2 ScanBuilder → V1Scan → SnowflakeRelation |
| S3 Tables | native DSv2 (Iceberg) — no V1Scan |
Both JDBC paths cross Spark's V1Scan bridge, which Spark documents as transitional and intends to
remove. Aurora's successor is decided — a native DataSource V2 scan, once Spark 4.2 is available to
us. Snowflake's is not: a bespoke DSv2 scan there would forfeit spark-snowflake's bulk
unload-to-stage path, so it needs assessing rather than rewriting. Both rows are current behaviour, not
settled architecture.
Three consequences follow, and the first is the one readers usually assume backwards:
- Nesting does not cost predicate pushdown. A predicate on a nested Depot property reaches the store
on all three paths, in every
nestingmode. - A scan must materially produce whatever shape it advertises. That is why the connector presents
Depot-side column names at the scan boundary on the JDBC paths rather than physical ones, and why
nestingis a projection above the scan rather than part of the table's schema. On Iceberg the obligation does not arise, so nesting there is free. - The
nestingmodes share one scan. They differ only in the projection above it, so switching mode costs no re-resolution and, within a single query, no second scan.
nesting is absent from the table above because it is a projection: its cost is the same on all three
paths. What differs per store is the scan path itself.
Choosing a DB user (Aurora)
By default, reads authenticate as the platform's own IAM-auth user. To use a different one — for per-workload segregation, or as a developer running ad hoc queries — set it on the mount, not on the read, since which user you connect as is fixed once the table is resolved:
val mine = spark.mountDepot(uri).withOptions(dbUser = "<name>").as("depot_e2e_mine")
A job that genuinely needs two identities mounts twice. As a developer, the literal sentinel
@use-my-git-email derives a personal reader role from your git config user.email — useful for ad
hoc queries from a laptop, and refused loudly (naming the derived role) if that role does not exist.
Never set this sentinel in a CI or production job definition; it is meant to be typed by a person at a
terminal, not baked into a config that runs unattended.
This has no effect on Snowflake-backed or S3-Tables-backed datasets: Snowflake authenticates with the credential tied to a chosen executor (see below), and S3 Tables authenticates as the job's own AWS identity.
Snowflake: choosing an executor
Snowflake credentials, and therefore cost-tag allocation, come from a named executor rather than a
plain username. Like dbUser, it is settled on the mount rather than per read — it selects the
credential the table is resolved against:
val tagged = spark.mountDepot(uri)
.withOptions(executorId = "snowflake-location-default")
.as("depot_e2e_tagged")
tagged.read("WsPantryNative", "pantry.Ingredient")
Omit it to use the location's default executor; a job that needs two executors mounts twice. Reading the Snowflake credential secret itself is your job's own responsibility once the Locator has pointed you at it — see Storage Locator → Client integration.
Notes
- Writing from Spark is not yet supported, but planned (DPT-3695). Until then, return data to Depot by writing Parquet or JSONL and using a standard transaction to merge the results back in.
- The connector's own AWS credentials are used to invoke Depot's Storage Locator and, where applicable, to mint short-lived database credentials or read a Snowflake credential secret — nothing is discovered or fetched by the job itself beyond that.