Storage Locator

The Storage Locator resolves a Depot dataset to everything a consumer needs to connect to it directly: an endpoint, a driver, connection properties, a credential strategy, and — when asked — the physical name a schema is deployed under.

Its design invariant is that the Locator resolves metadata, never authentication: it never hands a consumer anything it could use to authenticate on its own say-so. Every credential kind it returns is a pointer to something the caller must already be authorised for — rds-db:connect for an IAM token, a deploy-time grant for S3 Tables' caller-identity, read access to a Secrets Manager secret for the pointer it hands back for Snowflake. Without the Locator, anything wanting to read a Depot dataset over its native protocol has to discover the endpoint and fetch the credential itself, which meant holding Secrets Manager, RDS-describe and EC2-describe rights across the whole environment. With it, a consumer holds lambda:InvokeFunction on one function, plus whichever single authentication permission its own read actually requires.

Spark is the first consumer, not the definition of one. Any compute that can invoke a lambda and open a JDBC connection is a valid client — a custom query engine, a notebook, a one-off migration job.

Where it sits

The Locator reaches __admin through the IAM gateway, signing each request with SigV4 under its own execution role. API Gateway derives the caller's identity from that signature, so the Locator asserts nothing about who it is. It is not placed in a VPC — reaching Connector was the only thing that ever required one.

It queries over GraphQL rather than REST because a selection set is a projection: one query returns exactly the fields needed rather than whole entities. REST has no equivalent (DPT-2152, DPT-1912).

Addressing it

The function is named depot-<environmentId>-storage-locator, so a consumer can derive it from the environment id it already has. In CDK, take it from the environment rather than building the string:

const locatorArn = depotEnvironment.gatewayEndpoints.storageLocatorArn;

The request

{
  operation: 'ResolveDataset',
  datasetId: string,        // id or alias
  environmentId: string,
  applicationName: string,  // caller label, log-only; sanitised and truncated to 40 chars
  readOnly?: boolean,       // defaults to true; see "Asking to write"
  partition?: string,       // from the depot: URL; echoed into connectionProperties
  schema?: string,          // also resolve this schema's deployed name
  versioned?: boolean,      // resolve the history table instead
  replicaTag?: { tag: string, value: string },
  executorId?: string,
  callerVpcId?: string,     // see Deployment scenarios
  needsBastionHint?: boolean,
  acceptsWif?: boolean,     // forward compatibility; ignored today
}

Most of this mirrors the depot: URL that Spark consumers already pass:

depot:<partition>:<accountId>:<region>:<environmentId>:<datasetId>[?replica=<tag>=<value>&executorId=<id>]

The URL's accountId and region are the caller's own AWS coordinates, used to sign its own connection. They are not part of the request: the Locator resolves where to connect, and never needs to know where the caller sits to do it.

replicaTag selects an Aurora reader instance by AWS resource tag — useful for steering heavy analytical reads onto a dedicated replica. If no instance carries the tag, resolution falls back to the cluster's reader endpoint; it will never quietly hand back the writer.

executorId names an executor whose warehouse should be used, for Snowflake. Omit it and the location's own executor supplies the warehouse; name an executor that declares none and the location's is used anyway. The credentials pointer always comes from the location's executor, whichever executor the request names — naming one chooses a warehouse, not an identity.

The response

{
  outcome: 'RESOLVED',
  datasetId: string,
  datasetAlias?: string,
  underlyingUrl: string,            // the best URL the Locator can name
  canonicalEndpoint: { host, port },
  reachability: 'DIRECT' | 'VPC_ENDPOINT' | 'REQUIRES_TUNNEL',
  bastionInstanceId?: string,
  credential:
    | { kind: 'SECRETS_MANAGER', secretArn: string }
    | { kind: 'RDS_IAM_TOKEN' }
    | { kind: 'CALLER_IDENTITY' },
  connectionProperties: Record<string, string>,
  driverClassName: string,
  locationType: string,
  expiresAt: string,                // ISO 8601
  schema?: { name, deployedName, propertyColumn, members },
}

Or, instead:

{ outcome: 'ERROR', error: { kind, message, retryable, retryAfterSeconds? } }

Switch on outcome, and then on credential.kind and error.kind — never on the presence of a field or on message text.

Nested column structure

An S3 Tables dataset stores an array, set, map, non-flattened struct or any as one Iceberg column, and the keys nested inside follow the table's naming strategy: the Depot name barkColour may be stored as bark_colour. schema.members.<name>.structure carries the stored names, so a client can map them back.

type ResolvedStructure =
  | { kind: 'struct'; fields: Record<string, { name: string; structure: ResolvedStructure }> }
  | { kind: 'array'; element: ResolvedStructure } // arrays and sets
  | { kind: 'map'; value: ResolvedStructure } // keys are data, never renamed
  | { kind: 'scalar' }
  | { kind: 'any'; storage: 'json-text' | 'variant' };

fields is keyed by Depot name and each name is the key as stored. A ref inside a structure appears as a struct with id and version fields. The same shape appears on members nested in a flattened group, and on versioned: true resolutions, where history columns are identical to the canonical ones.

Absent is not scalar. Scalars, flattened groups, top-level refs, the __deleted tombstone and every member of an Aurora or native-Snowflake dataset carry no structure.

Reachability

reachability says what the caller must do to reach canonicalEndpoint, given where it said it sits (callerVpcId). It is an instruction, not a description of the database.

Value	What it means	What the caller does
DIRECT	The endpoint is reachable as named.	Connect to canonicalEndpoint.
VPC_ENDPOINT	A VPC endpoint fronts the database in the caller's VPC.	Connect to underlyingUrl, which carries the endpoint's alias. Sign against canonicalEndpoint regardless — an RDS IAM token is only valid for the canonical host.
REQUIRES_TUNNEL	Nothing in the caller's network reaches the database.	Port-forward through the bastion named in bastionInstanceId (SSM Session Manager), connect to the local end, and still sign against canonicalEndpoint.

Snowflake is always DIRECT: it is reached over the public internet, so neither VPC endpoints nor bastions apply.

Credential kinds

RDS_IAM_TOKEN carries no secret at all. The token is a locally-computed signature, valid about fifteen minutes, and must be minted by whichever process opens the connection — for Spark that includes executors, which a token minted centrally would never reach before expiring. Minting it needs no AWS API permission, only rds-db:connect.

SECRETS_MANAGER hands over a pointer, today for Snowflake — secretArn, and nothing derived from the secret's contents. The Locator never places secret material in the response: reading the secret, and deciding which driver option it belongs in, is the caller's own job, once it has confirmed it is authorised to read that specific secret. That includes the connect-as username, which lives inside the secret. See Client integration for how to do that.

CALLER_IDENTITY hands over nothing at all, and is what S3 Tables datasets return. There is no secret to mint and none to relay: the caller authenticates as whatever identity it already runs under, and whether that identity may read — or write — was settled at deploy time by the dataset's declaration.

The Locator deliberately does not name a role for the caller to assume. It cannot: a directly-invoked lambda sees nothing of its caller, so with several per-(service, environment) identities in play it has no trustworthy way to pick one. applicationName is self-asserted and must never be used for an authorisation decision. A caller that wants to assume a role already knows which one.

Later work adds a kind for Snowflake federated credentials (DPT-3690). Because the kind is a discriminated union, it arrives without disturbing these three.

The Snowflake warehouse is always named

A resolved Snowflake dataset always carries sfWarehouse in connectionProperties. If neither the requested executor nor the location's executor declares a warehouse, the Locator refuses the resolution with NOT_FOUND rather than returning one without.

This diverges from Connector deliberately. Connector connects anyway and lets the Snowflake user's DEFAULT_WAREHOUSE apply. The Locator will not, in order to enforce workload attribution.

Asking to write

readOnly defaults to true. Requesting false requires naming a schema, and that schema must be a record schema (RecordSchema). Anything else — object above all, plus view, query, struct, stage, format, sequence — is refused with INVALID_REQUEST.

That is deliberately narrow for now: metadata-bearing schemas carry invariants the Locator cannot reason about, so it declines rather than hand out a connection it cannot vouch for. Widening it is a decision to be taken explicitly.

readOnly appearing in connectionProperties is therefore not a relayed preference but the Locator's recorded assent — it checked the schema kind, and permits writing.

What the Locator does not set

ApplicationName, Snowflake query tags and query_tag are all set by whoever opens the connection. The Locator never connects to anything, so echoing a label back to the caller that supplied it would inform nobody. Consumers assembling connection options should set these themselves.

applicationName on the request is a different thing: a required, self-asserted label used only in the Locator's own log line. It is required because a directly-invoked lambda cannot see who invoked it, so without it a log line names the dataset and nothing about who asked — and an unattributable resolution is no use in an incident. A request without one, or with one that is blank once sanitised, is refused with INVALID_REQUEST. Being unverified, it must never inform an authorisation decision.

The Locator does not say who you are

It resolves where to connect. Which DB user to connect as, and the AWS coordinates needed to sign as them, are the caller's: it knows its own identity, and it parsed the coordinates out of its own depot: URL. The envelope carries neither.

expiresAt is a contract, not a decoration

Resolve once, cache until expiresAt, then resolve again. It exists so that resolution load does not scale with worker count, and so that any future minted credential has a stated lifetime. See Client integration — this is the single most important thing to get right.

S3 Tables datasets: the Iceberg target

For a dataset on an S3 Tables location (locationType: 's3tables'), target.kind is ICEBERG_CATALOG, not JDBC. There is no host, port or driver: the target carries catalog configuration for an Iceberg engine such as Spark, PyIceberg or DuckDB. The credential is always CALLER_IDENTITY: the engine signs as its own AWS identity, which the dataset must grant, with directReadPrincipals / directWritePrincipals or a DatasetPrincipalPermission; see Iceberg interop case studies.

target: {
  kind: 'ICEBERG_CATALOG',
  credential: { kind: 'CALLER_IDENTITY' },
  table: { namespace: string, name: string, historyName?: string },
  routes: { kind, warehouse, properties }[],   // most-preferred first
  readOnly: boolean,
  tableBucketArn: string,
  icebergFormatVersion: number,                // always 2
  deleteMode: string,                          // MERGE_ON_READ or COPY_ON_WRITE
}

Take the first route whose kind the engine supports, matching on kind, never on position:

kind	Reached through	Authorised by	Writes	warehouse
S3_TABLES_CATALOG	the s3-tables-catalog-for-iceberg-runtime library, JVM only	s3tables:* IAM and the table-bucket policy	yes	the table bucket's ARN
S3_TABLES_REST	the S3 Tables Iceberg REST endpoint	s3tables:* IAM and the table-bucket policy	yes	the table bucket's ARN
GLUE_REST	the Glue Iceberg REST endpoint, over the s3tablescatalog catalog	Lake Formation grants, plus lakeformation:GetDataAccess	no	<account>:s3tablescatalog/<bucket>

properties are catalog properties, ready to use as they are: prefix them with spark.sql.catalog.<name>. for Spark, or pass them to PyIceberg's load_catalog(**props). The REST routes carry the signing region in rest.signing-region. DuckDB takes only the warehouse; see DuckDB ↔ Snowflake.

readOnly is false only when the request asked to write and named a record schema. A writer honours deleteMode; a reader must read the delete files that MERGE_ON_READ implies. Iceberg columns carry physical names, and schema.propertyColumn maps each Depot property to its column.

What it does not do

• It does not authorise per dataset. Any principal permitted to invoke the Locator can resolve any dataset in the environment, including having its Snowflake credential secret pointed at. This is still narrower than the situation it replaces, where consumers held environment-wide Secrets Manager access — but it is a real limitation, recorded against DPT-2934, and it is why invoke rights should be granted narrowly.
• It does not name a role to assume. For S3 Tables it returns CALLER_IDENTITY and expects the caller to authenticate as itself; see "Credential kinds" for why it cannot choose an identity for you.
• It does not open tunnels. It can say a tunnel is needed and name the bastion; establishing one is the caller's business, because the local port is the caller's to choose.
• It does not write to metadata-bearing schemas. Writable connections are limited to record schemas — see "Asking to write" above.