Skip to main content

Environment Config Pipeline

Overview​

When you deploy depot-cdk constructs (Locations, Datasets, Executors), each construct fires a CloudFormation Custom Resource that stores your configuration in the Depot backend. A subsequent CodeBuild run then transforms all stored configs into a single content-addressed JSON blob uploaded to S3. Every runtime resource — Lambda functions and API Gateway — receives the S3 URI of that blob and fetches it at runtime to obtain the live environment configuration.

This page documents that pipeline end-to-end so you understand where each field comes from and how it reaches the Java runtime.

Step 1 — depot-cdk Custom Resources store ILocationConfig​

Each Location.* construct in @stage-tech/depot-cdk synthesises a CloudFormation Custom Resource of type Custom::Location, handled by the sdp-bootstrap-resource Lambda. Its properties match the ILocationConfig interface:

{
  "LocationType": "s3tables",
  "LocationName": "my-data-lake",
  "EnvironmentId": "prod-abc123",
  "Enabled": true,
  "LocationProps": {
    "TableBucketArn": "arn:aws:s3tables:eu-west-1:123456789012:bucket/my-bucket"
  }
}

LocationType is the discriminator used by every subsequent step. LocationProps is a free-form object whose fields vary by type — only the properties defined in a given location type's implementation are meaningful.

Step 2 — sync-env.ts: transform and upload​

Once the Custom Resources have stored all configs, CodeBuild runs sync-env.ts, which calls transformToEnvConfig() → transformLocations().

locations-transformer.ts​

transformLocations() iterates over every stored ILocationConfig and calls parseLocationProperties(), which dispatches to a type-specific parsing function:

  • aurora → parseAuroraLocationProperties() (complex: cluster config, bastion users, tags)
  • s3tables → parseS3TablesLocationProperties() (extracts enabled, tableBucketArn)
  • everything else → generic EnvironmentPropertiesDeclaration (Snowflake, DynamoDB, OpenSearch, etc.)

Each parsing function returns a typed *LocationProps object (e.g. S3TablesLocationProps). The result is placed in the properties field of the output location entry.

important

locations-transformer.ts is in Java's read path. Any LocationProps field that is not explicitly extracted here will be absent from the JSON that Java deserialises. Adding a new field to LocationProps requires a matching line in the relevant parse*LocationProperties() function.

Output shape (EnvironmentDeclaration)​

{
  "environmentId": "prod-abc123",
  "locations": [
    {
      "id": "a1b2c3d4e5f6",
      "name": "my-data-lake",
      "type": "s3tables",
      "properties": {
        "enabled": true,
        "tableBucketArn": "arn:aws:s3tables:eu-west-1:123456789012:bucket/my-bucket"
      }
    }
  ],
  "datasets": [ ... ],
  "executors": [ ... ]
}

Content-addressed S3 upload​

sync-env.ts calls putJson() (in bin/sync/s3-client.ts), which:

  1. Stable-stringifies the JSON (fast-json-stable-stringify)
  2. Computes a SHA-256 hash of the result
  3. Uploads to s3://<bucket>/<env-name>/environment/<sha256>.json

The hash is the filename, so the URI changes whenever the content changes, giving automatic cache-busting at the cost of never deleting old blobs. Three separate blobs are produced:

VariableS3 path
environmentUri<env-name>/environment/<sha256>.json
baseEnvironmentUri<env-name>/base-environment/<sha256>.json
datasetsUri<env-name>/datasets/<sha256>.json

Step 3 — URI distribution to runtime resources​

sync-env.ts passes the three URIs into the CDK Environment stack. The stack wires them to runtime resources via three distinct mechanisms:

Lambda environment variable​

All transaction Lambdas (Snowflake, Aurora, Iceberg), the gateway Lambda, the DynamoDB stream Lambda, and the EMR Serverless Lambda receive:

ENVIRONMENT_URI = s3://<bucket>/<env-name>/environment/<sha256>.json

Storage Custom Resource property​

The Aurora and Snowflake storage provisioning Custom Resources (which run Java to create databases, schemas, users, etc.) receive baseEnvironmentUri as a CloudFormation property — not an env var. See aurora-storage.ts and snowflake-storage.ts:

BaseEnvironmentUri = s3://<bucket>/<env-name>/base-environment/<sha256>.json

This is a separate blob from environmentUri and is used during the provisioning phase rather than at request-handling time.

REST API Gateway header injection​

The REST and IAM gateways inject URIs as static headers on every inbound request (see gateways.ts, rest-gateway.ts):

X-Environment-Uri: s3://<bucket>/<env-name>/environment/<sha256>.json
X-Datasets-Uri:   s3://<bucket>/<env-name>/datasets/<sha256>.json

These headers are injected at the APIG integration level — the upstream service never needs to know the URI in advance.

Step 4 — Java runtime consumption​

The Java runtime obtains the URI through one of three paths:

  • Lambda path: System.getenv("ENVIRONMENT_URI")
  • Storage provisioning path: CloudFormation Custom Resource property BaseEnvironmentUri (Aurora, Snowflake)
  • APIG path: request.getHeader("X-Environment-Uri")

In both cases it fetches the JSON from S3 and deserialises it into an EnvironmentDeclaration. Location entries are polymorphically deserialized via Jackson @JsonSubTypes, keyed on the type field:

@JsonSubTypes({
  @JsonSubTypes.Type(value = S3TablesLocation.class, name = "s3tables"),
  @JsonSubTypes.Type(value = AuroraLocation.class,   name = "aurora"),
  // ...
})
public abstract class Location { ... }

The properties object maps to the corresponding *Properties inner class (e.g. S3TablesLocation.S3TablesProperties { tableBucketArn }).

Adding a new location type — checklist​

When implementing a new location type, touch the following files in order:

  1. packages/depot-cdk/src/stage-depot-location.ts

    • Add FOOTYPE = 'footype' to the internal LocationType enum
    • Add FooTypeLocationProps extends LocationProps interface
    • Add class FooType extends ConcreteLocation (model on Iceberg / S3Tables)
    • Add public static FooType = FooType to the Location facade

  2. packages/depot-platform-cdk/lib/api/location-declaration.ts

    • Add FOOTYPE = 'footype' to the exported LocationType enum
    • Add FooTypeLocationProps extends EnvironmentPropertiesDeclaration interface

  3. packages/depot-platform-cdk/bin/sync/locations-transformer.ts

    • Add parseFooTypeLocationProperties(config): FooTypeLocationProps
    • Add else if (config.LocationType === 'footype') branch in parseLocationProperties()
    • Import FooTypeLocationProps at the top

  4. depot-schema (Java, stage-depot-platform repo)

    • Register @JsonSubTypes.Type(value = FooTypeLocation.class, name = "footype") on Location
    • Create FooTypeLocation class with FooTypeProperties inner class

  5. Tests

    • Create test/resources/sdp-footype/locations/<id>.json fixture
    • Create test/sync/locations/footype.test.ts asserting the transformed properties