Skip to main content

Gateways

Overview​

Depot Gateways enable connectivity to the internal Depot Connector API via AWS API Gateway, or other resources. There are a number of available gateway types:

  • API Gateway (a standard, Public REST API Gateway)
  • IAM Gateway
  • WebSocket Gateway
  • Lambda Gateway

API Gateway​

import { Gateway } from "@stage-tech/depot-cdk";

new Gateway.API(stack, "ApiGateway", {
    environment: depotEnvironment
});
Adding a WAF

Depot Gateways support adding WAFs (AWS Web Application Firewalls) to help secure them against common known vulnerabilities or bot attacks. You can also whitelist or blacklist certain IP addresses or ranges of IP addresses.

Here is an example of a WAF configuration for a Depot Gateway which sets adds two specific managed rule sets and blocks access to all IP addresses except the allowed list:

import { Gateway } from "@stage-tech/depot-cdk";

new Gateway.API(stack, "ApiGateway", {
    environment: depotEnvironment,
    firewall: {
        ipAddresses: {
            allowed: ["10.0.1.0/24"],
            blocked: []
        },
        managedRules: [
            StageDepotAwsManagedWafRule.AWSManagedRulesCommonRuleSet,
            StageDepotAwsManagedWafRule.AWSManagedRulesKnownBadInputsRuleSet
        ]
    }
});

Read more about the Depot WAF functionality here.

IAM Gateway​

The IAM Gateway exposes an endpoint which can be used in combination with a list of allowed IAM principals. See IAM authentication.

WebSocket Gateway​

The WebSocket gateway allows you to use WebSocket Subscriptions.

import { Gateway } from "@stage-tech/depot-cdk";

new Gateway.WebSocket(this, "WebSocketGateway", {
    environment: depotEnvironment
});

It is possible to create WebSocket subscriptions with GraphQL queries that will result in matching data being published back to the subscribing client.

The subscription request needs to define a GraphQL query which is used to locate the data to be returned. The workflow is:

  • connection_init
  • Subscribe to some query
  • When changes occur that match the query, you will be notified with a payload containing the data returned from the query.

Read more about using WebSocket Subscriptions.

Lambda Gateway​

The Lambda Gateway is an AWS Lambda function that provides an easy way of managing a Depot environment and data from your applications, without having to use the Depot GraphQL or CDK APIs.

Call it using the AWS SDK Invoke Lambda method from your C++, Go, Java, Javascript, .NET, NodeJS, PHP, Python or Ruby applications.

info

Use the Depot Lambda Client (depot-lambda-client) TypeScript library to interact with the Depot Lambda Gateway where preferable. The Depot Lambda Client page has a useful table that explains when to prefer using it over directly invoking the Depot Lambda Gateway.

  • Interact using the Depot Lambda Client where convenient
  • Use IAM permissions to manage access
  • Integrate with AWS Step Functions or Event Bridge
  • Supports both Depot API (e.g. create transaction or manage data store) and Data API (e.g. query a dataset, update a record) requests
  • Every Depot environment has a Lambda Gateway created by default. This Lambda function can be used to execute Data Operations against the environment, including Admin related queries, such as store management (create, delete, pause), and Transactions (create, list).

The Lambda Gateway function proxies requests (coming in as the data payload) across to the internal REST API that runs in the Depot Connector application. Every Lambda Gateway (environment) function is created with a name according to convention:

depot-{environment-id}-lambda-depot-gateway

You can leverage this naming convention to predictably know how to get it in your code. E.g. With aws-cdk and the lambda.Function.fromFunctionArn method of referencing existing Functions.

The Depot CLI invokes the Lambda Gateway function for some of it's commands such as create-transaction, list-transactions, as well as the get, create, patch, and delete data commands.

You can integrate your own custom integrations and automation by leveraging the Depot Lambda Gateway. For example, you might have a custom data loading routine or step function that could call the Lambda Gateway function with a specific payload to upsert data. You can also run any number of data querying commands too.

In your CDK code you can get the a handle to the Lambda function like this:

const lambdaFunc = lambda.Function.fromFunctionArn(this, "DepotApiLambdaGateway", yourEnvironmentRef.gatewayEndpoints.depotGatewayArn);

Executor Override​

By default, Lambda Gateway requests use the executor configured for the dataset or environment. You can optionally override this on a per-request basis by specifying an executor property in your Lambda invocation payload:

const params = {
    FunctionName: lambdaARN,
    InvocationType: "RequestResponse",
    Payload: {
        "schema": "my.contacts.Person",
        "operation": "get",
        "id": "person123",
        "dataset": {"id": "cf45"},
        "executor": {"id": "my-custom-executor"}  // Optional: override default executor
    }
};

For transaction operations, the executor is automatically injected into the transaction payload:

const params = {
    FunctionName: lambdaARN,
    InvocationType: "RequestResponse",
    Payload: {
        "operation": "transaction",
        "dataset": {"id": "cf45"},
        "executor": {"id": "long-running-executor-1"},  // Applied to the transaction
        "data": {
            "actions": [/* transaction actions */]
        }
    }
};

This allows you to route specific requests to different executors based on workload requirements, such as using a long-running executor for complex transactions or isolating high-priority operations.

Sending a Request to the Lambda Gateway​

When invoking the Lambda Gateway (or any Lambda function), the invoke method takes a params argument that includes a Payload property. This property should be populated with your request. You then use a lambda client to invoke the function with the specified payload.

const lambdaARN = `arn:aws:lambda:eu-west-1:123456789:function:depot-abc123-lambda-depot-gateway`;
const params = {
    FunctionName: lambdaARN,
    InvocationType: "RequestResponse",
    Payload:
      {
        "schema": "my.contacts.Person",
        "operation": "create",
        "data": {"name": "myName", "address": {"id": "61728"}},
        "dataset": {"id": "cf45"}
      }
};

let response;
try {
    response = JSON.parse((await this.lambda.invoke(params).promise()).Payload?.toString());
} catch (e) {
    console.log("Error: could not complete request\n" + e);
}

Request Properties​

schema

The fully qualified name of the schema

  • Type: string
  • Required: true

dataset

The target dataset for the operation

  • Type: Dataset
  • Required: true

operation

The API operation to perform. See Operations

  • Type: create | update | patch | delete | get | link | list | batch
  • Required: true

id

The identifier for the object

  • Operations: update, patch, delete, get, link
  • Type: string
  • Required: true

data

The body of the request

  • Operations: create, update, patch
  • Type: any
  • Required: true

expand

The properties to expand in the response

  • Operations: get, link, list
  • Type: List of string
  • Required: false

filter

The expression to filter paged results

  • Operations: link, list
  • Type: string
  • Required: false

sort

The sort order for the pages

  • Operations: link, list
  • Type: string
  • Required: false

The sort order is specified as a field name followed by an optional :asc or :desc to indicate the sort direction, followed by an optional :nulls-first or :nulls-last to indicate how null values should be sorted relative to non-null values. If no direction is specified, it defaults to ascending order. If no nulls order is specified, it defaults to nulls-last if the sort direction is ascending, and nulls-first if the sort direction is descending.

It is permissible but discouraged to omit the nulls- prefix in the null order specification.

Examples:

--sort "myField:asc"
--sort "myField:desc:nulls-first"
--sort "myField:asc:last"
note

Note: Depot handles sort order for ascending values: numbers first, then uppercase letters (A-Z), then lower case letters (a-z), and then lastly nulls. Reversed of course for descending order.

For example, sort: a:asc will sort values like this:

[{"a"="1", "b"="7"},
 {"a"="2", "b"="6"},
 {"a"="3", "b"="2"},
 {"a"="3", "b"=null},
 {"a"="3", "b"="3"},
 {"a"="3", "b"="5"},
 {"a"="3", "b"="4"},
 {"a"="3", "b"="1"},
 {"a"=null, "b"="1"},
 {"a"=null, "b"=null}]

limit

The page size

  • Operations: link, list
  • Type: integer
  • Required: false

nextToken

The token returned in the response to request the next page

  • Operations: link, list
  • Type: string
  • Required: false

link

The name of the link to follow

  • Operations: link
  • Type: string
  • Required: true

actions

The map of batch requests to perform

  • Operations: batch
  • Type: Map<string, Request>

Operations​

Create​

Create a new object

Example:

{
  "schema": "my.contacts.Person",
  "operation": "create",
  "data": {"name": "myName", "address": {"id": "61728"}},
  "dataset": {"id": "cf45"}
}

Create a new Transaction (Depot API).

Example:

{
  "schema": "Transaction",
  "operation": "create",
  "data": {
    "dataset": {
      "id": "cf45"
    },
    "actions": [
        {
            "locationUri": "s3://example-bucket/indexed-data",
            "operation": "UPSERT",
            "format": "PARQUET"
        }
    ],
    "context": {
        "example": "example value"
    }
  }
}

Update​

Update the entire object. Any properties not specified are set to null.

Example:

{
  "schema": "my.contacts.Person",
  "operation": "update",
  "id": "91726",
  "data": {"name": "myNewName", "address": {"id": "61728"}},
  "dataset": {"id": "cf45"}
}

Patch​

Update the specified properties. Any properties not specified are unchanged.

Example:

{
  "schema": "my.contacts.Person",
  "operation": "patch",
  "id": "91726",
  "data": {"address": {"id": "61728"}},
  "dataset": {"id": "cf45"}
}

Delete​

Delete the object.

Example:

{
  "schema": "my.contacts.Person",
  "operation": "delete",
  "id": "91726",
  "dataset": {"id": "cf45"}
}

Get​

Get the object.

Example:

{
  "schema": "my.contacts.Person",
  "operation": "get",
  "id": "91726",
  "expand": ["address"],
  "dataset": {"id": "cf45"}
}

Get a page of links for an object.

Example:

{
  "schema": "my.contacts.Person",
  "operation": "link",
  "id": "91726",
  "link": "addresses",
  "sort": "city",
  "limit": 50,
  "expand": ["city"],
  "dataset": {"id": "cf45"}
}

List​

Get a page of query results.

Example:

{
  "schema": "my.contacts.Person",
  "operation": "list",
  "filter": "address.postcode == 'P05TC0D3'",
  "sort": "name",
  "limit": 50,
  "expand": ["address", "address.city"],
  "dataset": {"id": "cf45"}
}
info

Note: list ordering is not guaranteed when Snowflake is the backing storage engine for the Dataset's Location.

Batch​

Perform a batch of create, update and delete operations

For example

{
  "operation": "batch",
  "actions": {
    "ren":  {
      "operation": "create",
      "schema": "pet.Pet",
      "data": {
        "name": "Ren",
        "age": 2
      }
    },
    "luna":  {
      "operation": "create",
      "schema": "pet.Pet",
      "data": {
        "name": "Luna",
        "age": 3
      }
    }
  },
  "dataset": {"id": "cf45"}
}

Returns

{
  "data": {
    "ren":  {
        "operation": "create",
        "schema": "pet.Pet",
        "data": {
          "name": "Ren",
          "age": 2,
          "version": 1
        }
    },
    "luna":  {
      "operation": "create",
      "schema": "pet.Pet",
      "data": {
        "name": "Luna",
        "age": 3,
        "version": 1
      }
    }
  },
  "status": {
    "code": 200
  }
}

Query​

Get results from a query schema, passing in custom arguments:

{
  "operation": "query",
  "limit": 1,
  "dataset": {
    "id": "1abdb6e1ba43"
  },
  "data": {
    "schema": "example.ingest.ValidateFooQuery",
    "arguments": {
      "itemIdArg": "12345"
    }
  }
}