Skip to main content

Subscriptions: under the hood

caution

Except for published interfaces (the client-supplied topics), the implementation details of subscriptions are subject to change without notice.

Under the hood, subscriptions rely on the following infrastructure:

Subscriptions​

The granularity of subscriptions is per schema name + operation type + field conditions (evaluated Basestar expression)

  • With two exceptions, all actions sent to the Connector via the GraphQL, REST, Lambda are delivered to subscribers in near real time after completion. They are sent directly to the topic(s) designated in the subscription(s) that match the schema name and/or fields of the object being changed.

The two exceptions:

  1. When a Dataset is deployed on a Snowflake Location with enableEvents=true, then real-time event generation is suppressed. Instead, a hourly job polls the Snowflake Data Change Capture mechanism to retrieve events and deliver them to the SNS Topic.
  2. Actions made on the Admin dataset are always suppressed

Notifications​

Notifications relate to the Transaction API.

The user-visible granularity of Notifications is per Dataset.

The (internal) Transaction and Poll handler lambdas emit events related to the lifetime of each transaction: Started, Completed (SUCCESS), Completed (FAILURE). These events are always sent in real time to the internal Broadcast topic, then if the notifications field is set on the Dataset, a per-dataset lambda subscribed to the Broadcast topic picks these events up and delivers them to the User-supplied SNS Topic.

If Change Capture is deployed at the Location level (SnowflakeLocation.enableEvents = true): In addition to Transaction-level events (still emitted in real time), a background process called hourly will poll the storage layer for pending change capture events, and will deliver row-level Created/Updated/Deleted to the internal Broadcast Topic. These events can be picked up and relayed to the User-supplied SNS Topic by the per-dataset lambda if notifications have been enabled by the user.

caution

Enabling Change Capture at the Location level suppresses subscriptions, so no real-time row-level change events will be emitted anymore.

This is made to avoid sending events twice (once in real-time when the unitary API is used, and another time when it is identified via Change Capture)