Skip to main content

Packages and Schemas

Overview​

A Package defines the structure of a dataset through one or more schemas. A Package can be re-used across datasets if they require the same structure.

To find out how to define Schemas in YAML, or read more about the features available when developing Depot projects with Schemas take a look at the Schema documentation here.

There are multiple ways to package up your schemas and consume them in a Depot Package. How you do this is ultimately up to you and your preferred develop, build, and package process. To help you get started here some options available:

Package with local schemas​

You can load schemas from YAML files directly in your CDK project. Use the @stage-tech/depot-package-common library to help.

import { Schemas } from "@stage-tech/depot-package-common";
const mergedSchemas = Schemas.fromYamlFiles(["./schemas/my.contacts", "./schemas/my.addresses"]);
const mergedSchemasPackage = new Package(this, "MergedSchemaPackage", {
    name: "my.package",
    version: "1.0.0",
    schemas: mergedSchemas
});

Advanced Package schema loading​

There are more options available for Package projects and creation of your Depot Schemas.

Providing additional schemas​

By using additionalSchemas property in Packages.depotPackage you can provide schemas defined using TypeScript to be included in the Depot package.

const additionalSchemas = {
            "human.Human": {
                type: "view",
                properties: {
                    name: { type: "string" },
                    age: { type: "number" }
                }
            }
        };

const myPackage = Packages.depotPackage({
    ...
    schemaDir: [path.join(__dirname, "src")],
    // the additional schema of `human.Human` will be added to schemas namespace while creating a package.
    additionalSchemas
});

UDF schemas​

You can create UDFs (user defined functions) for Snowflake using TypeScript with complex dependency imports, and have them automatically bundled and included in your exported and packaged npm modules. See the relevant Development documentation for examples and requirements.

UDTF schemas​

Depot supports defining a table function schema that is deployed to Snowflake. The code for the table function can be written as an embedded SQL statement or using external SQL files (see SQL schema definitions). You can read more about UDTF support in the Development documentation.

SQL definition schemas​

SQL definitions can be inline within your View schemas or added by name convention if they exist in the above mentioned schemaDir search paths (in your Package project). The requirement for being picked up from individual, separated .sql files is that they are named exactly the same as the schema definition files (.yml) that will be using them. You can learn more about how to use SQL definitions in your Depot schemas and Packages in the Development documentation area.

Dependencies between Packages​

As mentioned previously, Packages contain schemas for use in Datasets. Schemas may refer to schemas in other packages - for example when data needs to be read from one Dataset to another. When referring to schemas in another package, those schemas must be referenced using a fully qualified name (i.e. myPackage.schemaName).

In addition to this, the dependencies between packages (and therefore Datasets) need to be declared in both the project CDK repository and in this package repository.

Project CDK repository​

When a Dataset is defined using a Package that has these sorts of cross-package dependencies, the Dataset's construct in the project CDK repository should include a dependencies property, which contain a list of the other Datasets upon which it is dependent.

Package repository​

The names of other Packages that this Package is dependent on should be declared in the imports list in the Package's main exported index.js file. You should also have those Package modules added to your package.json file dependencies (i.e. with npm install or yarn add).

Example:

index.js

module.exports = pkg.Packages.depotPackage({
    ...
    imports: [
        { pkg: require("@your-org/example-contact-package"), as: "ecp" }
    ]
});

package.json

"dependencies": {
  "@your-org/example-contact-package": "1.0.1"
}