Skip to main content

Overview

The Snowflake connector provides a convenient way of importing data directly from Snowflake into the Decentriq platform.

Prerequisites

  • Have a Snowflake account

  • Have downloaded the snowsql CLI tool

  • Know the Snowflake account name

    • This takes the form <org_name>-<account_name> and can be found after logging in to your Snowflake account. Using the below screenshot as an example, the account name would be kbyudeq-su04811

Prepare data for export from Snowflake

Connect to Snowflake using the snowsql cli tool.

snowsql --accountname <ACCOUNT_NAME> --username <USER_NAME>

After successful login, execute the following commands:

  1. Select the database to connect to.

    USE DATABASE <DATABASE_NAME>;

    e.g. 

    USE DATABASE my_db;

  2. Create a file format compatible with Decentriq datasets.

    CREATE OR REPLACE FILE FORMAT <FILE_FORMAT_NAME>
      TYPE = 'CSV'
      FIELD_DELIMITER = ','
      FIELD_OPTIONALLY_ENCLOSED_BY='0X22'
      COMPRESSION = NONE;

    e.g. 

    CREATE OR REPLACE FILE FORMAT dq_file_format
      TYPE = 'CSV'
      FIELD_DELIMITER = ','
      FIELD_OPTIONALLY_ENCLOSED_BY='0X22'
      COMPRESSION = NONE;

  3. Create a Snowflake internal stage. Snowflake server-side encryption (SNOWFLAKE_SSE) must be enabled.

    CREATE OR REPLACE STAGE <NAME_OF_STAGE> ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');

    e.g.

    CREATE OR REPLACE STAGE my_stage ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');

  4. Unload the data from your database table into the previously created internal stage.

    COPY INTO @<STAGE_NAME> from <DB_NAME>.<DB_SCHEMA>.<TABLE_NAME> FILE_FORMAT = <FILE_FORMAT_NAME>;

    e.g.

    COPY INTO @my_stage from my_db.public.my_table FILE_FORMAT = dq_file_format;

Upon successful completion of the above steps, one or more csv files should be present in your internal stage. This can be verified by logging into your Snowflake account and clicking through on your database > schema > stages.

Your data is now ready to be imported into the Decentriq platform.

Importing data

  1. Follow the steps to select a dataset for import and choose Snowflake from the list of connectors.

  2. Input the requisite information:

    • Warehouse name: Name of the virtual warehouse in Snowflake to import data from.
    • Database name: Name of the database in Snowflake to import data from.
    • Schema name: Name of the schema in Snowflake
    • Table name: Name of the table in Snowflake to import data from.
    • Stage name: Name of the stage in Snowflake to import data from.
    • Stored dataset name: Name of the dataset when stored in the Decentriq platform.
    • Credentials:
      • Account ID: Unique Snowflake account ID.
      • Role: Role of the user in Snowflake.
        • This can be set to PUBLIC
      • Username: Snowflake username.
      • Password: Snowflake password.

  3. After clicking the Import button, navigate to the Imports/Exports tab on the Datasets page to track the status of the import.

  4. Once complete, the dataset will be visible in the Datasets page under the Datasets tab with the stored dataset name provided in step 2.