Custom Tests

Collate’s built-in tests cover most use cases, but sometimes you need something more specific. You can bring in results from your own test suite or define entirely new tests that run inside Collate — all through the API and the Python SDK. The setup involves five steps:

Create a Test Definition
Create a Test Suite
Create a Test Case
Write Test Case Results
(Optional) Make the test available in the Collate UI

Step 1: Create a Test Definition

A Test Definition tells Collate what your test is: its name, what entity type it applies to (table or column), and which platforms run it. Send a POST request to /api/v1/dataQuality/testDefinition with at least the following fields:

{
    "description": "<your test definition description>",
    "entityType": "<TABLE or COLUMN>",
    "name": "<your_test_name>",
    "testPlatforms": ["<any of Collate,GreatExpectations, dbt, Deequ, Soda, Other>"],
    "parameterDefinition": [
      {
        "name": "<name>"
      },
      {
        "name": "<name>"
      }
    ]
}

Here is a complete cURL example:

curl --request POST 'http://localhost:8585/api/v1/dataQuality/testDefinitions' \
--header 'Content-Type: application/json' \
--data-raw '{
    "description": "A demo custom test",
    "entityType": "TABLE",
    "name": "demo_test_definition",
    "testPlatforms": ["Soda", "dbt"],
    "parameterDefinition": [{
        "name": "ColumnOne"
    }]
}'

Save the universally unique identifier (UUID) from the response — you’ll need it to create the Test Case. Heads up: To make this test definition available in the Collate UI, include Collate in testPlatforms. That requires extra setup covered in Step 5. If you only want to build a UI-executable test, go to Step 5.

Step 2: Create a Test Suite

A Test Suite groups Test Cases together for scheduling and ownership. You can create a new one or reuse an existing suite. Send a POST request to /api/v1/dataQuality/testSuites/executable:

{
  "name": "<test_suite_name>",
  "description": "<test suite description>",
  "executableEntityReference": "<entityFQN>"
}

Here is a complete cURL example:

curl --request POST 'http://localhost:8585/api/v1/dataQuality/testSuites/executable' \
--header 'Content-Type: application/json' \
--data-raw '{
  "name": "<test_suite_name>",
  "description": "<test suite description>",
  "executableEntityReference": "<entityFQN>"
}'

Save the UUID from the response — you’ll need it in the next step.

Step 3: Create a Test Case

A Test Case is a specific instance of your Test Definition applied to a table or column. It uses the fully qualified name (FQN) to reference the test definition and suite. Send a POST request to /api/v1/dataQuality/testCases:

{
    "entityLink": "<#E::table::fqn> or <#E::table::fqn::columns::column name>",
    "name": "<test_case_name>",
    "testDefinition": {
        "FQN": "<test definition FQN>",
        "type": "testDefinition"
    },
    "testSuite": {
        "FQN": "<test suite FQN>",
        "type": "testSuite"
    }
}

Important: Include the opening and closing <> in the entityLink value. Here is a complete cURL example:

curl --request POST 'http://localhost:8585/api/v1/dataQuality/testCases' \
--header 'Content-Type: application/json' \
--data-raw '{
    "entityLink": "<#E::table::local_redshift.dev.dbt_jaffle.customers>",
    "name": "custom_test_Case",
    "testDefinition": {
        "id": "1f3ce6f5-67be-45db-8314-2ee42d73239f",
        "type": "testDefinition"
    },
    "testSuite": {
        "id": "3192ed9b-5907-475d-a623-1b3a1ef4a2f6",
        "type": "testSuite"
    },
    "parameterValues": [
        {
            "name": "colName",
            "value": 10
        }
    ]
}'

Save the UUID from the response

Step 4: Write Test Case Results

Note: This step is optional if you’re running the test through the Collate UI.

After your Test Case exists, push results to it with a PUT request to /api/v1/dataQuality/testCases/{test FQN}/testCaseResult:

{
    "result": "<result message>",
    "testCaseStatus": "<Success or Failed or Aborted>",
    "timestamp": <Unix timestamp in milliseconds>,
    "testResultValue": [
      {
        "value": "<value>"
      }
    ]
}

Here is a complete cURL example:

curl --location --request PUT 'http://localhost:8585/api/v1/dataQuality/testCases/local_redshift.dev.dbt_jaffle.customers.custom_test_Case/testCaseResult' \
--header 'Content-Type: application/json' \
--data-raw '{
    "result": "found 1 value, expected n",
    "testCaseStatus": "Success",
    "timestamp": 1662129151000,
    "testResultValue": [{
        "value": "10"
    }]
}'

Your test now appears in the Test Suite and on the table entity page.

Step 5: (Optional) Make Your Test Available in the Collate UI

To make your custom test available in the Collate UI — so anyone can run it without touching the API — wire it up using the Collate data_quality namespace submodule.

Create your namespace package Start by creating a Python package that holds your test logic. At minimum, your package needs this structure:
metadata/ setup.py
Place your test file in the right location based on the entity type:
- Table tests: metadata/data_quality/validations/table/sqlalchemy/<yourTest>.py
- Column tests: metadata/data_quality/validations/column/sqlalchemy/<yourTest>.py
The filename (<yourTest>) must match the test name you used in Step 1. Important: Add an __init__.py file to every folder with this line:
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
Create your test class In your <yourTest>.py file, create a class named <YourTest>Validator that inherits from BaseTestValidator. Optionally inherit from SQAValidatorMixin too — it gives you extra helper methods out of the box. Implement the run_validation method, which must return a TestCaseResult object. See a full working example in the OpenMetadata validator source tree — it implements an entropy test end to end.
class ColumnEntropyToBeBetweenValidator(BaseTestValidator): """Implements custom test validator for Collate.""" def run_validation(self) -> TestCaseResult: """Run test validation"""
Install your package Once your package is ready, install it in the same environment where the Collate Python SDK is installed:
pip install .

​Custom Tests

​Step 1: Create a Test Definition

​Step 2: Create a Test Suite

​Step 3: Create a Test Case

​Step 4: Write Test Case Results

​Step 5: (Optional) Make Your Test Available in the Collate UI

Custom Tests

Step 1: Create a Test Definition

Step 2: Create a Test Suite

Step 3: Create a Test Case

Step 4: Write Test Case Results

Step 5: (Optional) Make Your Test Available in the Collate UI