​

Auto Classification Workflow Configuration

​

Pipeline Configuration Parameters

​

Key Parameters Explained

​

enableAutoClassification

• Set this to true to enable automatic detection of sensitive columns (e.g., PII).
• Applies pattern recognition and tagging based on predefined criteria.

​

confidence

• Confidence level for tagging sensitive columns:
  • A higher confidence value (e.g., 90) reduces false positives but may miss some sensitive data.
  • A lower confidence value (e.g., 70) identifies more sensitive columns but may result in false positives.

​

storeSampleData

• Controls whether sample rows are stored during ingestion.
• If enabled, Collate stores the sampled rows of each table as sample data, up to a maximum of 50 rows. The number of rows sampled is set by sampleDataCount.

​

sampleDataCount

• Number of rows Collate samples from each table. Valid values range from 1 to 1000, and higher values are capped at 1000.
• Collate stores at most 50 of the sampled rows as table sample data, regardless of this value.

​

classificationFilterPattern

• Default (useFqnForFiltering: false) — match against the tag name only. For example, if your tag is POV.Key Data Asset, use key data asset:
  classificationFilterPattern:
    includes:
      - "key data asset"
  
• When useFqnForFiltering: true — match against the full tag FQN in the format Classification.TagName:
  classificationFilterPattern:
    includes:
      - "POV.Key Data Asset"
  

​

useFqnForFiltering

• When set to true, filtering patterns — including classificationFilterPattern — are matched against the Fully Qualified Name (for example, Classification.TagName).
• When set to false (default), filtering matches against raw names only (for example, the tag name without the classification prefix).

​

Auto Classification Workflow Execution

​

1. Install the Required Python Package

pip install "openmetadata-ingestion[pii-processor]"

​

2. Define and Execute the Python Workflow

​

Sample Auto Classification Workflow yaml

source:
  type: bigquery
  serviceName: local_bigquery
  serviceConnection:
    config:
      type: BigQuery
      credentials:
        gcpConfig:
          type: service_account
          projectId: my-project-id-1234
          privateKeyId: privateKeyID
          privateKey: "-----BEGIN PRIVATE KEY-----\nmySuperSecurePrivateKey==\n-----END PRIVATE KEY-----\n"
          clientEmail: client@email.secure
          clientId: "1234567890"
          authUri: https://accounts.google.com/o/oauth2/auth
          tokenUri: https://oauth2.googleapis.com/token
          authProviderX509CertUrl: https://www.googleapis.com/oauth2/v1/certs
          clientX509CertUrl: https://www.googleapis.com/oauth2/v1/certs
  sourceConfig:
    config:
      type: AutoClassification
      storeSampleData: true
      enableAutoClassification: true
      databaseFilterPattern:
        includes:
          - hello-world-1234
      schemaFilterPattern:
        includes:
          - super_schema
      tableFilterPattern:
        includes:
          - abc

processor:
   type: "orm-profiler"
   config:
    tableConfig:
      - fullyQualifiedName: local_bigquery.hello-world-1234.super_schema.abc
        profileSampleConfig:
          sampleConfigType: STATIC
          config:
            profileSample: 85
        partitionConfig:
          partitionQueryDuration: 180
        columnConfig:
          excludeColumns:
            - a
            - b

sink:
  type: metadata-rest
  config: {}
workflowConfig:
#  loggerLevel: INFO # DEBUG, INFO, WARN or ERROR
  openMetadataServerConfig:
    hostPort: http://localhost:8585/api
    authProvider: openmetadata
    securityConfig:
      jwtToken: "eyJraWQiOiJHYjM4OWEtOWY3Ni1nZGpzLWE5MmotMDI0MmJrOTQzNTYiLCJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImlzQm90IjpmYWxzZSwiaXNzIjoib3Blbi1tZXRhZGF0YS5vcmciLCJpYXQiOjE2NjM5Mzg0NjIsImVtYWlsIjoiYWRtaW5Ab3Blbm1ldGFkYXRhLm9yZyJ9.tS8um_5DKu7HgzGBzS1VTA5uUjKWOCU0B_j08WXBiEC0mr0zNREkqVfwFDD-d24HlNEbrqioLsBuFRiwIWKc1m_ZlVQbG7P36RUxhuv2vbSp80FKyNM-Tj93FDzq91jsyNmsQhyNv_fNr3TXfzzSPjHt8Go0FMMP66weoKMgW2PbXlhVKwEuXUHyakLLzewm9UMeQaEiRzhiTMU3UkLXcKbYEJJvfNFcLwSl9W8JCO_l0Yj3ud-qt_nQYEZwqW6u5nfdQllN133iikV4fM5QZsMCnm8Rq1mvLR0y9bmJiD7fwM1tmJ791TUWqmKaTnP49U493VanKpUAfzIiOiIbhg"

​

3. Expected Outcome

• Automatically classifies and tags sensitive data based on predefined patterns and confidence levels.
• Improves metadata enrichment and enhances data governance practices.
• Provides visibility into sensitive data across databases. This approach ensures that the Auto Classification Workflow is executed correctly using the appropriate OpenMetadata ingestion framework.

​

Auto Classification

​

1. Define the YAML Config

​

2. Run with the CLI

metadata classify -c <path-to-yaml>

​

Workflow Execution

​

To Execute the Auto Classification Workflow:

1. Create a Pipeline
   • Configure the Auto Classification JSON as demonstrated in the provided configuration example.
2. Run the Ingestion Pipeline
   • Use OpenMetadata or an external scheduler like Argo to trigger the pipeline execution.
3. Validate Results
   • Verify the metadata and tags applied to sensitive columns in the Collate UI.

​

Expected Outcomes

• Automatic Tagging: Columns containing sensitive information (e.g., names, emails, SSNs) are automatically tagged based on predefined confidence levels.
• Enhanced Visibility: Gain improved visibility and classification of sensitive data within your databases.
• Sample Data Integration: Store sample data to provide better insights during profiling and testing workflows.