​

Custom recognizers

​

What are recognizers?

• Customizable detection: Define your own patterns for organization-specific data (employee IDs, internal codes, custom formats)
• Multiple detection methods: Use regex patterns, exact terms, or 45+ pre-built detectors
• Learning from feedback: Users can report false positives, which automatically refine recognizer behavior
• Confidence-based tagging: Set minimum confidence thresholds to control precision

​

Recognizer types

​

Pattern recognizers

• Emails, phone numbers, IP addresses
• Custom organizational patterns (employee IDs: EMP-\d{5})
• Any data following predictable patterns

Pattern: \b\d{3}[-.]?\d{3}[-.]?\d{4}\b
Detects: US phone numbers (123-456-7890, 123.456.7890)

​

Exact Terms recognizers

• Known sensitive values (internal project codes, department names)
• Fixed vocabularies (country codes, status values)
• Cases requiring exact matches (no pattern variation)

Terms: ["PROJECT_ALPHA", "PROJECT_BETA", "PROJECT_GAMMA"]
Detects: Exact matches of confidential project codes

​

Predefined recognizers

• Standard PII (credit cards, SSNs, passports)
• International identifiers (IBANs, UK NHS numbers, ES NIF)
• When you don’t want to write custom regex

• Financial: CreditCardRecognizer, IbanRecognizer, UsBankRecognizer
• Personal ID: UsSsnRecognizer, UsPassportRecognizer, InPanRecognizer, InAadhaarRecognizer
• Healthcare: NhsRecognizer, MedicalLicenseRecognizer
• Contact: EmailRecognizer, PhoneRecognizer, UrlRecognizer

​

Creating a Recognizer

1. Navigate to Govern > Classification.
2. Select a classification (e.g., “PII”).
3. Click on a tag within that classification.
4. Go to the Recognizers tab.
5. Click Add Recognizer.
6. Configure the recognizer:
   • Name: Unique identifier (e.g., email_pattern)
   • Display Name: A unique descriptive name (e.g., “Email Pattern Detector”)
   • Description: What this recognizer detects
   • Supported Entity Target: Choose where to analyze:
     • Content: Analyze actual data values in sampled rows. Tags are applied when the confidence score meets the threshold.
     • Column Name: Apply the tag based on the column name alone — no sample data is needed. Use this when sampled values are masked, anonymized, or unavailable.
   • Recognizer Type: Select the detection method:
     • Pattern: Match using regular expressions. To set up Pattern recognizer, see Pattern Recognizer Settings.
     • Exact Term: Match specific values from a list. To set up Exact Term recognizer, see Exact Terms Settings.
     • Pre-Defined: Use a built-in Presidio recognizer. To set up Pre-Defined recognizer, see Predefined Recognizer Settings.
   • Confidence Threshold: Minimum score (0.0-1.0) to apply tag (default: 0.6)
   
   Auto-classification samples rows directly from the source during each run — it does not rely on previously stored sample data. The Store Sample Data setting controls whether those rows are saved in OpenMetadata, not whether classification can run. Tags won’t be applied if the connector cannot reach the source, sampled values are masked or anonymized, or the confidence score falls below the threshold. If classification is unreliable for a column, use Governance Automations to tag by name pattern instead.
7. Click Save

​

Pattern Recognizer Settings

• Add Pattern: Add one or more regex patterns, each with:
  • Name: A unique descriptive name, helps identify the purpose of each pattern when you have multiple patterns in one recognizer. For example, Visa Format, MasterCard Format, AmEx Format.
  • Regex: Regular expression pattern to match against the content or column name. Use standard regex syntax.
  • Score: Confidence score assigned when this pattern matches. Values range from 0.0 to 1.0. Use higher scores for more specific/reliable patterns and lower scores for broader patterns.
• Supported Language: Language scope for this recognizer. Defaults to All.
• Context (optional): List of context words that can help boost confidence score. These words provide additional context for pattern matching and can increase confidence when found near the matched pattern. For example, for a credit card recognizer, context might include: “card”, “credit”, “payment”, “cvv”.
• Regex Flags: Toggle Dot All, Multiline, and Ignore Case as needed.
  • Dot All: Enable dot-all regex flag. When enabled, the dot (.) metacharacter matches newline characters as well as any other character. Useful for multi-line pattern matching.
  • Multiline: Enable multiline regex flag. When enabled, the ^ and $ anchors match at line breaks in addition to the start and end of the entire string. Useful for patterns that should match line by line.
  • Ignore Case: Enable case-insensitive matching. When enabled, the pattern matches regardless of letter case (e.g., “EMAIL” matches “email”, “Email”, “EMAIL”).

Name: email_pattern
Target: Content
Patterns:
    Name: "Standard Email"
    Regex: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
    Score: 0.9
Context: ["email", "e-mail", "contact"]
Confidence Threshold: 0.7

​

Exact Terms Settings

• Term List: List of specific values to match against. The recognizer will identify data that exactly matches any value in this list. Useful for identifying specific sensitive terms, names, or identifiers that should be flagged. For example, [“confidential”, “secret”, “internal-only”, “do-not-share”]
• Supported Language: Language scope for this recognizer. Defaults to All.
• Regex Flags: Toggle Dot All, Multiline, and Ignore Case as needed.
  • Dot All: Enable dot-all regex flag. When enabled, the dot (.) metacharacter matches newline characters as well as any other character. Useful for multi-line pattern matching.
  • Multiline: Enable multiline regex flag. When enabled, the ^ and $ anchors match at line breaks in addition to the start and end of the entire string. Useful for patterns that should match line by line.
  • Ignore Case: Enable case-insensitive matching. When enabled, the pattern matches regardless of letter case (e.g., “EMAIL” matches “email”, “Email”, “EMAIL”).

Name: internal_dept_codes
Target: Content
Exact Terms:
  - DEPT_001_CONFIDENTIAL
  - DEPT_002_RESTRICTED
Ignore Case: true
Confidence Threshold: 0.9

​

Predefined Recognizer Settings

• Pre-Defined Recognizer: Select a pre-configured recognizer from the system.
  • Financial: AbaRoutingRecognizer, CreditCardRecognizer, IbanRecognizer, UsBankRecognizer
  • Personal ID: UsSsnRecognizer, UsItinRecognizer, UsPassportRecognizer, UsLicenseRecognizer, InPanRecognizer, InAadhaarRecognizer
  • Healthcare: NhsRecognizer, MedicalLicenseRecognizer, AuMedicareRecognizer
  • Contact: EmailRecognizer, PhoneRecognizer, UrlRecognizer, IpRecognizer
  • Geographic: EsNifRecognizer, EsNieRecognizer, UkNinoRecognizer, AuAbnRecognizer, PlPeselRecognizer
  • Advanced: SpacyRecognizer, StanzaRecognizer, TransformersRecognizer, GLiNERRecognizer
• Context (optional): Context that helps boost the confidence score when found near detected entities. For example, for EmailRecognizer, context might include: “email”, “contact”, “address”, “send”.
• Supported Language: Language scope for this recognizer. Defaults to All.

Name: us_ssn_detector
Target: Content
Predefined Recognizer: UsSsnRecognizer
Context: ["SSN", "social", "security"]
Confidence Threshold: 0.8

​

Column Name Recognizer Settings

Name: pii_column_name_detector
Target: Column Name
Type: Exact Terms
Exact Terms:
  - email
  - person_id
  - customer_email
Ignore Case: true
Confidence Threshold: 1.0

Name: pii_column_pattern_detector
Target: Column Name
Type: Pattern
Patterns:
  - Name: "Email columns"
    Regex: .*_email$
    Score: 1.0
  - Name: "ID columns"
    Regex: .*_id$
    Score: 0.8
Confidence Threshold: 0.8

​

Managing recognizers

​

View all recognizers

• Enabled: Toggle to activate/deactivate
• Name & Description
• Type: Pattern, Exact Terms, Predefined, or Context
• Target: Content or Column Name
• Exceptions: Number of entities excluded from this recognizer
• Confidence: Confidence threshold

​

Filter & Search

• Type: Pattern, Exact Terms, Predefined, Context
• Target: Content, Column Name
• Enabled: Enabled, Disabled

​

Edit a Recognizer

1. Click Edit (pencil icon) in the Actions column
2. Modify fields in the form
3. Click Save

​

Delete a Recognizer

1. Click Delete (trash icon) in the Actions column
2. Confirm deletion

​

Enable/Disable recognizers

​

Managing Exceptions

1. Click exceptions count
2. View list in the exceptions panel
3. Click delete on specific exceptions
4. Confirm removal

​

Best Practices

​

Creating effective recognizers

1. Start with high confidence: Begin with threshold 0.7-0.8, adjust if needed
2. Test patterns first: Validate regex patterns with sample data before creating the recognizer
3. Use context words: Add relevant context to reduce false positives
4. Multiple patterns: Create separate patterns for different formats (e.g., phone: (123)456-7890 vs 123-456-7890)
5. Descriptive names: Use clear, searchable names and descriptions

​

Managing False Positives

1. Review feedback regularly: Check pending feedback from users
2. Adjust thresholds: If too many false positives, increase confidence threshold
3. Refine patterns: Edit patterns to be more specific
4. Add context words: Boost confidence for true positives with relevant context

​

Performance Tips

1. Target appropriately: Use “Column Name” target when possible (faster than content analysis)
2. Disable unused recognizers: Deactivate recognizers you no longer need
3. Combine patterns: Use one recognizer with multiple patterns instead of many single-pattern recognizers
4. Limit context words: Keep context word lists concise (under 20 words)

​

Troubleshooting

​

Recognizer Not Detecting Data

• Recognizer is Enabled
• Confidence threshold not too high
• Pattern syntax is correct (test with a regex tool)
• Target matches your use case (Content vs Column Name)
• Entity is not in the exception list
• Profiler and auto-classification are enabled in ingestion config

​

Too Many False Positives

• Increase confidence threshold
• Add context words for true positives
• Make patterns more specific
• Consider using exact terms recognizer instead
• Let users submit feedback to build exception lists

​

Pattern Not Matching

• Missing escape characters in regex (use \\d not \d)
• Incorrect regex flags (check case sensitivity, multi-line)
• Pattern too specific or too broad
• Test your pattern at regex101.com first

​

Next Steps