​

Overview

The Anomaly Detection feature in Collate helps ensure data quality by automatically detecting unexpected changes, such as spikes or drops in data trends. Instead of requiring you to manually define rigid boundaries for data validation, Collate dynamically learns from your data patterns through regular profiling. Collate detects anomalies more accurately, alerting you only when deviations might indicate underlying issues.

​

Key Benefits of Anomaly Detection

• Automated Detection of Unexpected Data Changes: Collate can detect unexpected data behaviors, such as spikes or drops, that deviate from normal trends. This is crucial for identifying potential issues with data pipelines, backend systems, or infrastructure.
• Dynamic Learning: The system continuously profiles your data over time, learning its natural variations, including seasonal fluctuations. For example, if sales data varies throughout the year due to holidays, Collate’s dynamic assertions can detect this seasonality and prevent unnecessary error alerts. The system automatically adjusts to your data’s evolving patterns without manual configuration.
• Flexible Configuration: For more controlled scenarios, you can still define specific boundaries or thresholds to monitor data, such as ensuring values stay within a certain range. This offers both manual and automatic methods for managing data quality.

​

Use Cases

The following scenarios show how anomaly detection applies to different types of data.

​

1. Static Assertions for Simple Tests

• Problem: In many cases, you want to perform straightforward data tests, such as ensuring that values are not null or that there are no repeated values.
• Solution: Collate lets you configure simple assertions directly from the UI. For example, create tests to ensure:
  • Data should not be null.
  • There should be no duplicate values.
  • Data should not be older than a specific time frame (for example, one day).
  • Values should be greater than zero.
• Example: If you want to ensure that your sales data contains no null values or duplicates, configure these assertions via the UI.

​

2. Dynamic Assertions for Evolving Data

• Problem: Some data, such as sales figures, naturally evolves over time. For example, sales data might fluctuate daily or weekly, and manual bounds may not accurately capture these variations.
• Solution: Collate uses dynamic assertions, which automatically learn from the data by profiling it regularly. Over time, the system establishes a pattern for how the data behaves, allowing it to detect when values significantly deviate from this expected behavior.
• Example: If sales suddenly spike or drop beyond what is typical for your historical data, Collate will alert you to this anomaly.

​

How Anomaly Detection Works

Collate supports two modes of anomaly detection: manual test configuration and dynamic assertions.

​

1. Manual Configuration of Tests

Configure tests manually for specific data points to maintain tight control over your data quality checks. For instance, specify a range, such as 10 to 100. This method is useful for data that has well-understood constraints or when precise validation rules are required.

​

2. Dynamic Assertions

For more complex or evolving datasets, Collate offers dynamic assertions. These assertions automatically adapt to your data by learning its natural patterns over time. The profiling process typically takes around five weeks, during which the system builds an understanding of normal data fluctuations.

• Data Profiling: Collate continuously scans the data and trains its models based on the profiled data. Once this learning phase is complete, the system can detect significant deviations from expected patterns, alerting you to anomalies.
• Advantages of Dynamic Assertions:
  • Adaptability: No need to set manual thresholds for evolving datasets.
  • Efficiency: Focus on genuine anomalies instead of managing static tests that may quickly become outdated as data evolves.

​

3. Incidents and Notifications

When an anomaly is detected, Collate automatically generates incidents, including for rule-based test cases. These notifications help you quickly understand when and where your data may be behaving unexpectedly.

• Example: If sales data suddenly shows an abnormal spike or drop, Collate will notify you, allowing you to investigate potential causes such as system malfunctions or external influences.