Column Name Scanner: Collate validates the column names of the table against a set of regex rules that help identify
common English patterns to identify email addresses, SSN, bank accounts, and so on.
Entity Recognition: If the sample data ingestion is enabled, Collate validates the sample rows against an Entity
Recognition engine that brings up any sensitive information from a list of supported entities.
In that case, the confidence parameter lets you tune the minimum score required to tag a column as PII.Sensitive.
Note that if a column is already tagged as PII, Collate ignores its execution.
Unexpected error while processing sample data for auto pii tagging - HTTPSConnectionPool(host='raw.githubusercontent.com', port=443):Max retries exceeded with url: /explosion/spacy-models/master/compatibility.json(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable toget local issuer certificate (_ssl.c:1129)')))
This is a scenario identified on some corporate Windows laptops. The bottom-line here is that the profiler
is trying to download the Entity Recognition model but having certificate issues when trying the request.A solution here is to manually download the model on the ingestion container / Airflow host by running:
