Skip to main content

Backup and Restore Metadata

Collate SaaS Backups

Collate manages SaaS backups automatically at the platform level. There is no backup job to configure or monitor in the Collate UI, and backups do not appear among ingestion pipelines or applications.
  • Backups run once a day during a scheduled maintenance window.
  • Backups are retained for approximately two weeks.
  • A backup includes metadata, configurations, users, teams, policies, lineage, data quality data, ingestion pipelines, and other information stored in the Collate database.
  • Restores recover the full database. Selective database restores for individual assets, glossary terms, tags, or configurations are not supported.
To request a restore, open a ticket with Collate Support and provide the preferred backup timestamp. A typical restore takes approximately two to three hours and includes approximately one hour of downtime. Support is available 24 hours a day, Monday through Friday. Restore requests submitted during the weekend are reviewed on Monday.
Restoring a backup replaces the environment’s current database state. Changes made after the selected backup timestamp are lost. The backup should be restored to the same Collate version, and the search index must be rebuilt after the database restore.
For recovery of individual objects, use version history, soft delete, CSV exports and imports, or the API where applicable.

Self-Hosted Backups

Before upgrading your OpenMetadata version we strongly recommend backing up the metadata. The source of truth is stored in the underlying database (MySQL and Postgres supported). During each version upgrade there is a database migration process that needs to run. It will make changes to the OpenMetadata Application schema in the database and update the shape of the data to the newest OpenMetadata release. It is important that we backup the data because if we face any unexpected issues during the upgrade process, you will be able to get back to the previous version without any loss.
You can learn more about how the migration process works here.
Since version 1.4.0, OpenMetadata encourages using the builtin-tools for creating logical backups of the metadata: For PROD deployment we recommend users to rely on cloud services for their databases, be it AWS RDS, Azure SQL or GCP Cloud SQL. If you’re a user of these services, you can leverage their backup capabilities directly:

Requirements

  • mysqldump 8.3 or higher
  • pg_dump 13.3 or higher
If you’re running the project using docker compose, the ingestion container already comes packaged with the correct mysqldump and pg_dump versions ready to use.

Storing the backup files

It’s important that when you backup your database, you keep the snapshot safe in case you need in later. You can check these two examples on how to:
  • Use pipes to stream the result directly to S3, e.g. pg_dump ... | aws s3 cp - s3://<bucket>/<file> (see AWS CLI’s s3 cp stdin/stdout support).
  • Dump to a file and copy to storage (link).

Example with Docker

Start a local instance of OpenMetadata using the docker-compose file provided in the repository. Then, we can use the following commands to backup the metadata:

MySQL

1. Backup and Restore

2. Restart the docker deployment with the restored database

PostgreSQL

1. Backup and Restore

2. Restart the docker deployment with the restored database