Pagination

The Collate API uses cursor-based pagination for list endpoints. This ensures consistent results even when data changes between requests.

Pagination Parameters

Parameter	Type	Default	Description
`limit`	integer	10	Number of results per page. Main list endpoints commonly allow up to 1,000,000; individual endpoints can define lower caps.
`before`	string	-	Cursor for previous page
`after`	string	-	Cursor for next page
`offset`	integer	0	Number of results to skip. Available on specific sub-resource endpoints, for example `/v1/tables/{id}/columns`. Not supported on main list endpoints; use `before`/`after` cursors instead.

Response Fields

The exact paging fields vary by endpoint and pagination mode:

Field	Type	Description
`total`	integer	Total count of matching resources
`before`	string	Cursor for the previous page when available
`after`	string	Cursor for the next page when available
`offset`	integer	Current offset position when the endpoint returns offset-based paging
`limit`	integer	Page size when the endpoint returns offset-based paging

Examples

Basic Pagination

from metadata.ingestion.ometa.ometa_api import OpenMetadata
from metadata.generated.schema.entity.data.table import Table

metadata = OpenMetadata(config)

# Get first page
first_page = metadata.list_entities(
    entity=Table,
    limit=20
)

print(f"Total tables: {first_page.paging.total}")
for table in first_page.entities:
    print(table.fullyQualifiedName)

# Get next page using after cursor
if first_page.paging.after:
    next_page = metadata.list_entities(
        entity=Table,
        limit=20,
        after=first_page.paging.after
    )
    for table in next_page.entities:
        print(table.fullyQualifiedName)

import org.openmetadata.sdk.models.ListParams;

// Get first page
var firstPage = client.tables().list(new ListParams().setLimit(20));

System.out.println("Total tables: " + firstPage.getPaging().getTotal());
for (var table : firstPage.getData()) {
    System.out.println(table.getFullyQualifiedName());
}

// Get next page
if (firstPage.getPaging().getAfter() != null) {
    var nextPage = client.tables().list(
        new ListParams()
            .setLimit(20)
            .setAfter(firstPage.getPaging().getAfter())
    );
    for (var table : nextPage.getData()) {
        System.out.println(table.getFullyQualifiedName());
    }
}

# Get first page
curl -X GET "https://your-company.getcollate.io/api/v1/tables?limit=20" \
  -H "Authorization: Bearer $TOKEN"

# Response includes paging.after cursor
# {
#   "data": [...],
#   "paging": {
#     "total": 150,
#     "after": "eyJsYXN0SWQiOiIxMjM0NTY3ODkwIn0="
#   }
# }

# Get next page using after cursor
curl -X GET "https://your-company.getcollate.io/api/v1/tables?limit=20&after=eyJsYXN0SWQiOiIxMjM0NTY3ODkwIn0=" \
  -H "Authorization: Bearer $TOKEN"

Offset-Based Pagination

Use offset on sub-resource endpoints such as /v1/tables/{id}/columns. Endpoint-specific defaults and caps apply; table columns are limited to 1,000 columns per request and default to 50.

TABLE_ID="00000000-0000-0000-0000-000000000000"

# Get first 20 columns
curl -X GET "https://your-company.getcollate.io/api/v1/tables/${TABLE_ID}/columns?limit=20&offset=0" \
  -H "Authorization: Bearer $TOKEN"

# Get next 20 columns
curl -X GET "https://your-company.getcollate.io/api/v1/tables/${TABLE_ID}/columns?limit=20&offset=20" \
  -H "Authorization: Bearer $TOKEN"

Some endpoints that accept offset still return paging.before and paging.after as previous and next page markers. Follow the response fields returned by the endpoint you are calling.

Iterating Through All Results

Iterating Results

from metadata.ingestion.ometa.ometa_api import OpenMetadata
from metadata.generated.schema.entity.data.table import Table

metadata = OpenMetadata(config)

# Iterate through all tables
for table in metadata.list_all_entities(entity=Table, limit=100):
    print(table.fullyQualifiedName)
    # Process each table...

import org.openmetadata.sdk.models.ListParams;

String afterCursor = null;

do {
    var page = client.tables().list(
        new ListParams()
            .setLimit(100)
            .setAfter(afterCursor)
    );

    for (var table : page.getData()) {
        System.out.println(table.getFullyQualifiedName());
        // Process each table...
    }

    afterCursor = page.getPaging() != null ? page.getPaging().getAfter() : null;
} while (afterCursor != null);

#!/bin/bash

after_cursor=""

while true; do
  # Build URL with optional cursor
  url="https://your-company.getcollate.io/api/v1/tables?limit=100"
  if [ -n "$after_cursor" ]; then
    url="${url}&after=${after_cursor}"
  fi

  # Fetch page
  response=$(curl -s -X GET "$url" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json")

  # Process results
  echo "$response" | jq -r '.data[].fullyQualifiedName'

  # Get next cursor
  after_cursor=$(echo "$response" | jq -r '.paging.after // empty')

  # Exit if no more pages
  if [ -z "$after_cursor" ]; then
    break
  fi
done

Filtering with Pagination

Combine pagination with filters for efficient data retrieval:

Filtering

from metadata.ingestion.ometa.ometa_api import OpenMetadata
from metadata.generated.schema.entity.data.table import Table

metadata = OpenMetadata(config)

# List tables from a specific database
tables = metadata.list_entities(
    entity=Table,
    params={"database": "prod.analytics"},
    limit=50
)

for table in tables.entities:
    print(table.fullyQualifiedName)

Include Fields

Control which fields are returned in the response using the fields parameter:

# Request specific fields
curl -X GET "https://your-company.getcollate.io/api/v1/tables?fields=owner,tags,columns&limit=20" \
  -H "Authorization: Bearer $TOKEN"

Common field options for tables:

owner - Include owner information
tags - Include tags and classifications
columns - Include column definitions
followers - Include followers
tableConstraints - Include constraints
usageSummary - Include usage statistics

Best Practices

Use reasonable page sizes: Start with limit=50-100. Larger pages reduce API calls but increase memory usage. Check the endpoint’s maximum limit before increasing page size.
Use the pagination mode from the response: For cursor-based list endpoints, use before and after sequentially. For endpoints that expose offset, advance by the page size or follow the next marker returned in paging.
Handle empty results: Check if the data array is empty. For cursor-based pagination, stop when after is null. For offset-based requests, stop when the response returns fewer records than requested or no next marker is present.
Request only needed fields: Use the fields parameter to reduce response size and improve performance.
Implement retry logic: Handle transient errors gracefully when paginating through large datasets.

Pagination vs Search

For finding specific resources, consider using the Search API instead of paginating through all results:

# Search is faster for finding specific resources
curl -X GET "https://your-company.getcollate.io/api/v1/search/query?q=customers&index=table_search_index" \
  -H "Authorization: Bearer $TOKEN"

Search API

Learn about searching and filtering metadata

​Pagination

​Pagination Parameters

​Response Fields

​Examples

​Basic Pagination

​Offset-Based Pagination

​Iterating Through All Results

​Filtering with Pagination

​Include Fields

​Best Practices

​Pagination vs Search