> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcollate.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Pagination

> Navigate through large result sets using cursor-based pagination

export const CodeLayout = ({title, description, children}) => {
  return <div className="code-layout">
      <div>
        {title && <h2>{title}</h2>}
        {description && <div>
            {description}
          </div>}
      </div>

      <div className="code-layout-sticky">
        {children}
      </div>
    </div>;
};

# Pagination

The Collate API uses cursor-based pagination for list endpoints. This ensures consistent results even when data changes between requests.

<CodeLayout
  title="How It Works"
  description={
<>
When you request a list of resources, the response includes:
<ul>
  <li>A <code>data</code> array containing resources (up to the <code>limit</code>)</li>
  <li>A <code>paging</code> object with cursors for navigation</li>
  <li>A <code>total</code> count of matching resources</li>
  <li>Optional <code>before</code> and <code>after</code> cursors for pagination</li>
</ul>
</>
}
>
  <CodeGroup>
    ```json Pagination Example theme={null}
    {
      "data": [
        { "id": "...", "name": "table1", ... },
        { "id": "...", "name": "table2", ... }
      ],
      "paging": {
        "total": 150,
        "after": "eyJsYXN0SWQiOiIxMjM0NTY3ODkwIn0="
      }
    }
    ```
  </CodeGroup>
</CodeLayout>

## Pagination Parameters

| Parameter | Type    | Default | Description                                                                                                                                                                                 |
| --------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `limit`   | integer | 10      | Number of results per page. Main list endpoints commonly allow up to 1,000,000; individual endpoints can define lower caps.                                                                 |
| `before`  | string  | -       | Cursor for previous page                                                                                                                                                                    |
| `after`   | string  | -       | Cursor for next page                                                                                                                                                                        |
| `offset`  | integer | 0       | Number of results to skip. Available on specific sub-resource endpoints, for example `/v1/tables/{id}/columns`. Not supported on main list endpoints; use `before`/`after` cursors instead. |

## Response Fields

The exact `paging` fields vary by endpoint and pagination mode:

| Field    | Type    | Description                                                           |
| -------- | ------- | --------------------------------------------------------------------- |
| `total`  | integer | Total count of matching resources                                     |
| `before` | string  | Cursor for the previous page when available                           |
| `after`  | string  | Cursor for the next page when available                               |
| `offset` | integer | Current offset position when the endpoint returns offset-based paging |
| `limit`  | integer | Page size when the endpoint returns offset-based paging               |

## Examples

### Basic Pagination

<CodeGroup dropdown>
  ```python Basic Pagination theme={null}
  from metadata.ingestion.ometa.ometa_api import OpenMetadata
  from metadata.generated.schema.entity.data.table import Table

  metadata = OpenMetadata(config)

  # Get first page
  first_page = metadata.list_entities(
      entity=Table,
      limit=20
  )

  print(f"Total tables: {first_page.paging.total}")
  for table in first_page.entities:
      print(table.fullyQualifiedName)

  # Get next page using after cursor
  if first_page.paging.after:
      next_page = metadata.list_entities(
          entity=Table,
          limit=20,
          after=first_page.paging.after
      )
      for table in next_page.entities:
          print(table.fullyQualifiedName)
  ```

  ```java Basic Pagination theme={null}
  import org.openmetadata.client.api.TablesApi;
  import org.openmetadata.client.model.TableList;

  TablesApi tablesApi = client.buildClient(TablesApi.class);

  // Get first page
  TableList firstPage = tablesApi.listTables(null, 20, null, null, null);

  System.out.println("Total tables: " + firstPage.getPaging().getTotal());
  for (Table table : firstPage.getData()) {
      System.out.println(table.getFullyQualifiedName());
  }

  // Get next page
  if (firstPage.getPaging().getAfter() != null) {
      TableList nextPage = tablesApi.listTables(
          null, 20, null, firstPage.getPaging().getAfter(), null
      );
      for (Table table : nextPage.getData()) {
          System.out.println(table.getFullyQualifiedName());
      }
  }
  ```

  ```bash Basic Pagination theme={null}
  # Get first page
  curl -X GET "https://your-company.getcollate.io/api/v1/tables?limit=20" \
    -H "Authorization: Bearer $TOKEN"

  # Response includes paging.after cursor
  # {
  #   "data": [...],
  #   "paging": {
  #     "total": 150,
  #     "after": "eyJsYXN0SWQiOiIxMjM0NTY3ODkwIn0="
  #   }
  # }

  # Get next page using after cursor
  curl -X GET "https://your-company.getcollate.io/api/v1/tables?limit=20&after=eyJsYXN0SWQiOiIxMjM0NTY3ODkwIn0=" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

### Offset-Based Pagination

Use `offset` on sub-resource endpoints such as `/v1/tables/{id}/columns`. Endpoint-specific defaults and caps apply; table columns are limited to 1,000 columns per request and default to 50.

<CodeGroup>
  ```bash Offset Pagination theme={null}
  TABLE_ID="00000000-0000-0000-0000-000000000000"

  # Get first 20 columns
  curl -X GET "https://your-company.getcollate.io/api/v1/tables/${TABLE_ID}/columns?limit=20&offset=0" \
    -H "Authorization: Bearer $TOKEN"

  # Get next 20 columns
  curl -X GET "https://your-company.getcollate.io/api/v1/tables/${TABLE_ID}/columns?limit=20&offset=20" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

Some endpoints that accept `offset` still return `paging.before` and `paging.after` as previous and next page markers. Follow the response fields returned by the endpoint you are calling.

### Iterating Through All Results

<CodeGroup dropdown>
  ```python Iterating Results theme={null}
  from metadata.ingestion.ometa.ometa_api import OpenMetadata
  from metadata.generated.schema.entity.data.table import Table

  metadata = OpenMetadata(config)

  # Iterate through all tables
  for table in metadata.list_all_entities(entity=Table, limit=100):
      print(table.fullyQualifiedName)
      # Process each table...
  ```

  ```java Iterating Results theme={null}
  import org.openmetadata.client.api.TablesApi;

  TablesApi tablesApi = client.buildClient(TablesApi.class);
  String afterCursor = null;

  do {
      TableList page = tablesApi.listTables(null, 100, null, afterCursor, null);

      for (Table table : page.getData()) {
          System.out.println(table.getFullyQualifiedName());
          // Process each table...
      }

      afterCursor = page.getPaging().getAfter();
  } while (afterCursor != null);
  ```

  ```bash Iterating Results theme={null}
  #!/bin/bash

  after_cursor=""

  while true; do
    # Build URL with optional cursor
    url="https://your-company.getcollate.io/api/v1/tables?limit=100"
    if [ -n "$after_cursor" ]; then
      url="${url}&after=${after_cursor}"
    fi

    # Fetch page
    response=$(curl -s -X GET "$url" \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json")

    # Process results
    echo "$response" | jq -r '.data[].fullyQualifiedName'

    # Get next cursor
    after_cursor=$(echo "$response" | jq -r '.paging.after // empty')

    # Exit if no more pages
    if [ -z "$after_cursor" ]; then
      break
    fi
  done
  ```
</CodeGroup>

## Filtering with Pagination

Combine pagination with filters for efficient data retrieval:

<CodeGroup dropdown>
  ```python Filtering theme={null}
  from metadata.ingestion.ometa.ometa_api import OpenMetadata
  from metadata.generated.schema.entity.data.table import Table

  metadata = OpenMetadata(config)

  # List tables from a specific database
  tables = metadata.list_entities(
      entity=Table,
      params={"database": "prod.analytics"},
      limit=50
  )

  for table in tables.entities:
      print(table.fullyQualifiedName)
  ```

  ```bash Filtering theme={null}
  # Filter tables by database
  curl -X GET "https://your-company.getcollate.io/api/v1/tables?database=prod.analytics&limit=50" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

## Include Fields

Control which fields are returned in the response using the `fields` parameter:

<CodeGroup>
  ```bash Include Fields theme={null}
  # Request specific fields
  curl -X GET "https://your-company.getcollate.io/api/v1/tables?fields=owner,tags,columns&limit=20" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

Common field options for tables:

* `owner` - Include owner information
* `tags` - Include tags and classifications
* `columns` - Include column definitions
* `followers` - Include followers
* `tableConstraints` - Include constraints
* `usageSummary` - Include usage statistics

## Best Practices

<Steps>
  <Step title="Use reasonable page sizes">
    Start with `limit=50-100`. Larger pages reduce API calls but increase memory usage. Check the endpoint's maximum limit before increasing page size.
  </Step>

  <Step title="Use the pagination mode from the response">
    For cursor-based list endpoints, use `before` and `after` sequentially. For endpoints that expose `offset`, advance by the page size or follow the next marker returned in `paging`.
  </Step>

  <Step title="Handle empty results">
    Check if the `data` array is empty. For cursor-based pagination, stop when `after` is null. For offset-based requests, stop when the response returns fewer records than requested or no next marker is present.
  </Step>

  <Step title="Request only needed fields">
    Use the `fields` parameter to reduce response size and improve performance.
  </Step>

  <Step title="Implement retry logic">
    Handle transient errors gracefully when paginating through large datasets.
  </Step>
</Steps>

## Pagination vs Search

For finding specific resources, consider using the Search API instead of paginating through all results:

<CodeGroup>
  ```bash Pagination vs Search theme={null}
  # Search is faster for finding specific resources
  curl -X GET "https://your-company.getcollate.io/api/v1/search/query?q=customers&index=table_search_index" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

<CardGroup cols={2}>
  <Card title="Search API" icon="magnifying-glass" href="/api-reference/discovery/search" horizontal>
    Learn about searching and filtering metadata
  </Card>
</CardGroup>