Reindexing Search
If you are encountering any of the following issues:
- Mismatch in counts of Data Assets.
- Data not appearing.
- Errors such as
shards_exception.
- Problems with the search mechanism.
- Empty results in the Explore section.
- Missing lineage information.
Run the Search Indexing app to initiate the reindexing process.
Reindex
-
Select the Profile icon and go to Settings > Applications > Search Indexing.
-
In the Configuration tab, check whether Recreate Indexes is enabled or not. If not, enable it and save the settings.
-
Click Run Now.
If you want to reindex specific entities, in the Configuration tab, add the entities.
Tip:
Note: If you continue to experience issues, re-install the Search Indexing application.
Configuration Parameters for Reindexing
Batch Size
Description — Maximum number of entities sent in a single batch during reindexing. Default: 100.
Best Practice — Adjust based on system capabilities. A larger batch size improves throughput but may increase memory usage. Monitor performance and fine-tune accordingly.
Payload Size
Description — Maximum payload size in bytes per batch. Default: 104,857,600 bytes (100 MB).
Best Practice — Ensure the payload size is within your system’s capacity. If memory issues or timeouts occur, reduce this value.
Number of Producer Threads
Description — Number of threads used to produce reindexing events. Default: 1.
Best Practice — Increasing this improves throughput but may cause contention if set too high. Balance with system CPU and I/O capacity.
Number of Consumer Threads
Description — Number of threads used to consume reindexing events. Default: 1.
Best Practice — Keep in sync with producer threads. Increase when the queue fills up frequently.
Queue Size
Description — Internal queue size for reindexing operations. Default: 100.
Best Practice — A larger queue handles processing spikes but requires more memory. Adjust based on expected load.
Max Concurrent Requests
Description — Maximum number of simultaneous requests sent to the search index. Default: 100.
Best Practice — Tune based on your indexing server’s capacity. Too many concurrent requests can overwhelm the server.
Max Retries
Description — Maximum retry attempts for failed requests. Default: 3.
Best Practice — Keep this reasonable to avoid excessive load during failures.
Initial Backoff Millis
Description — Wait time in milliseconds before the first retry. Default: 1000 ms (1 second).
Best Practice — Increase if failures occur frequently due to server overload or network issues.
Max Backoff Millis
Description — Maximum wait time in milliseconds between retries. Default: 10,000 ms (10 seconds).
Best Practice — Align with your latency tolerance. A longer backoff reduces load during peak times but slows recovery.
Entities
Description — Entity types to include in the reindexing process. Defaults to All entities.
Best Practice — When troubleshooting a specific entity type, select only that type to speed up the reindex and reduce system load.
Recreate Indexes
Description — When enabled, drops the existing index and builds a new one from scratch.
Best Practice — Enable only when you need a clean slate — such as after major data model changes or during migration. Leave disabled for routine reindexing to preserve the existing index.
Search Index Language
Description — Language used when recreating indexes. Default: EN.
Best Practice — Change only if your data assets are primarily in a language other than English. Changing the language recreates indexes with updated language-specific analyzers.
Auto Tune
Description — When enabled, automatically adjusts performance settings based on cluster capabilities and entity count.
Best Practice — Leave enabled unless you want manual control over performance parameters.
Use Distributed Indexing
Description — When enabled, distributes the indexing workload across partitions for improved throughput.
Best Practice — Enable on large datasets or multi-node clusters. On small single-node setups, the overhead may outweigh the benefit.
Partition Size
Description — Chunk size used when distributing indexing work across partitions. Default: 10,000.
Best Practice — Use larger values for high-throughput environments. Reduce if individual partitions are causing memory pressure.
Time Series Max Days
Description — Limits reindexing to time-series data within the specified number of days. Default: 0 (reindex all data).
Best Practice — Set a positive value (e.g., 15) to limit reindexing to recent data. This significantly reduces reindexing time for large datasets.
Live Index Settings
Settings applied to the index before it goes live. These control read freshness and availability during normal operation.
Number of Replicas
Description — Number of replica shards for the live index. Default: 1.
Best Practice — Use 1 for high availability on multi-node clusters; use 0 for single-node setups.
Refresh Interval
Description — How often new writes become searchable. Default: 1s.
Best Practice — The default provides near-real-time search. Increase (e.g., 5s) to reduce overhead on write-heavy workloads.
Translog Durability
Description — Controls when the translog is flushed to disk. Options: request or async.
Best Practice — Use request for stronger durability. Switch to async only if you need higher write throughput and can tolerate the risk.
Translog Sync Interval
Description — How frequently the translog syncs when durability is set to async. Default: 5s.
Best Practice — Keep low if durability is important. Increase to improve write performance when some data loss risk is acceptable.
Bulk Index Settings (During Reindex)
Temporary settings applied during the bulk reindex write phase to maximize throughput. These automatically revert to Live Index Settings before the alias swap.
Number of Replicas
Description — Replica count during bulk write. Default: 0.
Best Practice — Keep at 0 during reindex. Replicas are populated after the new index is promoted.
Refresh Interval
Description — How often the index is refreshed during bulk write. Default: -1 (disabled).
Best Practice — Leave at -1. Disabling refresh dramatically speeds up indexing. The staged index is not searchable during this phase, so this is safe.
Translog Durability
Description — Durability setting used during bulk write. Default: async.
Best Practice — Using async improves write speed. Settings revert to live values before the index becomes searchable.
Translog Sync Interval
Description — How frequently the translog syncs during bulk indexing. Default: 30s.
Best Practice — A longer sync interval reduces I/O overhead during the write phase.
Force-merge before alias swap
Description — When enabled, forces a segment merge before the index goes live.
Best Practice — Leave disabled for most reindexing runs. Enable only when optimizing the index for read performance, at the cost of additional reindex time.
Example Configuration for Best Practices
For high-performance systems, consider the following values as a starting point:
Monitor system performance and adjust these parameters to optimize throughput and resource usage.