> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcollate.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Spark Engine Prerequisites | Collate Spark Profiling Setup

> Learn about the required infrastructure, network connectivity, and setup for using Spark Engine in Collate.

# Spark Engine Prerequisites

## Required Infrastructure

### Spark Cluster

* **Spark Connect available** (versions 3.5.2 to 3.5.6 supported)
* **Network access** from the pipeline execution environment to the Spark Connect endpoint
* **Network access** from the pipeline execution environment to the Collate server

### Database Drivers in Spark Cluster

Depending on your source database, ensure the appropriate driver is installed in your Spark cluster:

* **PostgreSQL**: `org.postgresql.Driver`
* **MySQL**: `com.mysql.cj.jdbc.Driver`

<Tip>
  The specific driver versions should match your Spark version and database version for optimal compatibility.
</Tip>

## Network Connectivity

The pipeline execution environment must have:

* **Outbound access** to your Spark Connect endpoint (typically port 15002)
* **Outbound access** to your Collate server (typically port 8585)
* **Inbound access** from Spark workers to your source database

## Verification Steps

1. **Test Spark Connect**: Verify connectivity from your pipeline environment to Spark Connect
2. **Test Collate**: Ensure your pipeline environment can reach the Collate API
3. **Test Database**: Confirm Spark workers can connect to your source database
4. **Verify Drivers**: Check that the appropriate database driver is available in your Spark cluster

## Example Verification Commands

### Test Spark Connect Connectivity

```bash theme={null}
# Test basic connectivity to Spark Connect
telnet your_spark_connect_host 15002

# Or using curl if available
curl -X GET http://your_spark_connect_host:15002
```

### Test Collate Connectivity

```bash theme={null}
# Test Collate API connectivity
curl -X GET http://your_openmetadata_host:8585/api/v1/version
```

### Test Database Connectivity from Spark

```python theme={null}
# Test database connectivity using Spark
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("DatabaseConnectivityTest") \
    .remote("<SPARK_CONNECT_HOST>:<SPARK_CONNECT_PORT>") \
    .config("spark.jars", "/path/to/your/database/driver.jar") \
    .getOrCreate()

# Test connection to your database
df = spark.read \
    .format("jdbc") \
    .option("url", "jdbc:your_database_url") \
    .option("dbtable", "your_test_table") \
    .option("user", "your_username") \
    .option("password", "your_password") \
    .load()

df.head()
```

<CardGroup cols={2}>
  <Card title="Partitioning Requirements" href="/how-to-guides/data-quality-observability/profiler/spark-engine/partitioning">
    Learn about the partitioning requirements for Spark Engine.
  </Card>

  <Card title="Configuration" href="/how-to-guides/data-quality-observability/profiler/spark-engine/configuration">
    Configure your profiler pipeline to use Spark Engine.
  </Card>
</CardGroup>