​

Spark Engine Prerequisites

​

Required Infrastructure

​

Spark Cluster

• Spark Connect available (versions 3.5.2 to 3.5.6 supported)
• Network access from the pipeline execution environment to the Spark Connect endpoint
• Network access from the pipeline execution environment to the Collate server

​

Database Drivers in Spark Cluster

• PostgreSQL: org.postgresql.Driver
• MySQL: com.mysql.cj.jdbc.Driver

​

Network Connectivity

• Outbound access to your Spark Connect endpoint (typically port 15002)
• Outbound access to your Collate server (typically port 8585)
• Inbound access from Spark workers to your source database

​

Verification Steps

1. Test Spark Connect: Verify connectivity from your pipeline environment to Spark Connect
2. Test Collate: Ensure your pipeline environment can reach the Collate API
3. Test Database: Confirm Spark workers can connect to your source database
4. Verify Drivers: Check that the appropriate database driver is available in your Spark cluster

​

Example Verification Commands

​

Test Spark Connect Connectivity

# Test basic connectivity to Spark Connect
telnet your_spark_connect_host 15002

# Or using curl if available
curl -X GET http://your_spark_connect_host:15002

​

Test Collate Connectivity

# Test Collate API connectivity
curl -X GET http://your_openmetadata_host:8585/api/v1/version

​

Test Database Connectivity from Spark

# Test database connectivity using Spark
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("DatabaseConnectivityTest") \
    .remote("<SPARK_CONNECT_HOST>:<SPARK_CONNECT_PORT>") \
    .config("spark.jars", "/path/to/your/database/driver.jar") \
    .getOrCreate()

# Test connection to your database
df = spark.read \
    .format("jdbc") \
    .option("url", "jdbc:your_database_url") \
    .option("dbtable", "your_test_table") \
    .option("user", "your_username") \
    .option("password", "your_password") \
    .load()

df.head()