How to connect to Stardog as a Spark Table in Databricks

Created by Steve Place, Modified on Thu, Jun 13 at 4:46 PM by Steve Place

First, follow the instructions here, using your Stardog BI endpoint as the database_host and 5806 as the port.

If you need to connect over SSL, include the encrypt and trustServerCertificate options, with both set to true.

Your code would look like this:

driver = "org.mariadb.jdbc.Driver"

database_host = "example-company-bi-server.stardog.cloud"
database_port = "5806" # update if you use a non-default port
database_name = "your_db_name_here"
table = "`your_catalog.your_schema.your_table`"
user = your_stardog_user
password = your_password

url = f"jdbc:sqlserver://{database_host}:{database_port};database={database_name}"

remote_table = (spark.read
    .format("jdbc")
    .option("driver", driver)
    .option("url", url)
    .option("dbtable", table)
    .option("user", user)
    .option("password", password)
    .option("encrypt", "true")
    .option("trustServerCertificate", "true")
    .load()
)

If you'd like to store your password more securely, you can do so with a Databricks secret like so:

Create a secret scope: databricks secrets create-scope --scope mySecretScope
Add your password: databricks secrets put --scope mySecretScope --key stardogPassword
1. You'll be prompted to enter the secret value.
Reference the secret in your code:

from pyspark.dbutils import DBUtils

dbutils = DBUtils(spark)
password = dbutils.secrets.get(scope="mySecretScope", key="stardogPassword")