Connecting Databricks

Read the concepts first for push vs pull and snapshots. This page covers what is specific to Databricks.

Where the data comes from

Databricks keeps usage history in system tables in the system catalog, managed by Unity Catalog:

We read only these. They are separate from your own catalogs, so granting access to system gives no access to your data.

Prerequisites

• A workspace on Unity Catalog. Without it there are no system tables.
• Databricks Community Edition won't work (no system tables). A trial or paid account on AWS, Azure or GCP does.
• A metastore admin to enable system schemas and grant access.
• For pull mode, an account admin to create the service principal and secret.

Enable the system schemas first

Some system schemas are off until an admin enables them (billing is on by default). Find your metastore id (Catalog Explorer → metastore details, or databricks metastores current), then:

databricks system-schemas enable <metastore-id> compute
databricks system-schemas enable <metastore-id> lakeflow
databricks system-schemas enable <metastore-id> query
databricks system-schemas enable <metastore-id> serving

Data starts accumulating from the moment a schema is enabled, so enable them early.

Option A — Push (recommended)

You import a ready-made notebook and schedule it as a daily job. It reads the system tables inside your workspace and uploads the result to us. We have no credential for your workspace and no way in.

Teksoptima                                Your Databricks workspace
──────────                                ─────────────────────────
Generates a notebook + upload token ────► 1. Store the token in a secret scope
                                          2. Import the notebook
                                          3. Schedule it as a daily Job
                                          4. Job reads system.* (SELECT only)
 validates + stores snapshot ◄─────────── 5. Uploads gzip'd parts over HTTPS

The pieces

Grants

Run as a metastore admin; <job-owner> is who the job runs as:

GRANT USE CATALOG ON CATALOG system TO `<job-owner>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.billing  TO `<job-owner>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.compute  TO `<job-owner>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.lakeflow TO `<job-owner>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.query    TO `<job-owner>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.serving  TO `<job-owner>`;

That's the entire permission footprint: SELECT on metadata, nothing on your data.

Privacy options in the notebook

• Pseudonymise identities (on by default). Emails (query authors, cluster owners, job run-as) are replaced with user- + 12 characters of a salted hash, computed inside your workspace before upload. The salt lives in your secret scope and we never receive it, so we can't reverse it. You can map a hash back to a person with one query in your own workspace. Hashes are stable across uploads, so per-user findings still work.
• Share query text (off by default). See the concepts.

Stopping it

Delete the job, or regenerate the upload token in Teksoptima — the old one stops working at once.

Option B — Pull (live) with a service principal

Here Teksoptima connects to your workspace itself. Databricks uses OAuth machine-to-machine (M2M): a non-human identity authenticates with a client ID and secret and receives a short-lived token.

The key idea: a service principal, not a person

A service principal is an identity for software. Using one means the connection doesn't depend on someone's login, MFA or job status, and you can grant it only the access it needs and see exactly what it did.

Like Snowflake, the service principal and its secret belong to your account, so each customer creates their own and enters it in the connection wizard. No Databricks credential is shared across customers.

Moving parts

How a pull works

1. We exchange the client ID and secret for an access token at your workspace's /oidc/v1/token (grant_type=client_credentials).
2. We run the dataset queries on the warehouse you designated, using the SQL Statement Execution API.
3. We store the results as a snapshot.

The access token lasts about an hour and is kept in memory only, never saved. Only the client secret is stored, encrypted (AES-256-GCM), and it is never shown, logged or returned to the browser again after you save it.

Verify the grants

The wizard runs a harmless SELECT … LIMIT 0 against each system table and tells you by name which one is missing.

Seeing what we did

Every query the service principal runs appears in your own query history and audit log, so you don't have to take our word for it:

SELECT start_time, statement_text
FROM system.query.history
WHERE executed_by = '<service-principal-application-id>'
ORDER BY start_time DESC;

Things that commonly go wrong

Undoing it

REVOKE the grants, delete the secret, or delete the service principal — each cuts access immediately. Switching the connection back to push deletes the stored secret.

Which should I pick?

• Push if you can import a notebook and schedule a job. We hold nothing of yours and have no inbound path.
• Pull if you'd rather not run a job in your workspace and are comfortable creating a service principal that we hold an encrypted secret for.