Unity Catalog in 2026: Decommissioning Hive Metastore Safely

Unity Catalog in 2026: Decommissioning Hive Metastore Safely...

Unity Catalog in 2026: Decommissioning Hive Metastore Safely

The real-world problem: you can’t hold UC at arm’s length anymore

You’ve got multiple Databricks workspaces, a Hive Metastore that’s been around for years, a pile of S3/ADLS mounts, and production jobs that reference two-part names like database.table. Things mostly work. But over the last year, more and more platform features expect Unity Catalog (UC): serverless compute, OpenSharing (formerly Delta Sharing), cross-workspace governance, consistent lineage, and centralized auditing.

Security is pressing for better permissions hygiene and discoverability. BI teams want consistent three-level names across workspaces. Compliance wants PPIs masked consistently. Meanwhile, the Hive Metastore (HMS) has become a liability: fragmented permissions, per-workspace silos, and no clean way to govern external data access.

You know you need to move to Unity Catalog and decommission Hive Metastore. The question is how to do it without breaking workloads, losing permissions, or marooning external storage.

This article lays out a pragmatic plan that teams are using to migrate with confidence.

What’s changing (and what teams misunderstand)

Let’s walk through a concrete plan with code you can adapt.

Clear explanation with code: design, build, migrate

We’ll cover:

  1. Inventory and decisions
  2. Designing your UC target
  3. Building UC foundations (with SQL and PySpark)
  4. Registering/migrating data and permissions
  5. Updating compute, jobs, and BI
  6. Phased rollout and cutover

1) Inventory and decisions

Start with facts. You can automate much of this with system tables and workspace APIs.

Decisions you’ll make:

2) UC target design (the blueprints)

3) Build UC foundations

We’ll show SQL and PySpark examples. Replace cloud-specific parts for AWS/Azure/GCP as needed.

Prerequisites:

3.1 Create storage credentials

SQL (AWS IAM role example):

-- Storage credential using an AWS IAM role
CREATE STORAGE CREDENTIAL sc_finance_iam
WITH ROLE ARN 'arn:aws:iam::123456789012:role/databricks-finance-access'
COMMENT 'Finance data access via IAM role';

SQL (Azure managed identity example):

-- Storage credential using Azure Managed Identity
CREATE STORAGE CREDENTIAL sc_finance_msi
WITH AZURE_MANAGED_IDENTITY
CLIENT_ID '00000000-0000-0000-0000-000000000000'
COMMENT 'Finance data access via managed identity';

PySpark (for either case, run SQL from Python):

# Choose one of the SQL blocks above for your cloud; execute via spark.sql
spark.sql("""
CREATE STORAGE CREDENTIAL sc_finance_iam
WITH ROLE ARN 'arn:aws:iam::123456789012:role/databricks-finance-access'
COMMENT 'Finance data access via IAM role'
""")
3.2 Create external locations

SQL:

-- External location for raw/landing data
CREATE EXTERNAL LOCATION el_finance_raw
URL 's3://finance-raw/'
WITH STORAGE CREDENTIAL sc_finance_iam
COMMENT 'Raw finance landing bucket';

-- External location for curated delta tables
CREATE EXTERNAL LOCATION el_finance_curated
URL 's3://finance-curated/'
WITH STORAGE CREDENTIAL sc_finance_iam
COMMENT 'Curated finance delta tables';

Grant location privileges:

-- Allow data engineers to read/write files in curated, read-only in raw
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION el_finance_curated TO `data_engineers`;
GRANT READ FILES ON EXTERNAL LOCATION el_finance_raw TO `data_engineers`;
3.3 Create catalogs and schemas with managed locations

SQL:

-- Create the finance catalog with a managed location (Databricks manages storage)
CREATE CATALOG finance
MANAGED LOCATION 's3://finance-managed/'
COMMENT 'Finance domain catalog';

-- Create lifecycle schemas
CREATE SCHEMA finance.bronze COMMENT 'Raw ingested data';
CREATE SCHEMA finance.silver COMMENT 'Conformed/canonical data';
CREATE SCHEMA finance.gold COMMENT 'Presentation/BI data';

-- Set default privileges at the catalog level
GRANT USE CATALOG ON CATALOG finance TO `data_engineers`, `bi_analysts`;
GRANT CREATE, USE SCHEMA ON CATALOG finance TO `data_engineers`;

PySpark:

spark.sql("CREATE CATALOG IF NOT EXISTS finance MANAGED LOCATION 's3://finance-managed/'")
for sch, cmt in [("bronze", "Raw ingested data"), ("silver", "Conformed data"), ("gold", "Presentation data")]:
    spark.sql(f"CREATE SCHEMA IF NOT EXISTS finance.{sch} COMMENT '{cmt}'")

# Example privilege grants
spark.sql("GRANT USE CATALOG ON CATALOG finance TO `data_engineers`, `bi_analysts`")
spark.sql("GRANT CREATE, USE SCHEMA ON CATALOG finance TO `data_engineers` ")
3.4 Create volumes to replace mounts/DBFS

SQL:

-- Volumes provide governed filesystems under UC
USE CATALOG finance;
USE SCHEMA bronze;

CREATE VOLUME IF NOT EXISTS raw_files
COMMENT 'Raw CSVs previously under /mnt/finance-raw';

-- Access path will be /Volumes/finance/bronze/raw_files/...

PySpark:

# Read from a former mount and write to a governed Volume
df = (spark.read
      .option("header", "true")
      .csv("/mnt/finance-raw/invoices/2026-01/*.csv"))

df.write.mode("overwrite").parquet("/Volumes/finance/bronze/raw_files/invoices/2026-01/")
3.5 Security policies (masking and row filters)

SQL:

-- Masking policy for PII
CREATE MASKING POLICY pii_mask AS (val STRING) RETURNS STRING ->
  CASE
    WHEN is_account_group_member('pii_readers') THEN val
    ELSE '***'
  END;

-- Row filter policy for region scoping
CREATE ROW FILTER region_filter
AS (country STRING) RETURNS BOOL ->
  (is_account_group_member('emea_analysts') AND country IN ('DE','FR','ES','IT'))
  OR
  (is_account_group_member('na_analysts') AND country IN ('US','CA'));

Apply policies:

ALTER TABLE finance.silver.customers
ALTER COLUMN email SET MASKING POLICY pii_mask;

ALTER TABLE finance.gold.sales
SET ROW FILTER region_filter ON (country);

4) Register and migrate data

You have two main patterns:

4.1 Register existing Delta tables (external)

SQL:

-- Register an existing Delta table stored under s3://finance-curated/...
USE CATALOG finance;
USE SCHEMA silver;

CREATE TABLE IF NOT EXISTS transactions
USING DELTA
LOCATION 's3://finance-curated/delta/transactions'
COMMENT 'Finance transactions curated delta table';

-- Migrate grants (example)
GRANT SELECT ON TABLE transactions TO `bi_analysts`;
GRANT SELECT, MODIFY ON TABLE transactions TO `data_engineers`;

PySpark:

spark.sql("USE CATALOG finance")
spark.sql("USE SCHEMA silver")

spark.sql("""
CREATE TABLE IF NOT EXISTS transactions
USING DELTA
LOCATION 's3://finance-curated/delta/transactions'
COMMENT 'Finance transactions curated delta table'
""")

Note: The external location (el_finance_curated) must govern the root path for UC to allow reads/writes without bypass.

4.2 Shallow clone to UC managed/external paths

Shallow clones copy metadata quickly and reference existing data files.

SQL:

-- Create a managed target table in UC with a shallow clone
CREATE TABLE finance.silver.customers_shallow CLONE hive_metastore.finance_customers;

-- Alternatively, shallow clone to an external path and register that path
CREATE TABLE finance.silver.customers_ext
DEEP CLONE hive_metastore.finance_customers
LOCATION 's3://finance-curated/delta/customers';

PySpark:

spark.sql("""
CREATE TABLE finance.silver.customers_shallow CLONE hive_metastore.finance_customers
""")

Validate row counts and samples before switching dependencies.

4.3 Permissions mapping with code

Define your mapping as data (YAML or JSON), then apply via a script. Example YAML:

catalogs:
  finance:
    grants:
      - privilege: "USE CATALOG"
        to: ["data_engineers", "bi_analysts"]
    schemas:
      silver:
        grants:
          - privilege: "USE SCHEMA"
            to: ["data_engineers", "bi_analysts"]
          - privilege: "CREATE"
            to: ["data_engineers"]
        tables:
          - name: "transactions"
            grants:
              - privilege: "SELECT"
                to: ["bi_analysts"]
              - privilege: "SELECT, MODIFY"
                to: ["data_engineers"]

PySpark script to apply grants:

import yaml

mapping = yaml.safe_load("""
catalogs:
  finance:
    grants:
      - privilege: "USE CATALOG"
        to: ["data_engineers", "bi_analysts"]
    schemas:
      silver:
        grants:
          - privilege: "USE SCHEMA"
            to: ["data_engineers", "bi_analysts"]
          - privilege: "CREATE"
            to: ["data_engineers"]
        tables:
          - name: "transactions"
            grants:
              - privilege: "SELECT"
                to: ["bi_analysts"]
              - privilege: "SELECT, MODIFY"
                to: ["data_engineers"]
""")

for cat, cat_cfg in mapping.get("catalogs", {}).items():
    for g in cat_cfg.get("grants", []):
        priv = g["privilege"]
        for principal in g["to"]:
            spark.sql(f"GRANT {priv} ON CATALOG {cat} TO `{principal}`")

    for sch, sch_cfg in cat_cfg.get("schemas", {}).items():
        for g in sch_cfg.get("grants", []):
            priv = g["privilege"]
            for principal in g["to"]:
                spark.sql(f"GRANT {priv} ON SCHEMA {cat}.{sch} TO `{principal}`")

        for tbl in sch_cfg.get("tables", []):
            tname = tbl["name"]
            for g in tbl.get("grants", []):
                priv = g["privilege"]
                for principal in g["to"]:
                    spark.sql(f"GRANT {priv} ON TABLE {cat}.{sch}.{tname} TO `{principal}`")

Validate grants using information schema:

SELECT table_catalog, table_schema, table_name, grantee, privilege_type
FROM system.information_schema.table_privileges
WHERE table_catalog = 'finance' AND table_schema = 'silver';

5) Update compute, jobs, and BI

The two biggest sources of breakage are two-part names and non-UC compute.

5.1 Use UC-capable compute and access modes

Cluster policy snippet (JSON) to enforce UC:

{
  "access_mode": {
    "type": "fixed",
    "value": "SINGLE_USER"
  },
  "spark_conf.spark.databricks.acl.dfAclsEnabled": {
    "type": "fixed",
    "value": "true"
  }
}
5.2 Adopt three-level names in code

SQL:

-- Set current catalog/schema at session start
USE CATALOG finance;
USE SCHEMA silver;

-- Always reference 3-level names in shared environments
SELECT count(*) FROM finance.silver.transactions;

PySpark:

spark.sql("USE CATALOG finance")
spark.sql("USE SCHEMA silver")

df = spark.table("finance.silver.transactions")  # 3-part name resolved via current catalog/schema
df.groupBy("merchant_id").count().write.mode("overwrite").saveAsTable("finance.gold.merchant_sales")

Avoid direct cloud paths in production code. Use tables or UC volumes.

5.3 Replace mounts with volumes or external locations

Old:

df = spark.read.parquet("/mnt/finance-raw/events/")

New:

# Volumes path
df = spark.read.parquet("/Volumes/finance/bronze/raw_files/events/")

# Or register as a table over an external location
spark.sql("""
CREATE TABLE finance.bronze.events
USING PARQUET
LOCATION 's3://finance-raw/events/'
""")
5.4 Update BI connections

Example SQL for BI:

USE CATALOG finance;
USE SCHEMA gold;

SELECT order_date, total_amount
FROM finance.gold.daily_revenue
WHERE order_date >= current_date() - INTERVAL 30 DAY;
5.5 Jobs: migrate in-place with minimal code churn

Example job task code (PySpark):

spark.sql("USE CATALOG finance")
spark.sql("USE SCHEMA silver")

staging = spark.read.table("finance.bronze.transactions_raw")
clean = staging.filter("txn_status = 'posted'") \
               .withColumnRenamed("txn_amount_usd", "amount_usd")

clean.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable("finance.silver.transactions")

6) A phased rollout that doesn’t break things

Here’s a pattern that works across organizations.

Phase 0 – Foundations (1–2 sprints)

Phase 1 – Inventory and read-only mirror (1–3 sprints)

Phase 2 – Shadow write and validation (2–4 sprints)

Phase 3 – Cutover window (hours to a day)

Phase 4 – Decommission HMS (1–2 sprints, controlled)

Practical implications: what to do differently starting now

How to think about it going forward

Appendix: end-to-end example (putting it together)

SQL (admin session):

-- 1) Storage and locations
CREATE STORAGE CREDENTIAL sc_finance_iam
WITH ROLE ARN 'arn:aws:iam::123456789012:role/databricks-finance-access';
CREATE EXTERNAL LOCATION el_finance_raw
URL 's3://finance-raw/'
WITH STORAGE CREDENTIAL sc_finance_iam;
CREATE EXTERNAL LOCATION el_finance_curated
URL 's3://finance-curated/'
WITH STORAGE CREDENTIAL sc_finance_iam;

-- 2) Catalog and schemas
CREATE CATALOG finance MANAGED LOCATION 's3://finance-managed/';
CREATE SCHEMA finance.bronze;
CREATE SCHEMA finance.silver;
CREATE SCHEMA finance.gold;

-- 3) Volumes for files
USE CATALOG finance;
USE SCHEMA bronze;
CREATE VOLUME raw_files;

-- 4) Register existing Delta table
USE SCHEMA silver;
CREATE TABLE transactions
USING DELTA
LOCATION 's3://finance-curated/delta/transactions';

-- 5) Policies and grants
CREATE MASKING POLICY pii_mask AS (val STRING) RETURNS STRING ->
  CASE WHEN is_account_group_member('pii_readers') THEN val ELSE '***' END;

ALTER TABLE finance.silver.customers
ALTER COLUMN email SET MASKING POLICY pii_mask;

GRANT USE CATALOG ON CATALOG finance TO `data_engineers`, `bi_analysts`;
GRANT USE SCHEMA ON SCHEMA finance.silver TO `data_engineers`, `bi_analysts`;
GRANT SELECT ON TABLE finance.silver.transactions TO `bi_analysts`;

PySpark (pipeline task):

# Configure Unity Catalog context
spark.sql("USE CATALOG finance")
spark.sql("USE SCHEMA silver")

# Ingest from a governed Volume (replacing /mnt)
raw = spark.read.parquet("/Volumes/finance/bronze/raw_files/transactions/2026-01/")

# Transform and write to UC table
clean = (raw.filter("status = 'posted'")
             .withColumnRenamed("amount_usd", "amount"))

clean.write.mode("append").saveAsTable("finance.silver.transactions")

# Validate counts
cnt = spark.table("finance.silver.transactions").count()
print(f"finance.silver.transactions row count: {cnt}")

BI query (SQL):

USE CATALOG finance;
USE SCHEMA gold;

SELECT merchant_id, SUM(amount) AS total_amount
FROM finance.silver.transactions
GROUP BY merchant_id
ORDER BY total_amount DESC
LIMIT 100;

Actionable next steps

For readers who want to explore this topic hands-on, BrickNotes covers it in the Unity Catalog chapter.