What the new credential vending GA and managed-table Beta mean for working data engineers.
For a long time, choosing a data platform felt like signing a long-term lease. If your team picked one engine, you slowly ended up copying data into every other tool you wanted to use. A copy for the warehouse. A copy for the data science notebook. A copy for the BI layer. Each copy came with its own access policy, its own freshness problem, and its own bill.
Databricks just published an update that quietly chips away at this old pain. It is a long announcement with a simple message: Unity Catalog is opening more doors to external engines, and it is doing it without giving up on governance.
This post walks through what changed, in plain words, and where it fits in the lessons you are already learning on BricksNotes.
Three things, all in the same announcement.
External engines can now create and write to UC managed Delta tables. This is in Beta. Engines like Apache Spark, Apache Flink, and DuckDB are no longer read-only visitors. They can stand up new managed tables, write to them, and even use them as streaming sources and sinks.
Credential vending is now generally available. This is the mechanism Unity Catalog uses to hand out short-lived, scoped credentials to external engines. It now supports machine-to-machine OAuth, and credentials refresh automatically inside long-running pipelines.
Volume credential vending is now in Public Preview. The same governed access model now extends to unstructured data sitting in Unity Catalog Volumes. Think PDFs, images, video, and any file that does not fit neatly into a table. Unity Catalog now also supports a native FILE type to govern these assets.
The headline, if you want one line, is this: more engines, same governance.
If you have read the Delta Lake lesson, you already know what makes a Delta table feel calm to work with. ACID writes. Time travel. Schema evolution that does not break readers downstream.
UC managed tables sit one layer above this. Unity Catalog owns the table lifecycle, runs Predictive Optimization in the background, and uses Liquid Clustering to keep the data layout healthy without you babysitting it. The promise is real performance gains and lower storage costs, without you writing a single OPTIMIZE command on a schedule. Managed Iceberg is also now GA for teams requiring that format.
Until now, that comfort had a catch. Only Databricks engines could fully write to these managed tables. External tools could read, but not write. The new Beta closes that gap.
If you want to understand why layout and clustering matter in the first place, the Partitioning and Performance lesson is the right place to slow down.
The technical foundation under all of this is something called catalog commits. The short version: instead of every engine writing its own commit log entries independently, the catalog itself coordinates the commits.
This is what makes safe concurrent writes from different engines possible. Two writers from two engines no longer race each other. The catalog serializes them. It also makes future work like multi-statement, multi-table transactions realistic, because there is now one place that knows the truth.
If you want a calm walkthrough of the Unity Catalog model, the Unity Catalog lesson covers the three-level namespace, grants, and how managed vs external tables fit together.
Credential vending is the part most production teams will care about today.
The old pattern looked like this. You create a personal access token. You paste it into a config somewhere. The token lives a long time. It belongs to a person, not a service. When that person leaves the company, something quietly breaks at 2 AM.
The new pattern is different. The external engine asks Unity Catalog for a credential at the moment it needs to access data. Unity Catalog issues a short-lived, scoped token. The token expires quickly. For long-running jobs, the engine refreshes it automatically. Machine-to-machine OAuth means there is no human identity tangled up in the pipeline.
This is a small change in API surface and a large change in operational risk. If you have ever spent a Friday hunting down which expired token broke which job, you already understand why this matters. This evolution is part of OpenSharing, a vendor-neutral protocol for sharing data and AI assets.
The Workflows lesson and the Medallion Architecture lesson both assume some kind of credentialed access between stages. This new model is what makes that picture cleaner in production.
Tables are not the only thing data engineers care about. Modern pipelines also handle PDFs for document processing, images for vision models, and raw logs that do not deserve a schema.
Unity Catalog Volumes is the home for that kind of data. With volume credential vending in Public Preview, external clients can request the same kind of short-lived, scoped credentials to read these files. The audit trail, the access controls, and the governance model are the same as for tables.
If you have not seen Volumes in action yet, the Resources page has practical material that uses the standard Volumes path on Free Edition.
A fair question: can you try this today on Databricks Free Edition?
The concepts, yes. The Beta itself, not directly. External access to managed Delta tables requires enrolling in the preview on a workspace that has external data access enabled on the metastore. That part is not available on Free Edition.
What you can do on Free Edition is learn the layers underneath. Create managed tables. Practice OPTIMIZE, VACUUM, and time travel. Build a Bronze, Silver, Gold pipeline using Lakeflow Declarative Pipelines (formerly Delta Live Tables). Use Unity Catalog Volumes for unstructured files. When the Beta becomes general availability and reaches more workspaces, the model will already be familiar to you.
This is the same pattern we recommend across BricksNotes. Learn the system first. The product features will keep arriving.
A few practical things, ordered from easy to harder.
Stop creating personal access tokens for service workloads. If your team still mints PATs for pipelines, this announcement is a good excuse to plan the move to machine-to-machine OAuth and credential vending.
Default new tables to managed. Unless you have a specific reason to keep a table external, managed is the path with more automatic optimization and now wider engine support. Apache Iceberg v3 is also now GA on Databricks for managed tables.
Plan for attribute-based access control. The announcement signals that row and column level ABAC for external reads is on the way. Designing your tables and tags with this in mind today will save migration work later.
Watch the connector ecosystem. Delta Kernel is making it easier for any engine to support UC. Expect more tools in your stack to show up as first-class UC citizens over the next year.
This update is not a flashy launch. It is the kind of plumbing change that makes the lakehouse model more honest. One copy of the data. One place to govern it. Many engines that can do real work on it. With Databricks Lakeflow now GA, this unified ingestion and transformation is fully integrated under Unity Catalog.
That is the version of the lakehouse most data engineers actually wanted.