ai_parse_document(): turn a PDF into a governed table with a single SQL statement

Extracting data from documents has always been one of the most thankless parts of data engineering. Invoices in PDF, contracts in DOCX, reports with tables inside tables, scanned images. The traditional path stitches together an OCR service, a Python layout library, a separate step to reconstruct tables and a good amount of glue code to tie it all together. Each piece is a point of failure, and any change in the document format brings the pipeline down.

Databricks' ai_parse_document() proposes a different approach: collapse that entire flow into a single declarative SQL statement, with the result landing straight into a governed table in Unity Catalog.

What the function does

ai_parse_document() takes the binary content of a document and returns its structure in a semi-structured format (VARIANT). It reads PDF, JPG, PNG, DOCX and PPTX and delivers:

• Text with reading order preserved;
• Tables exactly as they appear, including merged cells and nested structures;
• Figures and diagrams described automatically with AI-generated captions;
• Spatial metadata and bounding boxes, useful for citation, auditing and human validation.

All of this without provisioning a single OCR server or writing any parser.

The step by step

1. Read the files from the Volume

Put the documents in a Unity Catalog Volume and read them as binary with READ_FILES:

SELECT path,
       ai_parse_document(content) AS doc
FROM READ_FILES(
  '/Volumes/main/raw/docs',
  format => 'binaryFile'
);

In a single call, each document becomes a row with the doc column containing the full structure.

2. Materialize the result

Write the output to a Delta table so you can query and reprocess without repeating the parsing:

CREATE TABLE main.silver.docs_parsed AS
SELECT path,
       ai_parse_document(content) AS doc
FROM READ_FILES('/Volumes/main/raw/docs',
                 format => 'binaryFile');

3. Navigate the structure

Since doc is a VARIANT, you access the fields with path notation and extract only what you need — for example, the text of each page or the detected tables:

SELECT path,
       doc:document:pages AS pages,
       doc:document:elements AS elements
FROM main.silver.docs_parsed;

From there it's plain SQL: explode the pages, filter by element type, join with your dimensions. The extracted content feeds directly into RAG, Agent Bricks and AI/BI, because it's already in the lakehouse.

Why this matters for data engineering

Fewer moving parts. A SELECT replaces a dedicated parsing pipeline. Less code, fewer dependencies, less on-call.

Incremental by default. Integrated with Spark Declarative Pipelines, processing is incremental: new documents arriving in the Volume come in on their own, without reprocessing history.

Real governance. The result is a Unity Catalog table like any other — with permissions, end-to-end lineage and auditing in the same place as the rest of your data. No sensitive data leaking to an external service without a trace.

Competitive cost. Databricks positions the system with quality comparable to the best offerings on the market at a cost 3 to 5 times lower — relevant when the document volume scales.

When (and when not) to use it

It's a natural choice when the documents already live, or can live, in a Unity Catalog Volume and the destination is the lakehouse itself: extracting invoices, contracts, regulatory reports and bases for enterprise chatbots.

Watch the current limits: a maximum of 500 pages and 100 MB per document — above that, the function returns an error. For very large documents, split them into parts before processing. And, as with any AI-assisted extraction, keep a validation step for the higher-risk cases: the bounding boxes exist precisely for that.

Conclusion

ai_parse_document() moves document extraction from "an integration project" to "one more line of SQL". For data teams, that means delivering document data with the same governance, lineage and simplicity as any lakehouse table — and freeing up time that used to be spent maintaining fragile parsing pipelines.

If you work with Databricks, it's worth a quick test with a sample Volume this very week.