Azure Databricks Architecture Diagram: A Complete Guide to Modern Data Platforms

High-Level Azure Databricks Architecture

Azure Databricks operates out of two primary planes: the control plane and the compute plane.

Control Plane

The control plane includes the backend services that Azure Databricks manages in your Azure Databricks account. The web application lives in the control plane, providing the user interface for notebooks, jobs, and cluster management.

Compute Plane

The compute plane is where your data is processed. There are two types of compute planes:

• Serverless Compute Plane: Serverless compute resources run in a compute layer within your Azure Databricks account. Serverless compute runs within a network boundary for the workspace, with isolation between customer workspaces.
• Classic Compute Plane: Classic compute resources run in your Azure subscription. New compute resources are created within each workspace's virtual network in your Azure subscription. This gives you natural isolation because the compute runs in your own subscription.

Workspace Storage Account

Each Azure Databricks workspace has an associated storage account in your Azure subscription. The workspace storage account contains:

• Workspace system data: Notebook revisions, job run details, command results, and Spark logs
• DBFS (Databricks File System): A distributed file system in Azure Databricks environments—storing data using DBFS is a deprecated pattern and not recommended
• Unity Catalog workspace catalog: If your workspace was enabled for Unity Catalog, the workspace storage account contains the default workspace catalog

Azure Databricks Reference Architecture: Core Components

The Azure reference architecture for Databricks shows the following Azure-specific services for ingesting, storage, serving, and analysis:

Data Sources (Source)

There are three ways to integrate external data into the Data Intelligence Platform:

1. ETL: The platform integrates with systems providing semi-structured and unstructured data (sensors, IoT devices, media, files, logs) as well as structured data from relational databases.
2. Lakehouse Federation: SQL sources like relational databases can be integrated into Unity Catalog without ETL. Queries are pushed down to the source system, and data is governed by Unity Catalog.
3. Catalog Federation: Hive Metastore catalogs can be integrated into Unity Catalog, allowing Unity Catalog to control tables stored in Hive Metastore.

Ingestion (Ingest)

Data can be ingested via batch or streaming:

• Databricks Lakeflow Connect: Built-in connectors for ingestion from enterprise applications and databases, governed by Unity Catalog and powered by serverless compute.
• Auto Loader: Files delivered to cloud storage can be loaded directly.
• Partner ingest tools: For batch ingestion into Delta Lake.
• Structured Streaming: Streaming events can be ingested directly from event streaming systems like Kafka.

The Medallion Lakehouse Architecture

The medallion architecture is the standard pattern for organizing data in a lakehouse. It organizes data processing into layers to incrementally and progressively improve the structure and quality of data as it flows through each layer.

In its simplest form, the medallion architecture consists of three layers:

• Bronze Layer: Holds raw, unprocessed data as it arrives from source systems.
• Silver Layer: Contains cleaned, filtered, and validated data that has been transformed.
• Gold Layer: Stores aggregated data that's ready for business analytics and reporting.

This layered approach, combined with Delta Lake's ACID transactions, provides data reliability, schema evolution, and time travel capabilities.

Modern Analytics Architecture Data Flow

The modern analytics architecture with Azure Databricks follows this data flow:

1. Azure Databricks ingests raw streaming data from Azure Event Hubs using Lakeflow Spark Declarative Pipelines (SDP).
2. Fabric Data Factory loads raw batch data into Data Lake Storage.
3. Data Lake Storage stores all types of data—structured, unstructured, and partially structured—as well as batch and streaming data.
4. Delta Lake is the curated layer of the data lake, storing refined data in an open-source format.
5. Azure Databricks uses the medallion architecture to organize data into Bronze, Silver, and Gold layers.
6. Data scientists use the curated data for data preparation, data exploration, model preparation, and model training.
7. MLflow manages parameter, metric, and model tracking in data science code runs.

The Azure Databricks icon sits at the center of the diagram, alongside the Data Lake Storage Gen2 icon.

Unity Catalog: Centralized Data Governance

Unity Catalog adds a unified governance model to the lakehouse. It protects and audits data access and provides lineage information across downstream tables.

Key capabilities include:

• Account-level governance: Unity Catalog spans multiple workspaces, replacing the legacy workspace-level Hive metastore.
• Three-level namespace: Catalogs contain schemas, which contain tables, views, volumes, models, and functions.
• Centralized metadata management: A single place to discover and manage all data assets.

Microsoft's reference architectures show Azure Purview as the enterprise catalog to which Unity Catalog exports schema and lineage information.

Common Azure Databricks Architecture Patterns

Pattern 1: Stream Processing Pipeline

This reference architecture shows an end-to-end stream processing pipeline with four stages: ingest, process, store, and analyze and report.

Data flow:

1. Ingest: Two real-time data streams (e.g., fare and trip data from taxi cabs) publish events to Azure Event Hubs.
2. Process: Azure Databricks consumes both Event Hubs streams, correlates records, enriches data with a third dataset, and produces a unified dataset.
3. Store: Processed records are written into Azure Cosmos DB for NoSQL.
4. Analyze/Report: Fabric mirrors operational data from Azure Cosmos DB to enable analytical queries without affecting transactional performance.

Pattern 2: Data Science and Machine Learning Architecture

Azure Databricks, Delta Lake, and MLflow combine to support data science and machine learning workflows. This architecture typically includes:

• Data preparation and exploration in Databricks notebooks
• Support for SQL, Python, R, and Scala
• Integration with open-source libraries including PySpark, pandas API on Spark, pandas, and scikit-learn
• MLflow Model Registry for managing model versions
• Model deployment to Azure Machine Learning endpoints, batch endpoints, and AKS-backed Kubernetes endpoints

Pattern 3: Medallion Architecture Implementation

A practical implementation of the medallion architecture on Azure Databricks uses:

• Unity Catalog for data governance
• Azure Data Lake Gen2 for storage
• Schemas created with Spark SQL
• Data ingested from ADLS into the Bronze layer
• Transformation into Silver and Gold layers to improve data quality and accessibility

Network Architecture Options

Azure Databricks provides three network reference architectures for common requirements:

Managed Security (Default)

The default configuration provides secure networking with minimal configuration. Inbound, classic, and outbound connectivity are managed by Azure Databricks.

Enhanced Security and Compliance

Provides additional controls for regulated environments including:

• Private endpoint connectivity for inbound access
• VNet injection for classic compute plane resources
• Firewall rules for serverless compute plane access

Security Reference Architecture (SRA)

Designed for highly secure and regulated environments:

• Hub-and-spoke model for network segmentation and workload isolation
• Cross-environment connectivity simplified through the hub
• Granular control over inbound, classic, and outbound connectivity

Types of Network Connectivity

Best Practices for Azure Databricks Architecture

Data Organization

Adopt a medallion architecture (Bronze → Silver → Gold) to speed up data product delivery. Curating data by establishing a layered (or multi-hop) architecture is a critical best practice for the lakehouse, as it allows data teams to structure data according to quality levels and define roles and responsibilities per layer.

Eliminate data silos and minimize data movement.

Unity Catalog Adoption

Use Unity Catalog for centralized governance across workspaces. Manage catalogs, schemas, tables, volumes, models, and functions in a single place.

Network Architecture Selection

Choose the right network architecture based on your data governance needs and egress control policies:

• Start with Managed Security as your baseline
• Progress to Enhanced Security and Compliance as requirements grow
• Implement Security Reference Architecture (SRA) for highly secure and regulated environments

Compute Selection

Choose the right compute type for your workload:

• Serverless compute: For ingestion pipelines, ad-hoc queries, and workloads where infrastructure management overhead should be minimized.
• Classic compute: For workloads requiring custom networking configurations, VNet injection, or specific instance types.

Tools for Creating Azure Databricks Architecture Diagrams

Official Microsoft Resources

Azure Architecture Center: The primary source for Databricks reference architectures. Each reference architecture has a downloadable PDF in 11 x 17 (A3) format.

Visio files: Downloadable Visio files are available for modern analytics architecture and stream processing architectures.

AI-Powered Tools

AI Line Studio: Generates Azure Databricks architecture diagrams from natural language descriptions in seconds. Describe a Databricks architecture—"a lakehouse on Azure with Databricks, Delta Lake, Unity Catalog, ADLS Gen2, and Event Hubs streaming"—and it produces a structured diagram with official Azure icons. For data platform-specific workflows, use the dedicated AI cloud diagram generator.

Diagram-as-Code Tools

Mermaid: Create Azure Databricks architecture diagrams using code syntax that can be version-controlled in Git repositories. Microsoft's Azure Resource Visualizer skill can generate Mermaid diagrams from deployed Azure resource groups.

Summary

Azure Databricks architecture diagrams are essential for designing, documenting, and communicating modern data platforms. A well-crafted diagram shows how Databricks integrates with Azure services to build lakehouse architectures.

Key takeaways:

Best practices:

• Adopt the medallion architecture for data organization
• Use Unity Catalog for centralized governance across workspaces
• Choose the right network architecture for your security requirements
• Select appropriate compute types (serverless vs. classic) based on workload needs
• Eliminate data silos and minimize data movement

Reference architectures:

• Modern analytics architecture with Azure Databricks
• Stream processing with Databricks (ingest → process → store → analyze/report)
• Data science and machine learning with Databricks, Delta Lake, and MLflow
• Lakehouse reference architectures with medallion layers

To start building your own Azure Databricks architecture diagrams, explore the Azure architecture diagram tool for templates and examples. For automated diagram generation, try the AI cloud diagram generator to turn a Databricks description into a visual instantly. For complete data platform architecture beyond Databricks, the AI system architecture generator covers distributed and enterprise system designs.