Data Warehouse
A central, query-optimized store for structured analytical data. Schema is defined up front and data is loaded in cleaned form for BI and reporting.
Read the full guide →Glossary
Data engineering glossary
Plain-language definitions of the 47 concepts Datus agents work with day to day — from semantic layer and lakehouse to schema linking, MCP, and RAG. One page, no fluff.
Architecture
7 termsData Lake
Object storage that holds raw files — JSON, CSV, Parquet, logs — in their original form. Cheap and flexible, but requires discipline to stay queryable.
Read the full guide →Lakehouse
A hybrid that puts warehouse-style table semantics (ACID, schema, time travel) directly on top of a data lake via open formats like Iceberg, Delta, or Hudi.
Read the full guide →Data Mesh
An organizational pattern where domain teams own their data as products, instead of a central team owning one monolithic warehouse.
Read the full guide →Data Fabric
A metadata-driven layer that stitches together distributed data sources so they can be queried and governed as if they were one system.
Medallion Architecture
A layered convention — Bronze (raw), Silver (cleaned), Gold (aggregated) — popularized by Databricks for incrementally refining lakehouse data.
Read the full guide →Lambda vs Kappa
Two streaming architectures. Lambda runs batch and streaming pipelines in parallel; Kappa treats everything as a single streaming pipeline and replays history when needed.
Modeling
7 termsSemantic Layer
A shared definition of business entities and metrics (revenue, active user, churn) that sits between raw tables and consumers. Ensures every dashboard, notebook, and AI agent computes the same number the same way.
Read the full guide →Metric Layer
A narrower form of the semantic layer focused specifically on metric definitions — typically expressed in YAML or a DSL like MetricFlow or Cube.
Read the full guide →Dimensional Modeling
Kimball-style design that splits data into fact tables (events, measurements) and dimension tables (who/what/where). Star and snowflake schemas are its two main shapes.
Star Schema
A fact table surrounded by denormalized dimension tables. Simple, fast for BI, and the most common warehouse layout.
Slowly Changing Dimensions
Patterns for tracking how dimension values change over time. Type 1 overwrites, Type 2 keeps history with valid-from/valid-to columns, Type 3 keeps a previous-value column.
One Big Table (OBT)
A modeling style that pre-joins facts and dimensions into a single wide table. Trades storage and flexibility for query simplicity and speed on columnar engines.
Data Vault
A modeling approach using hubs (business keys), links (relationships), and satellites (descriptive attributes). Optimized for auditability and frequent schema change.
Storage & Formats
7 termsColumnar Storage
Storing data by column instead of by row, so analytical queries that touch a few fields over millions of rows only read what they need.
Parquet
An open columnar file format with compression and predicate pushdown. The de facto interchange format between warehouses, lakes, and processing engines.
Apache Iceberg
An open table format that adds schema evolution, hidden partitioning, and snapshot isolation on top of Parquet files in object storage.
Read the full guide →Delta Lake
A table format from Databricks that layers an ACID transaction log on Parquet, enabling MERGE, time travel, and streaming reads on a data lake.
Apache Hudi
An open table format focused on upserts and incremental processing — designed for use cases where records change frequently after they land.
Read the full guide →Lakehouse Catalog
The engine-facing metadata service that maps table names to their schema and file locations so query engines can run SQL over a lakehouse. Implementations include the Hive Metastore, AWS Glue, Databricks Unity Catalog, Apache Polaris, and Snowflake Horizon.
Read the full guide →OLAP vs OLTP
OLTP systems (Postgres, MySQL) optimize for many small reads and writes from applications. OLAP systems (Snowflake, ClickHouse, BigQuery) optimize for large scans and aggregations across history.
Processing
7 termsETL vs ELT
ETL transforms data before loading it into the warehouse; ELT loads raw data first and transforms it inside the warehouse using SQL. ELT dominates the modern stack thanks to cheap warehouse compute.
Batch vs Streaming
Batch jobs run on a schedule over a chunk of data; streaming jobs process events continuously as they arrive. Most platforms now mix both.
Change Data Capture (CDC)
Streaming row-level inserts, updates, and deletes out of an operational database in near real time, usually by reading its transaction log.
Read the full guide →Backfill
Re-running a pipeline over historical data — typically after a logic change, a schema fix, or to populate a new column for past dates.
Idempotency
A property of a pipeline step where running it twice produces the same result as running it once. Critical for safe retries and backfills.
Materialized View
A query whose result is physically stored and kept fresh by the engine. Speeds up repeated reads at the cost of storage and write overhead.
dbt
A framework for defining transformations as version-controlled SQL models that run inside the warehouse. Now the standard for the T in ELT.
Governance & Quality
6 termsData Catalog
A searchable inventory of tables, columns, owners, and documentation across the data platform. Answers “what data do we have and where does it live?”
Read the full guide →Data Contract
A machine-checked agreement between a data producer and its consumers, specifying schema, semantics, freshness, and ownership of a dataset.
Read the full guide →Data Lineage
The graph of how data flows from source systems through transformations to final tables and dashboards. Used for impact analysis and debugging.
PII & Data Masking
Personally Identifiable Information — names, emails, IDs — that must be protected. Masking replaces or hashes these values so analysts can work safely.
RBAC
Role-Based Access Control. Permissions are granted to roles (analyst, engineer, exec) and users inherit access by being assigned a role.
Data Quality
The set of checks — freshness, completeness, uniqueness, validity, distribution — that confirm a table is fit to use before it powers a decision.
AI & Agents
7 termsText-to-SQL
Generating SQL from a natural-language question grounded in a real schema. Quality depends heavily on schema linking, business context, and feedback loops.
Read the full guide →Schema Linking
The step where the model figures out which tables and columns a question is actually about, before any SQL is written. Often the single biggest accuracy lever.
Read the full guide →Retrieval-Augmented Generation (RAG)
Pulling relevant context — table docs, prior queries, glossary entries — into the model's prompt at query time, instead of relying on what it memorized during training.
Read the full guide →Model Context Protocol (MCP)
An open protocol for exposing tools, data, and context to LLM clients like Claude, Cursor, and IDEs. Lets one server power many AI front ends.
Read the full guide →Embedding
A numerical vector representation of text (or a table, or a query) that places semantically similar items near each other. The backbone of vector search.
Vector Search
Finding the items whose embeddings are closest to a query embedding. Used to retrieve relevant tables, examples, and documentation for AI agents.
Data Engineering Agent
An LLM-driven system that plans and executes data workflows end-to-end — schema discovery, SQL generation, validation, and iteration — instead of just autocompleting a single query.
Read the full guide →Observability
6 termsData Observability
Continuous monitoring of the health of tables and pipelines across five pillars: freshness, volume, schema, distribution, and lineage.
Freshness
How recently a table was updated relative to its expected cadence. A daily table that hasn't moved in 36 hours is a freshness incident.
Volume Checks
Alerts when a table's row count for a period falls far outside its historical range — usually the first sign that an upstream job partially failed.
Schema Drift
An unannounced change to a column's type, name, or presence. Often breaks downstream models silently until someone reads a NULL chart.
Anomaly Detection
Statistical or ML-based checks that flag unexpected shifts in a metric or distribution, instead of relying on hand-written thresholds.
Data SLA / SLO
An explicit promise about a dataset — e.g. “this table is fresh by 6am UTC 99% of days.” Borrowed from software reliability practice.
From definitions to a working agent
Datus turns these concepts into an evolvable Context Engine — so your data engineering agent understands your warehouse, not just the words.