1.2k Get started

What Is a Data Lake? Definition, Architecture & Data Lake vs Data Warehouse

John Smith · Jun 24, 2026

Three years ago a team pointed every pipeline they had at a cheap S3 bucket: clickstream JSON, database dumps, third-party CSVs, application logs, model outputs. The promise was “keep everything, model it later.” Today that bucket holds 400 terabytes, nobody remembers what prod_v2_final/ contains, two folders both claim to hold “users,” and the data scientists have quietly gone back to asking the data engineering team for extracts. The bucket became a data swamp — which is the most common way a data lake fails, and the fastest way to understand what one actually is. This glossary entry defines the data lake, walks through its schema-on-read architecture, contrasts it with a data warehouse, and explains why AI agents struggle on raw lake storage without a layer of context on top.

Disclosure: Datus is a data engineering agent platform. This article explains the data lake as a general concept, referencing Datus alongside other tools and architectures in the category. See the end for more detail.

TL;DR

  • A data lake is centralized object storage that holds raw data in its original format — structured, semi-structured, and unstructured — applying schema only when the data is read (schema-on-read).
  • Its advantages are low cost and total flexibility: you store anything, including data you cannot yet model, which makes lakes the default landing zone for ML and exploratory analytics.
  • Its risk is the data swamp — without catalogs, zones, and governance, a lake degrades into an unsearchable dump that no one trusts.
  • Data lake vs data warehouse is not either/or: a lake optimizes for cheap, flexible ingest; a warehouse optimizes for governed, fast BI — and the lakehouse exists to get both.
  • AI agents and text-to-SQL stumble on lakes because they see file paths, not table semantics — missing partitions, formats, and which zone is trustworthy — so lake context has to be made explicit.

1. Data lake: a working definition

The data lake idea arrived with big data and cheap cloud object storage. Hadoop’s HDFS first made it economical to dump raw files at scale; Amazon S3 and its peers (Azure Data Lake Storage, Google Cloud Storage) then made it nearly free and effectively infinite. The motivating insight was a reaction against the warehouse: modeling data up front is slow and throws away anything that does not fit the schema, but storage had become cheap enough that you could simply keep everything in its raw form and decide what it means later.

That deferred decision is the defining trait: schema-on-read. A lake imposes no structure when data lands. A JSON event, a Parquet export, a CSV, and a folder of images all sit side by side as files. Structure is applied only when a query engine reads them — Spark, Trino, Athena, or similar — interpreting the bytes against a schema you supply at read time. This is the exact inverse of a warehouse’s schema-on-write, and it is the source of both the lake’s flexibility and its tendency to rot.

A useful working definition:

A data lake is a centralized repository, typically on object storage, that stores raw data in its native format — structured, semi-structured, and unstructured — at low cost and large scale, deferring schema and structure to read time rather than enforcing them on ingest.

The boundaries matter as much as the definition. A data lake is not simply “a folder of files” — at scale it requires a catalog (such as a Hive Metastore or AWS Glue Data Catalog) so engines know what tables the files represent. It is not a database: there are usually no ACID transactions across files, no enforced types, and no guarantees that two writers will not corrupt each other. And a lake without governance is not an asset — it is a liability, because storing data you cannot find or trust costs money and creates risk without producing value.

2. Data lake vs data warehouse

This is the comparison the term is most often searched alongside, so it deserves a direct answer. The two are not competitors so much as opposite ends of a flexibility-versus-governance trade-off.

Dimension Data lake Data warehouse
Schema Schema-on-read, applied at query Schema-on-write, enforced on load
Data shape Raw: structured, semi-structured, unstructured Structured, cleaned, modeled
Cost Lowest — cheap object storage Higher — managed platform + compute
Primary users Data scientists, ML engineers Analysts, BI, finance
Transactions Usually none across files Full ACID
Governance Optional, must be added Built in, enforced
Best at Cheap retention, exploration, ML features Certified, fast, repeatable reporting

A data warehouse cleans and models data before it lands so that analytical SQL is fast and every consumer gets the same governed answer — at the cost of flexibility and slower onboarding of new sources. A data lake makes the opposite bet: land everything cheaply and unstructured, and pay the structuring cost later, per query, only for the data you actually use. Neither is “better”; they fail in mirror-image ways. Force raw exploratory data into a rigid warehouse and onboarding grinds to a quarter-long project. Force certified financial reporting onto an ungoverned lake and you get the swamp.

This tension is exactly why the lakehouse emerged — it layers warehouse-style table semantics (ACID, schema enforcement, time travel) onto lake storage through open formats, trying to keep lake economics while adding warehouse trust. Most organizations past a certain scale end up running some blend of all three rather than picking a single winner.

3. Inside a data lake: zones and formats

A healthy lake is not an undifferentiated pool — it is zoned. The most common convention progresses data through increasing levels of refinement, which is also the discipline that keeps a lake from becoming a swamp.

  • Raw (landing) zone — data exactly as it arrived, immutable, append-only; the system of record for “what the source actually sent.”
  • Cleaned (staging) zone — deduplicated, typed, standardized keys; still flexible but trustworthy enough to build on.
  • Curated (serving) zone — modeled, aggregated datasets that feed dashboards, features, and downstream products.

This raw → cleaned → curated progression is the same idea that the lakehouse world formalized as the medallion architecture (Bronze, Silver, Gold). The zones exist to answer a question paths alone cannot: for any given dataset, how processed and trustworthy is this? An ML engineer building features wants the cleaned zone; an analyst publishing a KPI wants the curated zone; a data engineer debugging an ingest issue goes back to raw.

File format is the other half of lake architecture, and the choice has outsized consequences. Storing analytics data as CSV or raw JSON is convenient but slow and expensive to scan; the de facto standard for lake analytics is Apache Parquet, a columnar format with compression and predicate pushdown so a query reads only the columns and row groups it needs. Partitioning — laying files out by, say, date=2026-06-24/region=us/ — lets engines skip irrelevant data entirely. Get format and partitioning right and a lake query is fast and cheap; get them wrong and the same query scans terabytes and times out. None of this is enforced by the storage layer, which is precisely why lakes demand more engineering discipline than warehouses do.

4. The data swamp problem

The data swamp is not a rare edge case; it is the default end state of a lake left ungoverned, and understanding it is the clearest lens on what a lake needs to succeed. The progression is almost always the same. Ingestion is easy, so everything gets dumped in. Because schema is deferred, nobody is forced to document what a dataset means. Folders multiply with names like final, final_v2, and temp_do_not_delete. Two teams independently land overlapping “users” datasets at different grains. A year later, finding the right table requires tribal knowledge, and the people who held it have left.

The fix is not technical heroics — it is the governance the warehouse got for free and the lake makes optional. A working lake needs a catalog so datasets are searchable with owners and descriptions; zones so processing state is explicit; lineage so you can trace where a curated table came from; and quality checks so a downstream consumer knows whether a table is fit to use. A data catalog is the single highest-leverage addition: it is the difference between a lake you can query and a swamp you can only guess at.

Concretely, picture a data scientist asked for “weekly active users by country.” In a swamp, they grep through s3://datalake/ paths, find three plausible candidates, pick one based on which folder looks freshest, and ship a number that silently double-counts because the dataset is at the session grain, not the user grain. In a governed lake, they search a catalog, find one curated wau_by_country table with a documented grain and an owner, and get the right answer in minutes. Same storage, same files — the difference is entirely the context layer wrapped around them.

5. Data lakes and AI agents

Text-to-SQL and data engineering agents inherit every weakness of the lake they sit on, amplified. Where a warehouse at least hands an agent a typed catalog of tables, a raw lake hands it file paths and leaves it to guess the rest — which partition column to filter on, whether events_v2/ supersedes events/, whether a folder is raw or curated. The model can produce syntactically valid SQL against the wrong files and present a confident, wrong answer, because nothing in the storage layer told it which zone to trust.

The gap is the same one humans hit, made explicit:

Context gap on a lake Symptom What the agent needs
Path vs table SQL targets a raw s3:// prefix, not a registered table Catalog entries mapping files to tables
Zone ambiguity Reads raw landing data as if it were curated Zone tags on every dataset
Format / partition blindness Full-table scan; query times out Partition keys and required filters
Schema drift A column type changed silently upstream Schema versioning and drift checks

This is the layer Datus is built for. Rather than treating the lake as a pile of paths, it reads lake metadata through a catalog service, builds a navigable Context Tree from the table structures, and attaches a semantic model that records which zone a dataset lives in, its grain, and its partition layout. Generated SQL is grounded in that structure plus vetted reference queries, so the agent targets registered curated tables instead of guessing at raw prefixes.

The realistic caveat is that an agent cannot manufacture context a lake never captured. If datasets are unregistered and zones undocumented, the agent inherits the swamp — the same way a new analyst would. The value shows up only once the lake has the catalog, zones, and definitions that turn raw storage into something queryable; the agent makes that context far more useful, but it does not replace the work of creating it.

6. When a data lake is the right choice

A data lake earns its place when flexibility and cost at scale matter more than up-front structure. It is the wrong default when your needs are entirely structured and governed — being honest about that boundary is part of using a lake well.

A data lake fits when:

  • You need to retain large volumes of raw, semi-structured, or unstructured data — logs, events, clickstream, images — cheaply.
  • ML and data science teams need direct access to raw data to engineer their own features.
  • You want to capture data now and decide how to model it later, without losing anything that does not fit a schema today.

Lean on a warehouse or lakehouse instead when:

  • Certified, fast, repeatable BI over modeled data is the primary job — that is a data warehouse strength.
  • You want warehouse-grade ACID and governance without giving up open storage — that is the lakehouse pitch.
  • Domain teams should own and serve their own data products rather than dumping into one shared bucket — that is the data mesh organizational model.

In most real stacks the lake is not the whole architecture but the foundation of it: a cheap, flexible base where data lands and is refined, with a warehouse or lakehouse serving the certified layer on top, and a catalog plus semantic layer making the whole thing navigable for analysts and agents alike. The lake’s job is to make sure you never have to throw data away; the layers above it make sure you can actually use what you kept.

Conclusion

A data lake is best understood as cheap, flexible, schema-on-read storage for raw data — its strength is that it keeps everything, and its danger is that “everything, ungoverned” becomes a swamp no one trusts. The data-lake-versus-data-warehouse question is rarely a real either/or: the lake optimizes for flexible, low-cost retention and exploration, the warehouse for governed, fast reporting, and the lakehouse tries to fuse the two. For AI agents, the lake’s challenge is not generating SQL but knowing which files are a table, which zone is trustworthy, and how the data is partitioned — context the storage layer never supplies, and exactly what separates a queryable lake from a swamp, and demo-grade text-to-SQL from production-grade data engineering agents.

Frequently asked questions

What is the difference between a data lake and a data warehouse?

A data lake stores raw data in its native format on cheap object storage and applies schema only when the data is read; a data warehouse cleans and models data before loading it and enforces schema on write. The lake optimizes for flexibility and cost — including data you cannot yet model — while the warehouse optimizes for governed, fast, repeatable analytics. Most mature organizations run both, often bridged by a lakehouse layer.

Is a data lake just a folder of files in S3?

At the smallest scale it can look like that, but a real data lake is more: it needs a catalog that maps files to tables, zones that separate raw from curated data, and governance so datasets are searchable and trustworthy. A bucket of files with none of that is how a lake becomes a data swamp — technically full of data, practically unusable.

What is a data swamp?

A data swamp is a data lake that has degraded because data was ingested without catalogs, zones, ownership, or quality controls. Datasets pile up with cryptic names and overlapping meanings until no one can reliably find or trust anything. The cure is the governance a warehouse enforces by default and a lake makes optional: a catalog, clear zones, lineage, and quality checks.

Do I need a data lake if I already have a data warehouse?

Only if you have data the warehouse handles poorly. If everything you analyze is structured and fits a governed model, a warehouse alone is simpler. You add a lake when you need to cheaply retain raw logs, events, or files, or when ML and data science teams need direct access to unmodeled data. Many teams adopt a lakehouse precisely to avoid running two separate systems.

Can AI agents query a data lake directly?

They can, but accuracy depends entirely on the context wrapped around the raw storage. Without a catalog mapping files to tables, zone tags, and partition metadata, an agent will guess at paths and can return confident, wrong answers from the wrong files. With that context injected, agents target registered, curated tables reliably. The bottleneck is the lake’s metadata layer, not the model.

Disclosure: Datus is a data engineering agent platform. This glossary entry explains the data lake as a general concept and how cross-stack agents approach lake context — alongside other tools and architectures in the category.

Try Studio free ★ Star on GitHub