Spotify's RAP Index Turns Data Lakes into Low-Latency Query Engines

One Dataset, Every Workload

For years, the data world has been split down the middle. You have your data lake — cheap, scalable, perfect for big analytics jobs. And you have your operational database — fast, indexed, built for serving individual records to users in real time. The problem is that most companies end up maintaining both, copying data back and forth, paying for storage twice, and praying that the pipelines stay in sync.

Spotify thinks it has found a way out. The company recently introduced Random Access Parquet, or RAP, a storage architecture that lets you run low-latency point queries directly against a data lake. No copying to a separate database. No duplicate storage. Just the same Parquet files you already have, with an index layer bolted on top.

The Pain of Point Queries in a Data Lake

Modern data lakes have become the central repository for analytics and AI workloads. That's great if you're scanning billions of rows to train a model or generate a weekly report. But it falls apart when you need to fetch a single record — say, a user's profile or a specific transaction — in under a hundred milliseconds.

Distributed query engines like Trino and BigQuery are built for analytical scans, not key-based lookups. They optimize for reading large chunks of data sequentially, not for jumping to a specific row. Even though cloud object storage like Google Cloud Storage can now deliver millisecond-level access latency, the surrounding machinery — query planning, metadata traversal, file discovery — adds enough overhead to make point queries painfully slow.

Spotify knows this pain firsthand. They store petabytes of online data in Bigtable and exabytes of data in a Google Cloud Storage-backed lake. Copying data from the lake to the operational database at that scale is not just expensive; it's becoming logistically impossible.

RAP: An External Index for Parquet

RAP attacks the problem with a simple but powerful idea: add an external index that maps query keys — like user IDs — directly to Parquet files and row positions. Instead of scanning thousands of files to find one record, the query engine first resolves the key through the index, then issues a targeted range read against object storage.

Because the index is external, it doesn't touch the immutable Parquet files. As new data lands in Apache Iceberg tables, an index builder generates append-only index fragments. The original data files stay untouched, which means your existing analytics pipelines, machine learning workflows, notebooks, and AI agents can keep using the same dataset without any changes.

One Dataset, Many Uses

The payoff is that Spotify can now serve online, latency-sensitive applications from the same data lake that powers their analytics. No more maintaining a separate copy of the data for each use case. The same dataset supports:

• Analytical processing with engines like Trino or Spark
• Machine learning pipelines that need full scans
• Interactive notebooks for data scientists
• AI agents that need quick lookups
• Online services that demand sub-100ms responses

This is a significant shift. Historically, you had to choose between the scalability of a data lake and the performance of a database. RAP aims to give you both.

Storage Layout Tricks for Faster Reads

An index alone isn't enough. Spotify also implemented several storage layout optimizations to squeeze out every bit of latency.

First, they sort data by query key. This reduces the number of files a query touches, because records with the same key tend to live in the same file or even the same block. Second, they group related records together, so a single read can fetch multiple relevant rows. Third, they interleave value columns — instead of storing each column separately, they arrange related columns side by side so that reading one record gets you all its attributes in a single sequential read.

Finally, they use covering indexes. In some cases, the index itself contains enough data to answer a query without ever touching the Parquet file. That means some point queries can be satisfied with a single range read of just a few kilobytes. The tradeoff is a modest increase in file or index size, but the reduction in storage operations is dramatic.

Secondary Indexes Without Rewriting Files

What if you need to query by a different key? Say you have buyer IDs and seller IDs. RAP supports secondary indexes that are managed at the serving layer, so you can add a new access path without rewriting the Parquet files or changing your data pipeline.

There are two types of indexes: hash-based for exact lookups and sorted for range queries. The hash index is perfect for equality checks like "get user 12345". The sorted index handles queries like "all transactions in the last hour". Both live outside the data files, so they can be updated independently.

To further improve data locality for secondary dimensions, Spotify uses techniques like Z-ordering and Hilbert curves. These layout methods cluster related data points together, reducing the number of files a secondary-index query needs to scan.

RAP in the Broader Data Landscape

Spotify isn't alone in trying to push data lake technology beyond analytics. Google Cloud recently introduced a lakehouse architecture based on Apache Iceberg that also aims to reduce data duplication while enabling operational access. But RAP takes a different approach: it adds a dedicated external index optimized for point queries, while remaining fully compatible with existing Parquet files and Iceberg tables.

The data engineering community has taken notice. Andrew Lamb sees RAP as an example of extending open data formats to support interactive workloads. In a LinkedIn discussion, Vikas Singh noted that as cloud object storage gets faster, the bottleneck shifts to query planning and metadata access — and RAP addresses that by precomputing the index.

This is part of a broader trend toward making data lakes more versatile. The idea is that you shouldn't need a separate database just to serve records to users. If you can make the lake fast enough for point queries, you can consolidate your infrastructure, reduce costs, and simplify your architecture.

Will RAP become a standard? It's still early. But it's a compelling proof point that with the right index layer, a data lake can do more than analytics. It can be the backbone for your online services too.