Why did ELT become the most effective pattern for structured data?
A key innovation in the past decade that unlocked the modern data stack was the decoupling of storage and compute enabled by cloud data warehouses as well as cloud data platforms like Databricks. This decoupling and the capability to scale and prioritize compute independently, supported an effective architectural pattern for getting data into the warehouse, turning the traditional ETL pattern–extract, transform, and then load–on its head.
In the new pattern, ELT or extract, load, and then transform, data engineers used simple point-to-point extract and load tools to move structured data from source systems into staging areas in the data warehouse. The transform step to make these objects useful for analytics was written in SQL that ran against these staged, raw tables in the cloud data warehouse. With the rise of analytics engineering came more rigor and more tooling, like dbt to orchestrate the transformations that would execute within the data warehouse.
The ELT pattern became more effective than ETL approaches, especially as storage became cheaper and data volumes grew, because it 1) reduced complexity, especially of the EL step, 2) it leveraged the scaled compute of the cloud data warehouse rather than requiring expensive, separate middleware that also had to scale, and 3) provided more flexibility to data engineers to easily create new analytical objects for the presentation layer without having to go back to source systems.
In recent years, it’s been interesting to watch this trend evolve and accelerate further as the cloud data warehouse continues to be unbundled: with open formats like Iceberg and Hudi handling table formats on cloud object storage, and in-process query engines like DuckDB or distributed engines like Trino as the compute layer.
The emergence of multimodal data and embeddings
As we’ve written about in a previous blog, as an industry we’ve been talking about the transformative potential of unstructured documents and data for over a decade, but this latest wave of breakthroughs has been monumental and put engineers in a very strong position to succeed. A lasting impact of this wave will be the emergence of embeddings as a first class data citizen within the ecosystem. Embeddings are dense vector-based representations of language, images, etc., which are learned in the first layers of deep neural nets–they are mathematical representations of the input data.
As enterprise adoption converges on proprietary foundational models–with open ones likely having something to say in the future–RAG and related patterns are the primary way in which enterprises will get their data to these models. Enterprises will be less concerned with model customization via fine tuning models, as this approach tends to be challenging and costly. Therefore, embedding APIs and vector databases will continue to be important picks and shovels for multimodal data engineering.
Why will ETL be important for multimodal data?
Once chunks of text are transformed to embeddings and indexed in a vector store or search index, it’s not possible to transform or enrich the data further. This is in direct contrast to the ELT pattern for structured data, which relies on the similarity of staged, raw data and finalized, presentation data for analytics and the applicability of SQL to execute transformations after data has landed. This means that the steps of enriching chunks of data–with other metadata, such as what document it came from, the date of that document, metrics associated with that document like views, etc.–becomes a critical step that will often precede the loading step of these pipelines.
In particular, since the metrics are associated with the chunk (and sometimes its parent) RAG apps will rely upon a mapping of chunks to vectors in order to update the vector in the database, after it gets loaded–since the chunk itself is not colocated with the vector in the database.
This sequencing means that the classical ETL pattern will come back to the fore in the case of multimodal data and the processing steps that need to occur to make such data valuable for AI app development. Point-to-point, extract-and-load oriented tooling will be of limited value in this setting, as data engineers will need rich capabilities within the transform step of the pipeline, inclusive of enrichment via integrations with other enterprise data systems. The absence of SQL-based transformation, and of any DSL or API for pushing transformations down to the data stores, will directly challenge the consensus approaches that have emerged in the structured setting.
This is a primary reason that we’re incredibly excited to bring Datavolo to the market to help data engineers solve these challenges with purpose-built tooling. Our solution was built from the ground up to handle in-flow transformations for multimodal data, with a secure, scalable, and codeless developer experience. Please reach out with your feedback, we’d love your insights on the challenges your organization has faced with multimodal data!