Select Page

ETL is dead, long live ETL (for multimodal data)

Why did ELT become the most effective pattern for structured data?

A key innovation in the past decade that unlocked the modern data stack was the decoupling of storage and compute enabled by cloud data warehouses as well as cloud data platforms like Databricks. This decoupling and the capability to scale and prioritize compute independently, supported an effective architectural pattern for getting data into the warehouse, turning the traditional ETL pattern–extract, transform, and then load–on its head.

In the new pattern, ELT or extract, load, and then transform, data engineers used simple point-to-point extract and load tools to move structured data from source systems into staging areas in the data warehouse. The transform step to make these objects useful for analytics was written in SQL that ran against these staged, raw tables in the cloud data warehouse. With the rise of analytics engineering came more rigor and more tooling, like dbt to orchestrate the transformations that would execute within the data warehouse.

The ELT pattern became more effective than ETL approaches, especially as storage became cheaper and data volumes grew, because it 1) reduced complexity, especially of the EL step, 2) it leveraged the scaled compute of the cloud data warehouse rather than requiring expensive, separate middleware that also had to scale, and 3) provided more flexibility to data engineers to easily create new analytical objects for the presentation layer without having to go back to source systems.

In recent years, it’s been interesting to watch this trend evolve and accelerate further as the cloud data warehouse continues to be unbundled: with open formats like Iceberg and Hudi handling table formats on cloud object storage, and in-process query engines like DuckDB or distributed engines like Trino as the compute layer.

The emergence of multimodal data and embeddings

As we’ve written about in a previous blog, as an industry we’ve been talking about the transformative potential of unstructured documents and data for over a decade, but this latest wave of breakthroughs has been monumental and put engineers in a very strong position to succeed. A lasting impact of this wave will be the emergence of embeddings as a first class data citizen within the ecosystem. Embeddings are dense vector-based representations of language, images, etc., which are learned in the first layers of deep neural nets–they are mathematical representations of the input data. 

As enterprise adoption converges on proprietary foundational models–with open ones likely having something to say in the future–RAG and related patterns are the primary way in which enterprises will get their data to these models. Enterprises will be less concerned with model customization via fine tuning models, as this approach tends to be challenging and costly. Therefore, embedding APIs and vector databases will continue to be important picks and shovels for multimodal data engineering.

Why will ETL be important for multimodal data?

Once chunks of text are transformed to embeddings and indexed in a vector store or search index, it’s not possible to transform or enrich the data further. This is in direct contrast to the ELT pattern for structured data, which relies on the similarity of staged, raw data and finalized, presentation data for analytics and the applicability of SQL to execute transformations after data has landed. This means that the steps of enriching chunks of data–with other metadata, such as what document it came from, the date of that document, metrics associated with that document like views, etc.–becomes a critical step that will often precede the loading step of these pipelines.

In particular, since the metrics are associated with the chunk (and sometimes its parent) RAG apps will rely upon a mapping of chunks to vectors in order to update the vector in the database, after it gets loaded–since the chunk itself is not colocated with the vector in the database.

This sequencing means that the classical ETL pattern will come back to the fore in the case of multimodal data and the processing steps that need to occur to make such data valuable for AI app development. Point-to-point, extract-and-load oriented tooling will be of limited value in this setting, as data engineers will need rich capabilities within the transform step of the pipeline, inclusive of enrichment via integrations with other enterprise data systems. The absence of SQL-based transformation, and of any DSL or API for pushing transformations down to the data stores, will directly challenge the consensus approaches that have emerged in the structured setting.

This is a primary reason that we’re incredibly excited to bring Datavolo to the market to help data engineers solve these challenges with purpose-built tooling. Our solution was built from the ground up to handle in-flow transformations for multimodal data, with a secure, scalable, and codeless developer experience. Please reach out with your feedback, we’d love your insights on the challenges your organization has faced with multimodal data! 

Top Related Posts

Custom code adds risk to the enterprise

Data teams are actively delivering new architectures to propel AI innovation at a rapid pace. In this blog, we’ll explore how Datavolo empowers these teams to accelerate while addressing the critical aspects of security, observability, and maintenance for their data...

Fueling your Chatbots with Slack

The true power of chatbots is not in how much the large language model (LLM) powering it understands. It’s the ability to provide relevant, organization-specific information to the LLM so that it can provide a natural language interface to vast amounts of data. That...

Datavolo Architecture Viewpoint

The Evolving AI Stack Datavolo is going to play in three layers of the evolving AI stack: data pipelines, orchestration, and observability & governance. The value of any stack is determined by the app layer, as we saw with Windows, iOS, and countless other...

FlowGen Improvements (already!)

In the past week, since Datavolo released its Flow Generation capability, we've witnessed fantastic adoption as users have eagerly requested flows from the Flow Generation bot. We're excited to share that we have recently upgraded our models, enhancing both the power...

Seven Strategies for Securing Data Ingest Pipelines

Introduction Information security is an elusive but essential quality of modern computer systems. Implementing secure design principles involves different techniques depending on the domain, but core concepts apply regardless of architecture, language, or layers of...

The Evolution of AI Engineering and Datavolo’s Role

Humility is the first lesson In the machine learning era of software engineering, one persistent truth has emerged: engineers are increasingly submitting to the will of the machine. A significant milestone in the transition from classical machine learning to deep...

Introducing our GenAI NiFi Flow Builder!

Hey everyone, it's been an incredible journey over the past ten years since we open-sourced Apache NiFi. Right from the beginning, our mission with NiFi was crystal clear: to make it easier for all of you to gather data from...

Field CTO Perspectives: Why Datavolo and Why Now?

Setting the Stage There are a few times in our lives when we feel the ground shifting under our feet due to seismic shifts in technology. You know these paradigm shifts are truly seismic when they lead to broader changes in society–the web, search engines, mobile, and...

GenAI/RAG is “Homecoming for NiFi”

At the enterprise level today every company is a data company and the only constant is change itself.  Generative AI is our latest compelling reminder.  Legacy systems are serving their noble purpose of ensuring the business of today is getting done while...

Multimodal AI Demands Multimodal Data Pipelines

Innovation is the driving force behind human progress and we believe in the power of technology to enable humans to push beyond the boundaries of what's possible. In this rapidly evolving landscape, staying ahead of the curve is what sets good organizations apart from...