We at Datavolo like to drink our own champagne, building internal tooling and operational workflows on top of the Datavolo Runtime, our distribution of Apache NiFi. We’ve written about several of these services, including our observability pipeline and Slack chatbots....
Data Engineering
Apache NiFi – designed for extension at scale
AI systems need data all along the spectrum of unstructured, structured, and multi-modal. The protocols by which these diverse types of data are both acquired and delivered are as varied as the data types themselves. At the same time data volumes and latency requirements grow ever stronger which demands solutions which scale down and up first – then out. In other words we need maximum efficiency, we can’t resort to remote procedure calls for every operation, and we need to support hundreds if not thousands of different components or tools in the same virtual machine.
Data Pipeline Observability is Key to Data Quality
In my recent article, What is Observability, I discussed how observability is crucial for understanding complex architectures and their interactions and dependencies between different system components. Data Observability, unlike Software Observability, aims to...
Building GenAI enterprise applications with Vectara and Datavolo
The Vectara and Datavolo integration and partnership When building GenAI apps that are meant to give users rich answers to complex questions or act as an AI assistant (chatbot), we often use Retrieval Augmented Generation (RAG) and want to ground the responses on...
Datavolo Announces Over $21M in Funding!
Datavolo Raises Over $21 Million in Funding from General Catalyst and others to Solve Multimodal Data Pipelines for AI Phoenix, AZ, April 2, 2024 – Datavolo, the leader in multimodal data pipelines for AI, announced today that it has raised over $21 million in...
Data Engineering for Advanced RAG: Small-to-Big with Pinecone, LangChain, and Datavolo
Data Engineering for Advanced RAG Datavolo helps data teams build multimodal data pipelines to support their organization’s AI initiatives. Every organization has their own private data that they need to incorporate into their AI apps, and a predominant pattern to do...
Custom code adds risk to the enterprise
Data teams are actively delivering new architectures to propel AI innovation at a rapid pace. In this blog, we’ll explore how Datavolo empowers these teams to accelerate while addressing the critical aspects of security, observability, and maintenance for their data...
Datavolo Architecture Viewpoint
The Evolving AI Stack Datavolo is going to play in three layers of the evolving AI stack: data pipelines, orchestration, and observability & governance. The value of any stack is determined by the app layer, as we saw with Windows, iOS, and countless other...
ETL is dead, long live ETL (for multimodal data)
Why did ELT become the most effective pattern for structured data? A key innovation in the past decade that unlocked the modern data stack was the decoupling of storage and compute enabled by cloud data warehouses as well as cloud data platforms like Databricks. This...
Seven Strategies for Securing Data Ingest Pipelines
Introduction Information security is an elusive but essential quality of modern computer systems. Implementing secure design principles involves different techniques depending on the domain, but core concepts apply regardless of architecture, language, or layers of...