GenAI in the enterprise is still in its infancy.  The excitement and potential is undeniable.  However, enterprises have struggled to derive material value from GenAI and the hype surrounding this technology is waning.  We have talked with hundreds of organizations (including 80% of the Fortune 100) in the last six months and a clear picture around the state of GenAI in the enterprise has emerged – along with a clear path forward to truly transformative change within most enterprises.

Are enterprises using Generative AI?

The answer to this question is nuanced.  Most organizations will state they are using generative AI.  However, by and large, the pattern by which they are using GenAI is one of two ways.  First, software vendors have rapidly integrated GenAI capabilities into their products and enterprises using these vendors often claim, by proxy, to be using GenAI.  This is an important but undifferentiated use of AI.  Using coding co-pilots or a corporate-branded instance of ChatGPT is great for employee productivity and those that use these AI assistants effectively will have a leg up, but this incremental boost is widely available to everyone.

The next step we see is organizations that have built their own internal chatbots that have been somewhat tuned for their business or perhaps leveraging a RAG pattern to augment the chatbots with some corporate context.  This commonly looks like a chatbot where employees can ask questions about broadly available, low security, internal documents.  This is yet another incremental step forward.

There are some enterprise organizations that have indeed gone beyond these incremental steps but they are few and far between.  These organizations tend to be very R&D heavy organizations that were already significantly investing in data scientists and AI before the hype cycle of GenAI began.

What is holding enterprises back from using GenAI?

There are three primary factors holding back the use of GenAI.  These are security concerns, hallucination concerns, and the complexity of modern AI systems.

Enterprises are inherently risk averse and GenAI presents a plethora of unknowns and risks to be mitigated.  How do we securely feed data to our models?  How do we make sure people only see results they are authorized to see? How do we know what data these models were trained on? How do we ensure that our data is not being leaked to the companies making these models?  These are the reasons that most enterprises are starting with either public data sets or low-risk documents.  However, focusing on these data sets will also present low value as an outcome.  Not only is your AI only as trusted as your data (“garbage in garbage out”), but the value of your business outcomes is also correlated with the value of the data inputted.  When the business sees low value outcomes of costly investments, it typically results in decreased investment in the future.

Hallucinations (aka, the AI fabricating answers/facts) are another concern holding back GenAI in the enterprise.  LLMs are fundamentally built to “hallucinate” by generating answers in natural language which is why the models have “temperature” settings to tune the degree of creativity allowed in responses.  There have been many public examples of chatbots responding with false and potentially dangerous answers.  This presents both financial and brand risk to enterprises that could outweigh any benefits of using GenAI.  We have talked to many enterprises that are discussing natural language inputs to models but restricting the models to answer only with a very limited set of pre-defined answers.

Finally, the complexity of modern AI Systems is often an unforeseen blocker for enterprises.  It is easy for one to assume, based on using ChatGPT, that a GenAI system is as simple as interacting with a simple LLM API with very little effort.  However, the reality is that there is a tremendous amount of data engineering that needs to be done to serve data to the models in a way that they can effectively harness the data.  Whether this is fine tuning, retrieval augmented generation, or agentic AI architectures, the number one bottleneck we see to successful GenAI implementations is data engineering.

Ironically, when organizations set out to demonstrate the value of AI for their businesses they tend to overlook data engineering and focus primarily on the selection of models and prompt engineering.Our belief is that properly managed data is the true key to unlocking the value of AI.  There is an art and science that goes into the data pipelines for AI Systems.  Extraction, Parsing, Chunking, and Embeddings are all necessary and often non-trivial steps.  They are also not one-size-fits-all by nature.  For example, different data chunking strategies provide dramatically different outputs from your AI models and different data sets and different prompts require different strategies.

Sounds pretty bleak.  Is there light at the end of the tunnel?

Absolutely!  Modern AI Systems will transform the world.  Enterprises that are able to successfully harness AI will have transformational gains.  It is no coincidence that companies investing the most in AI are having outsized returns.  For example, Meta is one of the largest purchasers in the world of GPUs.  At the same time, they reduced employee headcount by 20% and increased revenue by 20%.  This is unprecedented.  Truly accomplishing more with less thanks to AI.  

What do enterprises need to do to break through the current barriers to GenAI?

We will be producing a series of content focused on addressing the challenges above but the key begins and ends with data.  Securely, observably, and continuously feeding high value data to your secured models is very achievable today.  The best way to mitigate hallucinations is by properly tuning and augmenting models with enterprise specific data (as well as controlling the temperature of output).  Finally, the data pipeline – the number one bottleneck for AI Systems – can be securely and properly managed by Datavolo.  It is not simply about the ability to rapidly and continuously ingest enterprise data to AI Systems.  It is also about providing the optionality and mechanisms to rapidly integrate with the best and latest models, architectures, and strategies in an environment that is evolving at an incredible rate. It is about empowering your data engineers with the capability to build well governed and managed pipelines but at the same time the ability to rapidly iterate and evaluate different data processing strategies to find the most optimal results for the business in the fastest time possible and at the lowest cost.  

These are the problems Datavolo is built for.  It’s why we exist.  To empower the 10x Data Engineer for AI Systems.  We would love to partner with you on your journey.