On-premises AI

83% of the world’s data is on-premises. This means your company likely has on-premises data, and for good reasons. You’ve worked hard to bring both your analytics and data application workloads in-house to achieve the security you need in your highly regulated industry, such as finance, health care, insurance, and the public sector. 

But there’s a problem. 

Like many other organizations, you’re looking to embrace AI workloads and leverage this transformative technology. But, just like your analytics workloads, you need AI workloads to run on-premises. Is this even possible? 

It is with Starburst on-premises AI—a solution designed specifically for regulated industries. 

Why many AI workloads will not be in the cloud

When you think about AI, you likely think of the cloud. This is understandable. Many generative AI use cases leverage Large Language Models (LLMs) running as Software as a Service (SaaS). Additionally, many agentic AI solutions are deployed using cloud infrastructure, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud. 

But regulated industries often run on-premises by choice, driven by mandatory regulatory requirements, security needs, or other concerns. Your workloads hold sensitive personal information, and that’s as true of AI workloads as it is of others. 

In this article, I’ll explore how AI workloads can be enacted using an on-premises environment. I’ll also discuss the key challenges and decisions involved. 

 

An overview of on-premises AI data architecture

Let’s begin at the start. To be successful, your AI data architecture should achieve three key things. Specifically, it should: 

• Provide a single foundation for data
• Be easy to use and solve real business problems across your organization
• Be governed securely

Far from being an impossible dream, on-premises AI can make these goals more straightforward to achieve. It does this by bringing AI to your data, not the other way around. This ensures data governance, including compliance and regulatory governance. That’s important, and it’s a key reason that on-premises AI matters. 

Let’s look at some ways in which an on-premises AI data architecture can be implemented.

Running on-premises Large Language Model (LLM)

An LLM is a deep learning model based on a set of neural networks that can parse text and understand the meaning between words. They are one of the foundational technologies driving AI. Users leverage LLMs by sending them human-language queries, also known as prompts. The LLM responds with uniquely generated text, audio, images, or video. 

LLMs can be operated in the cloud, but they can also be operated on-premises. 

Securing organizational context, on-premises 

An LLM knows little about your company’s offerings and processes, which are captured in internal documentation, knowledge bases, customer support logs, case law, regulatory filings, and other assets.  To fill this gap, data engineers use a technique called Retrieval-Augmented Generation (RAG) to supply context to an LLM. 

RAG stores information in a database and retrieves it based on the user’s query. It then provides this as additional information in the LLM prompt, improving accuracy and making results more specific to your individual context. Searches use nearest-neighbor search or graph traversal to find information with a high likelihood of relevance to the user’s query. 

While RAG architecture can be run in the cloud, it can also be run on-premises. This is another way that on-premises AI can be realized.

On-premises vector and graph databases

RAG workloads are typically implemented in either a vector or a graph database. Both provide a way to find related documents relevant to a user’s query by performing some form of approximate search, as opposed to an exact-match search. The goal is to surface documents that are relevant to the user’s intent.

Vector data provides support for both exact and approximate nearest neighbor searches, as well as a variety of distance and vector measurements. Knowledge graphs can outperform vector databases in terms of accuracy for complex “multi-hop” questions that require retrieving information from multiple sources. 

Like other AI data architectures, this can also be run on-premises using the right data platform.

On-premises AI agents

AI agents act as autonomous workers, performing specific tasks with little or no human intervention. They interact with external environments and tools to make decisions on behalf of the user. 

Agentic AI architectures are best suited for complex solutions that require retaining context across multiple queries. They can be simple, single-agentic architectures or multi-agent architectures that use coordinators to break down complex tasks into simpler, more focused queries assigned to specialized agents. 

Agentic architectures are becoming increasingly popular, and like other AI workflows, the right data architecture can run AI agents on-premises. Some analysts predict that by 2027, 50% of all companies will deploy agents. Agents have the potential to replace human-centric SaaS applications, delivering more value to customers in a shorter timeframe. 

In other words, tomorrow’s agents will become an integral part of your enterprise. They’ll be in every part of your daily data workflows, enabling natural language queries across data stores to accelerate insights and perform tasks across multiple domains. Since many organizations run on-premises, the ability to operate AI agents on-premises is critical to their project success. 

 

The importance of on-premises AI for sensitive use cases

Typically, the LLM component of an AI architecture is a commercial LLM available via an API call to the cloud. To achieve this, apps call out to an LLM outside of the company’s network, sending information across the public Internet. 

This doesn’t work if you’re in a highly regulated industry or have specific regulatory security requirements. Companies in industries such as finance, healthcare, insurance, and the public sector are often unable to send sensitive information via public networks, even if it’s encrypted. Such companies may also be unable to give a third-party LLM access to their data for compliance reasons. 

On-premises data and data sovereignty requirements

In addition, your company may also be bound by data sovereignty laws. These laws require data to remain within a specific geographic location. Depending on geography, this may necessitate storing data on-premises rather than in the cloud. 

Benefits of on-premises AI

In short, there is a range of use cases that necessitate keeping at least certain sensitive data within the corporate network. And doing so reaps numerous benefits, including: 

Increased data security and compliance

Keeping data on-premises reduces the risk of security breaches by third parties. It also facilitates the control and management of data flow and storage, thereby simplifying compliance with regulations such as the European Union’s GDPR.

Control over infrastructure

Hosted services are black boxes. You have no control over how they’re built or where they send your data. By contrast, an on-premises AI solution gives you complete control over how and where your data is processed. 

Faster to implement

Your organization may already have terabytes or petabytes of relevant data stored on-premises. Migrating all of this to the cloud is costly and time-consuming. Most of us know from bitter experience how mass data migrations can become bogged down and, ultimately, fail. Bringing your AI initiatives on-premises enables you to bring your AI use cases to market without a lengthy migration phase.

 

Connecting to your data where it lives

Analytics, data applications, and AI workloads are all slightly different. Yet each of them requires many of the same things from a data architectural standpoint to succeed.

• Easy access to data, no matter where it lives in the organization.
• Easy collaboration so that multiple teams can work together on producing high-quality datasets for general consumption. 
• First-class governance to ensure security, usage audits, and verify compliance with all applicable standards and regulations. 

Starburst: Powering your on-premises AI

Starburst was built to solve this problem, taking an open, hybrid architecture as our starting point.

We built Starburst to enable companies to support analytics, data apps, and AI use cases, making it easier and faster for them to discover, trust, and activate data wherever it lives. 

Our Icehouse architecture uses Apache Iceberg for low-latency storage and access. We’re powered by Trino, allowing for ultra-fast access to data in over 40 different data sources. With Starburst, you can connect to data where it lives immediately, bringing critical datasets into Iceberg for superior governance and performance. 

Explore on-premises AI today as the solution built for your organization. It allows you to implement AI projects successfully, accessing data from across your organization in a governed, compliant, and controlled manner. It’s the solution that’s allowing more organizations to adopt AI faster in ways that work for them.