AI isn’t the future anymore; it’s the present. Yet, as businesses rush to integrate AI into their workflows, they’re discovering that adoption is anything but straightforward. Adding to the complexity is a high-stakes tug-of-war: providers like OpenAI and Anthropic want businesses to move daily work to their AI-first platforms. On the other hand, software developers aim to embed AI as a feature that enhances their existing products and keeps users on board.
This tension underscores a critical truth: AI may be ubiquitous and a priority for any reasonably large business, but success is not guaranteed. A recent Bain study highlights this paradox, reporting fewer AI solutions in production than a quarter earlier despite a surge in development activity. Meanwhile, the leaders of the pack are expanding their lead. What’s causing this bottleneck? Bringing AI to production is complex, requiring rethought strategies, adapted processes, and careful data management, safety, and scaling choices.
The New AI Development Paradigm
Building AI applications is a balancing act. AI is probabilistic, unlike traditional software, where inputs and outputs are predictable. It’s like a very clever toddler who sometimes does what you ask and sometimes decides to finger-paint your walls instead. Guardrails, in the form of a watchdog process that follows and can cut short conversations, are your way of keeping the system on task and out of trouble.
This unpredictability also reshapes the way we need to think about the engineering of AI applications. Making iterative changes to AI systems is tricky because overall accuracy can easily slide backward. To avoid this, you need to think upfront about what you want the large language model (LLM) to handle versus what you want to manage yourself. It’s not just about the model, it’s about the entire system and use case.
Think big enough to make a real impact and secure executive sponsorship early. Even the best ideas can stall without it before they leave the starting gate.
Building the Foundation
The starting point for any AI integration is choosing a foundational model. Thankfully, most models are API-compatible, making them relatively easy to swap out depending on your needs. Whether it’s GPT-4o or -o1, Claude, or Gemini, each provider also offers its models at various levels of sophistication and price. Meta is following a different strategy and releases their Llama models as “open-weights”: a model you can download and run on your own (beefy) hardware.
This interchangeability means the real differentiators lie elsewhere: how you integrate your company data, design safety and guardrails, and adapt your development processes.
Time To Bring in Your Company Data
The true power of AI lies in its ability to work with your company’s data. There are two primary strategies for achieving this: fine-tuning and vector embedding. Fine-tuning is a bit like the gym membership of AI: Everyone talks about it, but only a few actually use it. It requires significant resources, expertise, and ongoing maintenance, making it impractical for many companies.
Instead, most successful implementations leverage vector embeddings. Documents or text are preprocessed into vectors, which are like coordinates on a map indicating their words’ meaning. Rather than just longitude and latitude, the coordinates consist of thousands of dimensions. Items close together have a similar meaning.
Large documents contain many thoughts and meanings that must be split into chunks, small enough to be queried effectively but not so small that they lose context. Various chunking strategies, including sliding window approaches with overlapping chunks, can be employed. There are many ways to select which chunks are relevant in a conversation and are shuttled to the AI for analysis.
Scaling the vector database and keeping it up to date as your business data evolves are challenging problems. To maximize the impact of an AI investment, look to specialist IT vendors that offer these services.
AI in Production: It’s Worth the Effort
A robust data management plan underpins successful AI integration. The first step is capturing all interactions with the AI, including conversations and implicit or explicit feedback. This continuous data stream provides the raw material to build training datasets and improve performance over time. Monitoring AI once in production is equally critical. Tracking outputs in real-time ensures that the system behaves as intended, identifies anomalies, and provides opportunities for corrective action. Together, these practices form the backbone of an adaptive, resilient AI strategy.
Bringing AI to production is hard, but it’s worth it. Foundational models may get the headlines, but the real magic lies in how businesses use their data, design safe and ethical systems, and adapt workflows to AI’s unique demands.
With careful planning and execution, AI can transcend the hype and become the transformative force it’s meant to be.
The post AI Adoption: Why Businesses Struggle to Move from Development to Production appeared first on The New Stack.
