AWS rolled out a new capability to its AI managed service solution that it says can fact-check AI models mathematically to reduce the chance of hallucinations.
Amazon Bedrock is a managed service that helps users build generative AI applications using a variety of foundation models. Bedrock underlies everything AI-related at Amazon, providing the foundation for, or integrating with, AWS SageMaker, AWS Lambda, Redshift, and OpenAI’s API.
The service provides access to a variety of AI models, including Titan from Amazon, Claude from Anthropic, Llama 2, and Mistral AI.
Bedrock Use Case
Bedrock is used to build and scale generative AI applications with foundation models. It can be used to create AI agents, for instance.
During his re:Invent keynote earlier this month, AWS CEO Matt Garman highlighted a Bedrock use case by Genentech. The biotech firm designed a generative AI system that allowed scientists to actually ask their data detailed questions, such as what cell surface receptors are enriched between specific cells and inflammatory bowel disease questions.
”For them, it’s really critical, because this system can identify the appropriate papers and data from this huge library, and it synthesizes all the insights and data sources,” Garman said. “It summarizes America’s information sites and sources, which is incredibly important for scientific reasons and to traceability, and they have that data [so] that they can go and do their work.”
The process used to take genetic scientists weeks to do just one token, but now, with AI, scientists can perform the task in minutes, he added.
“Genentech is expected to automate nearly five years of bidding efforts and ultimately deliver new medications to customers more quickly,” he said.
There were a number of improvements and new capabilities announced at re:Invent. Below, we’ve highlighted a few of the most important announcements.
Hallucination Detection
It’s established that AI hallucinates or makes things up. To help protect against hallucinations, AWS introduced automated reasoning checks as part of Bedrock Guardrails, which is a set of safeguards that can be customized to application requirements and responsible AI policies.
Although there are other solutions aimed at reducing hallucinations, automated reasoning check is unique to Amazon, the company said. It was developed internally to prove, for example, that identity access management policies are implemented in the way organizations intended.
“What we do is we use it to automatically check scenarios or software that makes up the vast majority of the big chunk of necessary storage systems, and we check those automated reasons before deployment — that includes validating things like correct behavior in response to unexpected events,” Garman said.
“Once automated reasoning checks are sure that the answer is right, only then would you send it back to the customer, so you can be 100% sure that you’re sending accurate results to your customers.”
— AWS CEO Matt Garman
Amazon’s experts thought it might be useful for reducing hallucinations.
“Spoiler, since I’m talking about this on stage right now, the answer is obviously yes,” Garman said.
Automated reasoning allows Bedrock to actually check statements made by the models to ensure they’re accurate based on “sound mathematical verifications,” AWS stated. It reportedly can block up to 85% of undesirable and harmful content and filter more than 75% of hallucinated responses from models for retrieval-augmented generation (RAG) and summarization use cases.
Bedrock can also show you exactly how it reached that conclusion, according to Garman.
“If the model isn’t actually sure that the answer is right, it’ll actually send it back and say ‘Suggest other prompts’, or give you as the customer ideas of how you can send these back to the model,” he said. “Once automated reasoning checks are sure that the answer is right, only then would you send it back to the customer, so you can be 100% sure that you’re sending accurate results to your customers.”
AWS said this capability should help customers build inference into mission-critical applications. Inference is the process of using a trained artificial intelligence model to analyze new, unseen data and make predictions or decisions based on the patterns it learned during training.
Bedrock Leverages Smaller, More Cost-Effective Models
Amazon Bedrock Model Distillation allows customers to use smaller, faster, more cost-effective models that deliver use-case-specific accuracy.
“Distilled models can run 500% faster and 75% cheaper than the previous the model they got distilled,” Garman said. “This difference in cost actually has the potential to completely turn around the ROI, as you’re thinking about if a generative AI application works for you or not.”
Model distillation can also automate the generation of synthetic data from the teacher model, train and evaluate the student model, and then host the final distilled model for inference.
Prompt Caching on Foundation Models
Bedrock also now supports prompt caching on Claude 3.5 Haiku and Claude 3.5 Sonnet v.2. This new capability can reduce costs by up to 90% and latency by up to 85% for supported models, according to AWS.
It works by caching frequently so that prompts can be used across multiple API tools, which uses fewer compute resources to generate an output and means you don’t have to constantly repeat yourself to the AI.
Other New Bedrock Features
Support for multi-agent collaboration. Organizations can now build and manage multiple AI agents that can work together to solve complex workflows. This allows developers to create agents with specialized roles tailored for specific business needs. It will reduce development time while ensuring integration and adaptability to evolving needs, according to AWS.
Amazon Bedrock Data Automation (BDA) enables developers to automate the generation of insights from unstructured, multimodal content such as documents, images, video, and audio. Examples of the types of insights it supports are video summaries of key moments, detection of inappropriate image content or automated analysis of complex documents.
Bedrock users can integrate company data into the generation process of a language model using Amazon Bedrock Knowledge Bases. That’s not new: What’s new here is that it now supports natural language querying to retrieve structured data from an organization’s data sources. Bedrock Knowledge Bases can use advanced natural language processing to transform natural language queries into SQL queries, allowing users to retrieve data directly from the source without the need to move or preprocess the data, AWS noted.
Developers can now access more than 100 publicly available and proprietary foundation models and Amazon Bedrock’s serverless models on Amazon Bedrock Marketplace. The models can be accessed through Bedrock’s unified APIs, and models which are compatible with Bedrock’s Converse APIs can be used with Amazon Bedrock’s tools such as Agents, Knowledge Bases and Guardrails.
It’s important to note that not all of these capabilities are available in all AWS regions as of this publication date. Some are still in preview or limited to select regions.
The post Amazon’s Bedrock Can Now ‘Check’ AI for Hallucinations appeared first on The New Stack.