For many businesses, 2024 has been the year of the chatbot, and many experts are predicting that 2025 is the year that Agents will start to make huge progress. Our customers are asking questions like: "Should we make a chatbot?", "Should we use one?", "Do we really need to build one, or can we just make a better search box?". Meanwhile, dozens of tools have sprung up to help you make your own chatbot in minutes with slogans like: "Chat with your Data!", "Chat with your PDF!", "Chat with <insert enterprise CMS here>".
So, why not just use one and call it a day?
Problems with point-and-click chatbot implementations
"Unfortunately, unstructured data is difficult to extract, and it needs to be processed and transformed to make it ready" - Dr. Swami Sivasubramanian, VP of AI and Data at AWS, re:Invent 2024
Early and unsophisticated brute-force chatbot implementations have involved throwing a bunch of unstructured data at a Retrieval Augmented Generation (RAG)-enabled chatbot. While they technically deliver on their promise of letting you "chat with your data", the LLM backing the chatbot will be confused by what constitutes "your data" if it has the following characteristics:
1. It is unstructured, generated on-demand, or “dirty” in that it is cluttered with undifferentiated metadata and boilerplate text. For example:
- HTML has a lot of boilerplate tags that look the same on every document; so how is one document really distinguished from another?
- PDFs have odd table layouts and images, graphs, or other non-textual data that may not be fully understood or requires an understanding of layout to make sense. In fact, PDFs are often ingested via visual interpreters rather than text parsers.
- Proprietary or specialized file types, file encodings (I’m looking at you, base64) or folder structures may be in use.
- In other cases, “your data” may actually be generated on-demand via JavaScript or other scripting languages at time of access.
2. Conflicting and versioned data. Corporate wikis might have a huge volume of abandoned and out of date information that looks, to a language model, to have equivalent relevancy to today’s updated pages.
3. Volume - there is a lot of it. Simply dumping Terabytes of data into a vector store might make it difficult to find the relevant information to a query with a single search. Introducing some structure and intelligent chunking to your data allows for more focused Retrieval (the “R” in RAG), so that you can send more relevant information to your model. This will save you time and money, make your queries more efficient, and delight your users with faster and more intelligent responses.
4. Data lacking semantic meaning via context. Consider the phrases “This course will cover SQL” and “This course will not cover SQL”. These phrases are similar in that they’re both about SQL; however, if the purpose of your application is to help people find courses about SQL then it might erroneously pull up both courses unless your chatbot accounts for this type of data pollution. More control over the search process (and possibly, the embedding process) to customize it for your use cases could vastly improve the chatbot experience.
This is why at least half the work of designing a chatbot is in data pre-processing and determining appropriate search and agentic workflows.
Solutions to brute-force RAG drawbacks
At Caylent, our AWS-certified architects have developed sophisticated approaches to overcome these challenges, delivering chatbots that truly understand and effectively utilize your organization's knowledge.
Here are some key strategies we have implemented:
1. Structured Data Transformation
Instead of feeding raw documents directly into your vector store, we can establish a robust data preprocessing pipeline leveraging AWS services. This can involve:
- Converting unstructured data into well-defined schemas using custom ETL processes
- Stripping irrelevant boilerplate content through intelligent filtering
- Extracting meaningful metadata
- Breaking documents into logical, semantic chunks with advanced tokenization strategies
For example, rather than embedding entire PDF documents, we can break them down into sections with clear hierarchical relationships and metadata about their source, date, and relevance.
2. Version Control and Data Freshness
We implement a systematic approach to managing document versions using AWS services:
- Tag content with clear timestamp information in Amazon DynamoDB
- If your data is in Amazon S3, we can establish a deprecation workflow for outdated content using Amazon S3 lifecycle policies
- Create explicit relationships between document versions
- Set up automated processes to archive or remove obsolete information
This ensures your chatbot prioritizes current information while, if necessary, maintaining access to historical context when needed.
3. Strategic Chunking and Hierarchical Search
Our team may implement sophisticated search strategies such as:
- Develop a multi-stage search strategy using multiple data stores
- Use metadata to refine semantic search results and filter content that the user is unauthorized to see
- Implement document summarization using AWS Bedrock
- Create explicit relationships between related content pieces
4. Context Enhancement
We can enrich your data with additional context:
- Include business rules and domain-specific knowledge as metadata
- Create structured templates for different types of content
- Maintain clear provenance information for all data sources
5. Context Segmentation and Intelligent Routing
We can implement multi-context architectures that partition your data into distinct knowledge domains. This approach:
- Reduces costs by limiting search scope to relevant contexts
- Improves accuracy through specialized embeddings per domain
- Enables dynamic routing based on query intent
Our team implements this through either semantic routing engines that pre-process queries, or by allowing the LLM itself to select optimal contexts - choosing the approach that best fits your use case and data characteristics. Data segmentation can also be used to restrict information domains to different groups of users.
6. Continuous Refinement
As your users interact with the tool, we can treat your chatbot's data layer as a living system by implementing features that allow:
- Monitoring user interactions using AWS CloudWatch to identify common failure patterns
- Analyzing search patterns to optimize chunking strategies
- Regularly updating and refining your preprocessing pipelines
- Creating feedback loops between user experience and data organization
Implementation Example
Instead of this basic point-and-click approach:
Tom Manning
