9 MIN READ
We are seeing an explosion of large language model (LLM)--based chat interfaces being injected into applications across almost every domain. The generative power of LLMs combined with Retrieval Augmented Generation (RAG), a method to introduce additional knowledge on top of the learned material inherent within the LLM, now allows LLMs to maintain the context of conversations for use in future interactions. Yet once you introduce the ability to maintain user-specific history, you open a host of vulnerabilities that need to be mitigated.
In this article we showcase an approach to tackle one of these vulnerabilities, that of keeping individual chat histories private and unavailable for the LLM to utilize when interacting with other users. This type of solution can not only be used for chat histories but also to filter or combine different knowledge bases in an ad hoc manner when interacting with an LLM. For instance, allowing a user to select which news sources they would like answers to be generated from.
Overview
RAG was originally developed to allow LLMs to incorporate any external knowledge that is innate within the model during its text generation into the learned knowledge. Many solutions use this capability to allow for the LLM to incorporate chat history into subsequent responses, allowing the LLM to maintain awareness of previous interactions. To keep user conversations private, a solution needs to separate these individuals' histories for each generation. It is not scalable to have a dedicated model instance for each user or for each user to have its own repository of context, so we need an alternative method.
In this solution, we leverage txtai, an all-in-one embedding database for semantic search, LLM orchestration, and language model workflows. Other embedding search mechanisms would work here as well, with the key criteria being that individual rows can be tagged and filtered as belonging to a specific user(s). What follows is an introduction to how systems can be built with a separation of user context into LLM workflows and can be adapted to partition data for any reason—not just user chat history.
Solution Walkthrough
The following solution describes a toy example of multiple users inquiring about restaurants to visit, given their individual chat histories. Individual histories are stored in the same repository and flagged as belonging to separate users. Information from the knowledge base can be filtered based on a specific user to keep chat histories private. This setup also allows for shared knowledge bases to be queried.
The complete code can be found on MetroStar’s GitHub here. Below is a high-level diagram of the solution outlining how a chat interface input, along with a user ID, gets routed through to the LLM, and a response is generated.
Figure 1: Conceptual diagram of solution workflow
Setup
This solution uses txtai and the Hugging Face transformers libraries and these should be installed in your development environment.
The LLM used in this example is the MistralLite 7b instruct model, which, while much smaller than many other LLMs, should still be run on a GPU. During model initialization, weights will be downloaded to your computer for loading into memory. This download process may take ten minutes but will only need to be done once.
Knowledge Base Creation
Our knowledge base is a repository of text embeddings that have been created from a list of user chat histories. There is also a shared common knowledge base that is identified with the user ID of ‘0’. Each section of text is embedded using a sentence transformer and stored with its associated user ID. In this example, each user has a chat history of where their food preferences are located. For simplicity, these preferences are the entire chat history, but this setup would work with an arbitrary amount of other dialogues covering any number of topics.
Baseline Question and Answer
To get a look at normal generative response behavior, we have our example question about the best places to eat in Chicago executed against the pre-trained LLM. This response has no additional retrieved context. Depending on your hardware, this response may take several minutes to generate.
This results in a generic response to the question, “What is the best type of food to eat in Chicago?”
Chicago is known for its diverse and delicious food scene, with something to offer for everyone. Here are some of the best types of food to try in Chicago:
Deep-dish pizza: Chicago-style pizza is famous for its thick, buttery crust and generous toppings. Try a slice from Lou Malnati's or Giordano's.
Hot dogs: Chicago-style hot dogs are served on a poppy seed bun with mustard, relish, onions, tomato wedges, pickle spear, sport peppers, and celery salt. Try a classic dog from Portillo's or Superdawg.
Italian beef sandwiches: These sandwiches are made with thinly sliced roast beef, dipped in au jus, and served on a crusty Italian roll. Try one from Al's Beef or Mr. Beef.
Chicago-style hot dogs: These are similar to regular hot dogs but served with mustard, relish, onions, tomato wedges, pickle spear, sport peppers, and celery salt. Try one from Portillo's or Superdawg.
Gyros: Greek-inspired sandwiches made with sliced lamb or beef, tzatziki sauce, and topped with onions and tomatoes. Try one from The Original Gyro Factory or Gyro King.
Barbecue: Chicago has a growing barbecue scene, with restaurants like Smoque and Lillie's Q serving up smoked meats and sides like mac and cheese and collard greens
Ethnic cuisine: Chicago has a large immigrant population, so there are plenty of great ethnic restaurants to try, including Mexican, Chinese, Indian, and Thai.
Desserts: Chicago is home to some of the best desserts in the country, including deep-dish pies from Lou Malnati's and Giordano's, and ice cream from Jeni's Splendid Ice Creams.
Seafood: Chicago has a growing seafood scene, with restaurants like Fish Bar and The Lobster Place serving up fresh seafood dishes.
Brunch: Chicago has some great brunch spots, including Eggsperience Cafe and The Allis.
Overall, Chicago has a diverse and delicious food scene, so be sure to try as many different types of food as you can while you're there!
Retrieval Augmented Generation
RAG allows us to add to the knowledge that is inherently contained within the LLM. Information is retrieved from a knowledge store (typically based on the user prompt) and added to that prompt before it is sent to the LLM to generate a response.
Retrieval
Here, we see an SQL query that mixes embedding similarity search with rule-based filtering to retrieve text that is similar to the question (user prompt) and also matches a specified user ID.
Generation
Given the results retrieved from the knowledge base in the previous step, we now incorporate them into an extended prompt, which has been restructured to allow for additional context.
Here, we see the contextual response for User #1, which incorporates their specific chat history as well as the shared knowledge base. From above, we have the following information in User #1’s chat history: "I hate spicy food.” and "My favorite type of food is seafood." There is also the relevant shared knowledge, "The best restaurant anywhere is Burger King!" and "The best restaurant in Chicago is Pequod's Pizza.”
The best type of food to eat in Chicago depends on personal preferences. If you enjoy seafood, LeTour! is a great option. If you prefer pizza, Pequod's Pizza is highly regarded. However, Chicago is known for its diverse food scene, so there are many other options to explore, such as deep-dish pizza, hot dogs, and Italian beef sandwiches.
Here is the output when we set the user_id to 2, thereby only retrieving User #2’s context. You can see the output is greatly expanded as User #2 includes in their history the sentence, “I am open to trying all types of food."
The best type of food to eat in Chicago depends on your personal preferences and what you are in the mood for. If you are a fan of pizza, then Pequod's Pizza is definitely worth trying. They are known for their deep-dish pizza with a caramelized crust. If you are a seafood lover, then LeTour is a great option. They have a variety of seafood dishes, including oysters, lobster, and crab cakes.
If you are open to trying all types of food, then Chicago has a lot to offer. The city is known for its deep-dish pizza, hot dogs, and Italian beef sandwiches. But it also has a vibrant food scene with a variety of cuisines from around the world. You can find everything from Mexican to Thai to Indian food.
If you are looking for something unique to Chicago, then you should try a deep-dish pizza. This type of pizza is different from traditional pizza because the dough is thicker, and the toppings are piled high. It is a must-try for any pizza lover visiting the city.
Another popular food in Chicago is the Chicago-style hot dog. This hot dog is served on a poppy seed bun and topped with mustard, relish, onions, tomato wedges, pickle spears, and a dash of celery salt. It is a classic Chicago food that you should definitely try at least once.
For something a little different, try the Italian beef sandwich. This sandwich is made with thinly sliced roast beef dipped in gravy and served on a crusty roll. It is classic Chicago food that is perfect for a quick lunch or dinner.
If you are looking for something a little more upscale, you can try one of the many fine-dining restaurants in the city. Chicago has many great steakhouses, seafood restaurants, and Italian restaurants. You can find something to suit your taste and budget.
Overall, Chicago has many great food options. Whether you like pizza, hot dogs, or something more upscale, you are sure to find something that you will love.
Conclusion
This post has covered an example of how to implement RAG with privacy-preserving features with txtai and Hugging Face transformers. RAG allows generative ML models to incorporate information that they weren’t trained on, rapidly increasing the pace at which models can be deployed for use. The privacy-preserving aspects of this solution could be extended to partition data for a number of reasons, from data source filtering to faceted search.
The capabilities demonstrated here power several of MetroStar’s innovative solutions. They have been built to scale for any demand and drive various use-case functionalities.
About The Author
Justin Downes is the Sr. Director of R&D at MetroStar. He formally led the computer vision practice at AWS National Security and has spent over 20 years in public sector technology.
About MetroStar Innovation Lab
The MetroStar Innovation Lab brings together researchers, creators, engineers, and changemakers to nurture ideas into industry-changing products. Housed in our Reston HQ, our lab is a focal point for innovation and MetroStar's primary research and development center. Our team plays an integral role in leading the discovery, development, and integration of customizable solutions for the public sector. The Innovation Lab provides a diverse portfolio of technology, ranging from open-source design solutions to AI-led labeling for classified documents.
