Category: Ai
Local LLM with Retrieval-Augmented Generation
Let’s build a simple RAG application using a local LLM through Ollama.
Edit (2025-03-26): Added some words about next steps in conclusion.
Edit (2025-03-25): I re-ran the example with a clean database and the results are better. I also cleaned up the code a bit.
Over the past few months I have been running local LLMs on my computer with various results, ranging from ‘unusable’ to ‘pretty good’. Local LLMs are becoming more powerful, but they don’t inherently “know” everything. They’re trained on massive datasets, but those are typically static. To make LLMs truly useful for specific tasks, you often need to augment them with your own data–data that’s constantly changing, specific to your domain, or not included in the LLM’s original training. The technique known as RAG aims to bridge this problem by embedding context information into a vector database that is later used to provide context to the LLM, so that it can expand its knowledge beyond the original training dataset. In this short article, we’ll see how to build a very primitive local AI chatbot powered by Ollama with RAG capabilities.
The source code used in this post is available here.