Creating a Personal RAG Research Assistant - Taiko Edition

Summary

Built a fully local and private RAG assistance using a corpus of anonymized interview data (MacBook M3 Pro, QWEN2.5-7B using Ollama). Hybrid search + group filtering correctly identified the answer in the top 5 for 86-96% of questions. Factual and narrative answers were accurate 94-100% of the time while aggregate questions requiring cross-checking across different documents were a weak spot (42-54% for Test set). Routing these to per-group retrieval improved the accuracy for the Test set by +8 percentage points which directionally was heading towards the right direction but not a statistically significant result. Github link.

Background

During graduate school I was a member of Stanford Taiko, a student-run drumming ensemble which performed pieces based on an artform originating from Japan. I had my many highs and lows with the group and will at some point share my personal experiences at another point in my life, but one noticeable personal gap was a solid grasp of collegiate taiko history in the United States.

My understanding at the time was mostly restricted to learning about the first three collegiate groups were: UCLA Kyodo Taiko, Stanford Taiko, and Jodaiko. That was pretty much it. I had no idea that there were groups across the entire US. Beside the Intercollegiate Taiko Invitational (ITI) on the West Coast, there was another major annual gathering called East Coast Taiko Conference (ECTC) that was also organized by collegiate groups. It was only after I had stepped away from Stanford Taiko had I learned about this. Existing literature really barely scratched the surface at the time so I thought this was a perfect opportunity to not only conduct a deeper study but also share my learnings with the greater taiko community.

After 2 years of collecting publicly available information, emails, and phone interviews, I co-authored a paper on SocArXiv entitled as follows: Community empowerment via symbiosis: the impact of collegiate taiko drumming in the United States, 1990-2019. In summary, a growing symbiotic relationship between the collegiate and greater taiko communities helped accelerate the expansion of the taiko artform in the US. The mechanism behind community empowerment is diverse and is worth reading about for anyone who's curious about it, but a huge takeaway that I got was just how generous a lot of people were. There were various stories about how artists would spend their weekends helping students build their drums, impromptu expeditions for tires and duct tape, and summers spent learning about taiko from other groups.

Looking back, while there was plenty for me to learn, I do remember the sheer frustration of having to aggregate information or connect the dots across different documents basically by hand. I had different types of sources (papers, websites, emails, interviews) to glean from and if I had a specific question I wanted to answered, it required having to go through them one at a time just to make sure I didn't miss anything. Obviously it wasn't efficient, but this got the job done at the end. What would have been amazing at the time is a personal research assistant where I could feed it information and ask for answers to insightful questions.

With the advent of LLMs over the past few years, I realized that this is very much possible. So I went ahead and built a RAG pipeline where I prepared several anonymized emails, created different types of questions that my corpus could reasonably answer or could not find, and tried various models to help improve its overall performance.

Data & Privacy

For the purpose of protecting the identities of both individuals and taiko groups, I first removed personal identifying information from 5 emails that I had received which have a Q&A format. Something worth noting is that all the naming conventions you choose need to be consistent across your documents. For example if my name is replaced with P001, that needs to be done for all instances where I'm being mentioned. This applies to any metadata that you decide to include at the top of the document which precedes the main text that you really care about. Needless to say, make sure to have a separate table located somewhere that the LLM cannot have access to that links the identities back to their actual names.

Since I had only 5 documents to review, checking for leaks was not an arduous task, but if you have a larger set, then having Claude assist you would be extremely useful, although at the end of the day, you are responsible for what eventually gets passed through. There was one instance where the interviewee mentioned something about how their students were really busy outside of classes due to having to work part time. With other information provided in the email, Claude actually pointed out that this can make it easy for anyone to identify precisely which school the interviewee is based from. So I had to either replace some sections with more generic statements or remove them completely.

Evaluation Design

The next step is to chunk the documents into smaller pieces consisting of Q&A pairs. If all the documents are formatted correctly and consistently, then this part is easy. The next part is a bit more interesting where you write up evaluation questions. The easiest ones are the factual type, which just requires the LLM looking up one chunk from one document. For example, a question you might have is "When was Group1 founded?" which has a precise answer that's found in one of the chunks that is linked to the interviewer that is in that group. A more interesting question might be "Which groups teach Renshuu to new members?" which requires the LLM to perform an aggregation step to answer this.

In total, there are six types of questions that will be part of your evaluation: factual, narrative, aggregate, multi-doc, unanswerable and privacy. Once we have a set of questions, they will be split into a Dev (66%) and Test (33%) set. It's important to check to make sure that each question is correctly answered because this is how we are going to score how well the retrieval models are doing. For the first retrieval system we're building, we're performing either a keyword search (BM25), embedding search, or both. BM25 is just a classic search algorithm which scores each chunk by how many words it shares with the question while embeddings uses a small neural network that turns each chunk into a list of numbers that represents its meaning. For measuring performance, we're calculating recall@5 and MRR (mean reciprocal rank) which measures what fraction of correct chunks show up in the top 5 results and how high the first correct chunk is ranked respectively (1 over the chunk rank). recall@5 is important to pay attention to because ultimately the LLM we query will ultimately only read the top few chunks.

Retrieval

Table 1: n = 47 questions

Not surprisingly, the hybrid approach outperforms the other methods. But more importantly, where is the model not doing well? A couple of ones I want to highlight are cases where the method yielded an answer that was "the right topic but wrong group" or "the right group but wrong topic." To improve upon this, we can do two things: filter by group and add a reranker. The first approach solves the "right topic but wrong group" mistake in a straightforward manner while the second approach still sticks with the hybrid search but we add an additional model that reads the question and chunks together with the top 10-20 results.

Table 2: n = 32 questions (Dev only)

Now that we added both steps in the search method, the overall performance is better based on looking at the recall@X and MRR metrics. But something notable to report is that questions involving multiple documents did worse, which is not obvious here. Furthermore, latency has gone from 7ms to ~400ms per question after making these changes. Since this will be done locally, this should not be a concern, but should definitely taken into account if it's publicly available. To really assess whether or not the reranker is ultimately needed, let's apply the same methods to our test set and see how well it performs.

Table 3: n1 = 32 questions (Dev), n2 = 15 questions (Test)

Across the board, the group filter is a reliable win since it raised the MRR for both the Dev and Test evaluation questions by +0.09 and +0.07 respectively. However, notice that the reranker has mixed results. While there is a noticeable improvement in recall@5 for Dev, there is virtually no gain on test. Looking at MRR scores, there is an improvement on Dev but a lower score for Test. Thus, let's lock into just using Hybrid Search + Group Filter method for now since we're not getting conclusive results for attaching the Reranker model onto our workflow.

Answer Generation

To begin the answer generation stage, we're going to install Ollama which will run an LLM model of our choosing (for us QWEN2.5:7b) that read the top retrieved chunks from the method we chose and provide an answer with citations. When running the evaluation script, we're going to get a separate CSV file which gives us the end results. Something to note is that this requires us to actually go in and fill out the human_grade column, where we go in and provide our own score based on what answer the LLM gave for each question.

Reported accuracy for the Dev set is 91% (95% CI: 82-99%) which indicates that the system is overall doing well, but the weak spot here is again the aggregate questions which stands at 60% (CI 30-90%). Something else that pops out of the evaluation is the kappa metric, which measures the level of agreement between the LLM and human judges. With a kappa of 0.66, this indicates that there is substantial agreement but again it's more valuable to really understand where it's still falling short.

Fixing Aggregate Questions

One glaring limitation in our method thus far is the lack of multi-document (3) and aggregate (5) questions. To gain clarity on the LLM's performance on these types of questions, let's write up an additional 15 questions that we could ask and add them to the Test set (note that 2 of them were personally written by me while the rest were in assistance with Claude). Once we carefully approve the wording of each Q&A pair, we'll run 4 different types of RAGs: (1) vanilla, (2) per-group, (3) table, and (4) references. For method (2), what this does is ask each group separately and then combine later, which helps better answer "which groups/how many groups" type of questions. For method (3), this aims to answer questions from a structured table, which helps answers that involve counts, ranges, and filters. For method (4), this follows references to other groups which helps address any questions that requires retrieving existing relationships between different groups. Based on the nature of the question in hand, a logical route is taken that goes along any of the different RAGs available to us.

For the DEV set, the overall winner is the route where it used the per-group and references hybrid approach. Notice that the table approach did worse for aggregate questions, which indicate a loss in nuanced understanding which the other methods were able to pick up on.

For the TEST set, it clearly had a harder time for the aggregate questions which are arguably harder here than the DEV set. Consequently, overall accuracy performance for each RAG pipeline fell. It's worth highlighting that the LLM judge did worse on harder questions which is where it's critical to have a human judge to ensure that they're being either corrected or tracked so that for other similar types of questions, there needs to be more scrutiny on the LLM's response.

Testing for Statistical Significance

In terms of whether or not the improvements we're seeing is of any statistical significance, we can perform a paired difference test, which asks for each question whether the per-group route did better, worse, or the same than just vanilla RAG. A score is assigned to each question that goes through both methods and the difference is calculated. After applying this same step to each question, we calculate the mean difference which indicates that the per-group route improved accuracy on both the DEV (+3 pts) and TEST (+8 pts) across all 66 questions. Given that we have 6 wins, 3 losses and 57 ties, we can perform a two-sided sign test which uses a binomial distribution to calculate the p-value. If our null hypothesis states that both setups are equally good, then our calculations indicate that we would see this exact split or more lopsided about 51% of the time by chance. Since we can't distinguish the improvement from luck given a strict cutoff requirement of 5%, we consider the observed improvements as statistically insignificant. All losses here were on aggregate questions, which is clearly the biggest area of improvement.

Closing Remarks

While it's clear that there are limitations to this RAG that I had built, if I need answers to factual or narrative questions, this is a really great start. A RAG that uses the per-group + references route is clearly the winner here and what I would likely use as a personal research assistant. Next steps include adding new documents to the corpus, coming up with more difficult questions, and even trying out other LLM models (i.e. 7b vs 14b). Definitely glad I started building this up from the ground up which I hope over time will enable more interesting studies that uncover the evolution of the taiko artform in North America.

Some closing remarks: (1) never immediately believe the answers the LLM is giving you, be skeptical of answer to complex questions, (2) RAGs are definitely helpful to your own research and can be a great way to organize your sources, (3) would be interesting to see how it makes connections across different types of sources as I only gave it a series of Q&A paired chunks.