RAG for Legal - is it actually that easy? We see a lot of advancement in using LLM in open question answering (mostly in the form of a chatbot) out there in the market. Behind the mask of those chatbots, many of them are built on Retrieval Augmented Generation (RAG) technology. This makes question-answer chatbots across different domains very easy to develop based on many different open-sourced frameworks such as Llamaindex, Langchain, and many others. One specific domain that we are interested in in this work is the legal domain, which is intensely localized, and knowledge is hardly shared between different languages/countries. Thanoy is one example of a product that also utilizes RAG for legal question-answering. However, within this RAG craze, nobody in the room ever questioned how well their model performs quantitatively and what are the current limitations that RAG can’t (or can) be solved.

mediaLibraryStrapi

For a fully immersive experience, we suggest the reader read our <a href="https://arxiv.org/pdf/2502.10868">50-page technical report</a>.

we come up with many names: ThaiLegal - which is quite lame, JJKBench - which is totally irrelevant but 10/10 for memes, LawBench - which sounds too generic. So we finally choose NitiBench which probably is the best name among this list.

To tackle the mystery of RAG performance in the legal domain, we must first have the benchmark. Specifically, our benchmark consists of two components: dataset and metrics - which can be later used to construct tables in our fancy paper.&nbsp;

We curate two datasets for assessing the legal capability of RAG frameworks:

1. <a href="https://huggingface.co/datasets/VISAI-AI/nitibench">NitiBench-CCL</a>: Our benchmark data on legal question-answering, where CCL stands for Corporate and Commercial Laws. The benchmark focuses on 35 legislations as listed here. We curate the test set fully manually without any AI in the process, where we ask a legal expert to curate a question based on a given law section. The test set consists of around 3.7k entries.

2. <a href="https://huggingface.co/datasets/VISAI-AI/nitibench#%F0%9F%8F%9B%EF%B8%8F-nitibench-ccl">NitiBench-Tax</a>: Another challenge provides a more complex legal QA, specifically on tax laws. We scrape tax rulings from an official Revenue Department website, preprocess to extract relevant law sections and construct the QA dataset out of it. Our test data consists of only 50 entries since we exclude rulings older than 2021 to ensure that the law cited in the ruling remains the latest.

Both dataset samples are in a triple <code></code><code class="inline-code"><code></code><code>(</code>q, T, y)</code> format where <code><code class="inline-code">q</code></code> denotes the question, <code><code class="inline-code">T</code></code> represents positive law sections, and <code><code class="inline-code">y</code></code> is the answer to <code><code class="inline-code">q</code></code> given <code><code class="inline-code">T</code></code>.

We also introduce the <a href="https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG"><code><code class="inline-code">WangchanX-Legal-ThaiCCL-RAG</code></code></a> dataset, which contains both train and test split. Our NitiBench-CCL was also derived from this dataset test split. More details can be found in the <a href="https://arxiv.org/html/2502.10868v1#S3">technical report section 3</a>.Both dataset samples are in a triple <code>(q, T, y)</code> format where <code>q</code> denotes the question, <code>T</code> represents positive law sections, and <code>y</code> is the answer to <code>q</code> given <code>T</code>.

Now that we know what data to use, we need to address how to use our data to measure the effectiveness of the framework. In short, given a dataset and the LLM framework, what value do we use to measure its effectiveness under different perspectives?

First, we decompose RAG evaluation into two separate modules:

1. Quality of retriever - How accurate the retriever is in retrieving relevant law sections given query

Quality of LLM - The quality of the generated response given retrieved documents

Measuring retriever effectiveness Typically, evaluating retriever is straightforward, just use hit rate, recall, or MRR. Where hit rate and recall describes the accuracy of retrieval model (counted as “hit” if at least one document was retrieved), MRR measure correctness weighted by rank of the retrieved document.

However, this is only if there is only one positive. In the real world (or in NitiBench-tax split), given a single query, there is more than one relevant law section that must be retrieved correctly. To do so, we modified MRR and Hit Rate by proposing their variant to make such metrics compatible with multi-positive setups:

where <code><code class="inline-code">T</code></code> represents positive law sections, <code><code class="inline-code">R</code></code> is the top-k retrieved documents, and <code><code class="inline-code">rank(d)</code></code> is the rank of the matched document.

If the equations hurt your eyes, here’s a more initiative explanation. Multi MRR can be viewed as a more strict version of MRR where a full score means that ALL positive law sections are retrieved at the top rank without any irrelevant law sections in between. Multi Hit Rate is also a more strict version of hit rate where we require ALL positives to present in the retrieved document to consider “hit”.

Measuring LLM effectiveness Given the retrieved documents, we want to evaluate the effectiveness of the generated response. This is the challenging part since it’s hard to justify the “correctness” of the response. Conventionally, there are two main approaches to evaluate the generated response under an open question answering setup like NitiBench: 1) algorithmic evaluation where no external model is needed to facilitate evaluation (ROGUE score, BLEU score) or 2) Model-based evaluation where an external model is needed to evaluate the response (e.g. win rate, <a href="https://arxiv.org/pdf/2403.16127v1">CHIE</a>)

Inspired by <a href="https://aclanthology.org/2024.emnlp-main.552.pdf">Summary of the Haystack</a>, we adopt the coverage and citation score approach to our benchmark. Specifically, we further measure

1. Citation correctness of the response - we prompt the LLM to always cite relevant documents when answering. Using those citations, we calculate recall, precision, and F1 score to reflect how well LLM utilizes context. We call these metrics E2E (E2E recall, E2E precision and E2E F1)

2. Coverage and contradiction of the response - Similar to a summary of the haystack, we use the LLM judge to evaluate the response’s coverage (scores 0, 50, and 100). We also assess whether the LLM response contradicts the reference answer.

In conclusion, on generated response, we measure

🔍&nbsp; Experiments Setups &amp; Results from NitiBench

Once we got NitiBench in our own hands, we asked ourselves: What can we cook? The answer is simple. We cook research.

Using NitiBench, we cooked three research questions:

1️⃣&nbsp; What impact do components tailored to the Thai legal system structure have on a Thai legal QA system?

Saying that RAG only has a retriever and LLM is like saying that Jujutsu Kaisen only has Gojo as MC; there are more components apart from these two modules that influence the final RAG response. Generally, RAG can be viewed as shown in the figure below

Similar to how important tokenization used to be in the old era, we hypothesized that these two components-chunking strategy and prompt augmentation- are as important as how tokenizers used to be. Why? These two components are the only components that can easily inject inductive bias regarding specific domain knowledge, such as

Hierarchicy-aware chunking: The naive chunking strategy assumes that chunks obtained from sliding windows on the document are sufficient for retrieving relevant contexts. This is not a bad assumption, considering that documents are mostly generic, where facts are often grouped enough to be placed into a chunk. However, this assumption doesn't apply to the Thai legal document structure - legislation especially. Applying naive chunking on the raw law content resulted in incompleted law sections, too many law sections in one chunk. This also ignores the hierarchical nature of Thai legislation, where a book is usually divided into multiple headers (ภาค, ลักษณะ, หมวด, มาตรา, อนุมาตรา ฯลฯ). Thus, we proposed a hierarchy-aware chunking strategy, which is just an extensive rule-based and regular expression crafted by the sleepless hours of our data scientist. This chunking strategy perfectly divides legislation books into multiple chunks where each chunk is exactly one section (มาตรา).

NitiLink NitiLink is just a fancy name prompt augmenter, but what’s cool about this is that it links referenced law in that specific section into the prompt. Let’s say your retriever retrieves law section 291 of securities and exchange:

Section 291 of Securities and Exchange Act B.E. 2535: Anyone who violates or fails to comply with orders issued under Section 186(2) shall be liable to a penalty of imprisonment for not more than one year, a fine of not more than 300,000 baht, or both.

Oops, content from section 186(2) is necessary to incorporate into the prompt for LLM to understand! NitiLink resolves this referenced law and adds it to the LLM prompt for a more comprehensive context to LLM, which could likely prevent hallucinations on the mentioned law sections.

Experimental Results Building from the above components, we conduct experiments to validate the effectiveness of both the hierarchy-aware chunking strategy and NitiLink.

First, we compared the performance of using the naive and hierarchy-aware chunking strategy. Since naive chunking might cause the chunks to either be incomplete or have too many law sections in one chunk, which would make evaluation very tedious, chunks that do not fully cover at least one section were discarded. The results on NitiBench-CCL are shown below.

From the results, we observed that hierarchy-aware chunking achieves a slight but consistent advantage over the naive chunking strategy. Under such a small sample and in a simple setting (we didn’t conduct this experiment on NitiBench-Tax because there are only 19 entries left after filtering).

We also conduct an experiment to see the effect of incorporating NitiLink without any filtering:

From the results, incorporating NitiLink doesn’t improve overall metrics that much on NitiBench-CCL. We suspect that this is because of the simplicity of the NitiBench-CCL, which doesn’t require intensive legal reasoning. Another thing to consider about the performance not improving is the bias when curating data, where we ask legal experts to curate questions based on a single law section. This could break dependencies on referencing, thus making NitiLink less effective.

On NitiBench-Tax, adding NitiLink substantially improves retriever recall. However, since MultiMRR doesn’t improve along with recall, it suggests that the samples that contribute to recall improvement were usually ranked at the bottom.

Also, although recall improved significantly on NitiBench-Tax, other LLM generation metrics, such as coverage and contradiction, showed the opposite results. Improvement in recall from augmenting more positives doesn’t always contribute to better generation performance. We hypothesized that this could be due to the fact that resolving more references leads to longer context length, which might cause LLM performance to degrade.

We summarize our findings on this research question as follows:

2️⃣&nbsp; How do the choices of retriever and LLM affect performance in a RAG-based Thai legal QA system?

Ok, now we know that injecting domain knowledge crafted by sleepless hours of data scientists hasn’t only resulted in a slight increase in overall performance. Now it’s time to ask the question, “Who are the best retriever and LLM in the house?”

To know what’s the best retriever for the Thai legal domain, we fixed the chunking strategy to hierarchy-aware chunking and further evaluated the effectiveness of different retrieval model strategies: Using a base strong retrieval model, a finetuned version annotated automatically without human supervision, and a finetuned version using human annotated samples. The setups are outlined below:

Actually, we experiment with 8 retrieval models, see more on&nbsp;<a href="https://arxiv.org/pdf/2502.10868">our technical report</a>&nbsp;😉

Our goal is to investigate the performance gain of including human annotators in the annotation pipeline compared to baseline and automated approaches. The retriever results are shown in the table below:

In the figure, you can see that in the in-domain setup (NitiBench-CCL), human finetuned BGE-M3 achieves the highest score as expected. However, looking into the challenge set for NitiBench-Tax, we can see mixed results where the base BGE-M3 is the best in both multi-label metrics. According to these results, we can conclude our findings on the retrieval model as follows:

After that, we fixed the embedding model and experimented with different LLM setups. We also explore the effect of using NitiLink in this experiment since the old experiment is inconclusive.

With these values, we summarize our insights as follows:

Finally, using the results from previous experiments, we compared the RAG performance in an E2E manner using four setups:

We use HF BGE-M3 as a retriever for all setups and Claude 3.5 Sonnet as a LLM. The results are summarized in the table below:

The retriever component for Naive RAG is fairly low because we assumed that an incomplete chunk is always negative due to failure to identify what section it was. Although this is not a fair comparison, we argued that incorporating a full law section content is critical for legal applications.

From the results, we can make several key observations:

3️⃣&nbsp; How does a Long-Context LLM-based Thai legal QA system perform compared to RAG-based systems?

With the rise of Long-Context LLM (LCLM), we see several arguments stating that LCLM might be able to replace RAG, where all documents can be placed as a context to the model (see <a href="https://arxiv.org/abs/2406.13121">here</a> as an example). Is this the case with Thai Legal QA?

Since Gemini-1.5-pro can consume up to 2 million context length, which fits 35 legislation books (consumed around 1.2M context length), we explore the effectiveness of LCLM in Thai legal QA using NitiBench. We run this experiment only on a 20% subset of the NitiBench-CCL dataset due to a limited budget and on full NitiBench-Tax. The results are summarized in the table below:

In the table, we can definitely conclude that LCLM's performance on Thai legal QA is still subpar compared to that of RAG while being more expensive to compute.

To further address the capability of LCLM, we think it’d be funny if LCLM can do better than retriever if we isolate it tasks to just solely focus on retrieving relevant law section. We evaluate the LCLM-as-a-retriever setup compared to other retrieval models, the result is shown below:

From the result, we can see that LCLM-as-a-retriever is actually possible and performs surprisingly well compared to other methods. The only downside to this is that it costs significantly more than conventional retrievers, which might not be economically feasible in real-world setups.

From these experiments, we can conclude several key points as follows:

While our work sheds light on the complex domain of Thai legal question answering, we highlight several limitations of our benchmark as follows:

Addressing these limitations will improve the relevance and effectiveness of Thai legal QA benchmarks and contribute to a more reliable assessment of language models in the legal domain.

With our release on NitiBench and our research, we lay a foundation for evaluating LLM applications in the legal domain. We showed that our benchmark can address several limitations in the current state of RAG and LCLM technologies. We hope our work sparks future research toward a more robust Thai LLM on Thai legal question answering.

Although this sounds like a comprehensive study, we actually go deeper than this in our technical report. We provide more insights, test more retrievers, study the effectiveness of our proposed multiple variants, conduct a very detailed error analysis, and many more! Go read the full 50-page-ish report <a href="https://arxiv.org/abs/2502.10868">here</a>!

This is our first research project at <a href="http://visai.ai/">VISAI.AI</a>, and we appreciate any comments or feedback on our work. We open-sourced everything we used in our experiment, and resources are available here: