Writeup — Retrieval
Hybrid retrieval for scanned Indonesian contracts
A contract question like “what does PASAL 5 say?” breaks a RAG pipeline in two unrelated places — and neither of them is the language model. Here is how the retrieval layer in Contract-Advisor RAG is put together, and which parts of it I have not finished.
The project is a retrieval-augmented contract advisor: you upload a contract — PDF, DOCX, or a photo of a printed page — and ask questions about it. It placed Top 50 at the Meta Llama Hackathon 2025. The interesting engineering is not the answering step. Groq running Llama 3.3 70B answers well enough once it has the right text in front of it. The difficulty is getting the right text there at all, and a scanned Indonesian contract fails that in two different ways.
Two failure modes that look like one bug
The first failure is optical. A phone photo of a printed contract gives EasyOCR a page with uneven lighting, compression noise, and a slight skew. Get the preprocessing wrong and PASAL comes back as PASAI or PASAL 5 with a doubled space — the clause is in the extracted text, but no longer matches anything a user would type.
The second failure is retrieval. Even with clean text, asking for a specific clause number is exactly the query type that embedding search is worst at. Embeddings encode meaning, and every clause in a contract means roughly the same thing to a sentence transformer: obligations between parties. The number is the whole query, and it is the part that gets averaged away. Ask for clause 5 and you reliably get clause 4, 6, and 12 — all excellent semantic matches, all wrong.
Both failures produce the same symptom: a confident answer about the wrong clause. That is why they are worth separating.
OCR: six variants, then pick a winner
Rather than tuning one preprocessing chain, the pipeline runs six and lets the results compete. From easyocr_implementation.py: the original image, grayscale, denoised (fastNlMeansDenoising), CLAHE contrast enhancement at clipLimit=2.0, a 3×3 sharpening kernel, and finally Otsu binarisation. Each variant is written to disk, run through EasyOCR with Indonesian and English enabled, and scored.
The selection rule is the part I would defend in review:
best_result = max(all_results, key=lambda x: x['confidence'] * x['word_count'])Confidence alone is the obvious choice and the wrong one. An aggressive binarisation pass often destroys most of the page but reads the four surviving words perfectly — a 0.98 confidence result carrying almost no document. Multiplying by word count makes a variant earn its confidence across the whole page, so a slightly noisier read of the full contract beats a pristine read of one heading. It is a crude objective function, but it encodes the thing that actually matters here: coverage is not optional when the answer might be in the clause that got thrown away.
After extraction, a normalisation pass repairs the specific damage OCR does to Indonesian legal structure — collapsing PASAL␠␠ back to PASAL␠, and folding Pasal into PASAL so casing does not fragment the token. It is deliberately narrow: it fixes what the OCR reliably breaks, and does not attempt to parse the document into a clause tree.
Retrieval: two lists, weighted, plus a bonus
The retriever runs both strategies and merges them rather than choosing between them. Semantic search returns 15 candidates; a TF-IDF keyword search returns 10. The merge in retriever.py weights them 0.7 to 0.3 in favour of semantics — but the detail that does the real work is the third term:
combined_scores[doc_id]['combined_score'] += keyword_score + 0.1 # bonus for appearing in bothA chunk that both searches surface independently gets a flat bonus on top of its weighted sum. The two methods fail in uncorrelated ways — semantics drifts to neighbouring clauses, TF-IDF matches boilerplate that shares vocabulary — so agreement between them is genuine evidence rather than a louder version of one signal. Weighted sums alone let a single very confident semantic hit dominate; the bonus rewards consensus instead.
On top of the merge sits a domain-specific boost: a tiered keyword dictionary (obligations, liability, breach ranked above party, shall, whereas, ranked above jurisdiction, governing) that nudges contract-bearing chunks upward. And when a query names a section explicitly, the retriever checks whether any merged result actually contains that reference — if none does, it falls back to a literal text scan of the chunks and injects those matches at the top with a hard-coded score of 2.0.
That fallback is the honest answer to “why not just use vector search”. Semantic retrieval is the right default for “what happens if we terminate early?” and structurally incapable of reliably answering “what does clause 5 say?”. Rather than making embeddings do a job they are wrong for, the exact path bypasses them entirely — but only for the queries that need it.
What is not finished
Three things, and I would rather name them than let a reader find them. The first is now fixed — I have left it in place with what changed, because the bug the fix uncovered is more instructive than the patch.
The exact-match fallback was hard-coded. Fixed Aug 2026 The pattern extraction in _exact_section_search tested for 'IV.15' in query — a debugging shortcut from the hackathon that never got generalised, so every other clause reference fell through to semantic search. References are now derived from a shared SECTION_PATTERNS list, which the three duplicated inline copies of that list also read from.
Finishing it surfaced a second bug, this time in the fix. Matching needs to tolerate the doubled spacing OCR leaves behind, so the reference is spliced with \s+ — but escaping the whole string first turns its spaces into \␠, and the \s+ then attaches to the backslash:
# wrong: matches a literal backslash followed by one or more 's'
re.sub(r'\s+', r'\\s+', re.escape("Section II.3")) → Section\\s+II\.3
# right: escape per word, then rejoin
r'\s+'.join(re.escape(p) for p in "Section II.3".split()) → Section\s+II\.3Every prefixed reference failed silently, and the tests still passed — the bare II.3 form matched the same chunk and masked it. What caught it was reading the debug trace rather than the green checkmark. The regression test now puts the bare form in an earlier chunk so the prefixed reference has to win on its own; the suite is 11 cases, and I checked that it fails when either bug is put back — one failure for the escaping, six for the hard-coding.
The section patterns are the wrong alphabet. Those regexes match Section IV.15 and Article IV — Roman-numeral conventions from English contracts. The OCR layer normalises PASAL and AYAT, but the retriever never learned to look for them, so the exact path does not currently fire for the Indonesian documents the OCR pipeline was built for. The two halves were developed against different sample documents and never met.
Retrieval quality is not measured. There is an evaluation harness, and it does record real numbers — 9 of 9 questions answered, 15.5s average response, context utilisation at 0.35. But the RAGAS metrics in the committed results are all zero because the runs had include_ragas: false, so faithfulness and context precision are unmeasured rather than good. Everything above is an argument from design, not from measurement. The weighting is 0.7/0.3 and the consensus bonus is 0.1 because those were reasonable starting values, not because a sweep said so.
That last one is the gap that matters. The architecture reflects a real understanding of how contract queries fail, and the failure modes it addresses are the correct ones. But until the RAGAS runs are switched on and the weights are tuned against them, the tuning is judgement rather than evidence — and it is worth being clear about which is which.
Code: github.com/fadhlillah2/llama-docs-auditor — retrieval in rag/retriever.py, OCR in rag/easyocr_implementation.py, and the fix described above in commit f47b6c4 with its tests in tests/test_section_references.py.