Writeup — Retrieval
Hybrid retrieval for scanned Indonesian contracts
A contract question like “what does PASAL 5 say?” can fail in a RAG pipeline in two independent places — and neither is the language model. Here is how the retrieval layer in Contract Advisor RAG is put together, and which limitations remain.
The project is a retrieval-augmented contract advisor: you upload a contract — PDF, DOCX, or a photo of a printed page — and ask questions about it. It placed Top 50 at the Meta Llama Hackathon 2025. The generation step is not the bottleneck explored here: even a capable language model cannot answer from context it never receives. The difficulty is getting the right text there at all, and scanned Indonesian contracts expose two different failure modes.
Two Failure Modes That Look like One Bug
The first failure is optical. A phone photo of a printed contract gives EasyOCR a page with uneven lighting, compression noise, and a slight skew. Get the preprocessing wrong and PASAL comes back as PASAI or PASAL 5 with a doubled space — the clause is in the extracted text, but no longer matches anything a user would type.
The second failure is retrieval. Even with clean text, a specific clause number is a poor fit for embedding search. Embeddings encode meaning, and many contract clauses appear semantically similar to a sentence transformer: obligations between parties. The identifier is the decisive part of the query, yet it carries little semantic meaning. Ask for clause 5 and the results can drift to neighboring clauses — plausible semantic matches, but the wrong answer source.
Both failures produce the same symptom: a confident answer about the wrong clause. That is why they are worth separating.
OCR: Six Variants, Then Pick a Winner
Rather than tuning one preprocessing chain, the pipeline runs six and lets the results compete. From easyocr_implementation.py: the original image, grayscale, denoised (fastNlMeansDenoising), CLAHE contrast enhancement at clipLimit=2.0, a 3×3 sharpening kernel, and finally Otsu binarization. Each variant is written to disk, run through EasyOCR with Indonesian and English enabled, and scored.
The selection rule is the part I would defend in review:
best_result = max(all_results, key=lambda x: x['confidence'] * x['word_count'])Confidence alone is the obvious choice and the wrong one. An aggressive binarization pass can destroy most of the page while reading the few surviving words with high confidence — a strong score carrying almost no document. Multiplying by word count makes a variant earn its confidence across the whole page, so a slightly noisier read of the full contract beats a pristine read of one heading. It is a crude objective function, but it encodes the requirement that matters here: coverage is not optional when the answer might be in the clause that was discarded.
As each detection comes out of EasyOCR it is cleaned in place: whitespace collapsed, a fixed list of stray punctuation stripped, and a small substitution table applied for recurring OCR confusions. It does not attempt to parse the document into a clause tree.
OCR Evidence
The project README reports 88%+ accuracy for Indonesian text recognition. At the linked source revision (609c0a0), the README does not define the accuracy measure or document the sample, reference transcriptions, measurement date, or run results behind that figure. The OCR code records model confidence and uses it to select a preprocessing variant; that confidence is not a comparison against reference text. The committed evaluation reports use a text document and do not establish OCR accuracy.
Retrieval: Two Lists, Weighted, plus a Bonus
The in-memory retriever runs both strategies and merges them rather than choosing between them. Semantic search returns 15 candidates; a TF-IDF keyword search returns 10. The merge in retriever.py weights them 0.7 to 0.3 in favor of semantics — but the detail that does the real work is the third term:
combined_scores[doc_id]['combined_score'] += keyword_score + 0.1 # bonus for appearing in bothA chunk that both searches surface independently gets a flat bonus on top of its weighted sum. The methods use different signals: semantic search can drift to neighboring clauses, while TF-IDF can match boilerplate that shares vocabulary. Agreement between them is therefore useful evidence rather than a louder version of one signal. Weighted sums alone let a single strong semantic hit dominate; the bonus rewards consensus instead.
On top of the merge sits a domain-specific boost: a tiered keyword dictionary (obligations, liability, breach ranked above party, shall, whereas, ranked above jurisdiction, governing) that nudges contract-bearing chunks upward. And when a query names a section explicitly, the retriever checks whether any merged result actually contains that reference — if none does, it falls back to a literal text scan of the chunks and injects those matches at the top with a hard-coded score of 2.0.
That fallback is the practical answer to “why not just use vector search.” Semantic retrieval fits “what happens if we terminate early?” but is a poor fit for exact identifier lookup such as “what does clause 5 say?”. The literal path handles that narrower job directly — but only for queries that need it.
What Changed — and What Remains
Five findings matter here. The first is now fixed, but it remains useful because the regression it exposed is more instructive than the patch itself.
The exact-match fallback was hard-coded. Fixed Aug 2026 The pattern extraction in _exact_section_search tested for 'IV.15' in query — a debugging shortcut from the hackathon that never got generalized, so every other clause reference fell through to semantic search. All section-detection paths now read from one shared SECTION_PATTERNS list.
Finishing it surfaced a second bug, this time in the fix. Matching needs to tolerate the doubled spacing OCR leaves behind, so the reference is spliced with \s+ — but escaping the whole string first turns its spaces into \␠, and the \s+ then attaches to the backslash:
# wrong: matches a literal backslash followed by one or more 's'
re.sub(r'\s+', r'\\s+', re.escape("Section II.3")) → Section\\s+II\.3
# right: escape per word, then rejoin
r'\s+'.join(re.escape(p) for p in "Section II.3".split()) → Section\s+II\.3Every prefixed reference failed silently, and the tests still passed — the bare II.3 form matched the same chunk and masked it. What caught it was reading the debug trace rather than the green checkmark. The regression test now puts the bare form in an earlier chunk so the prefixed reference has to win on its own; the suite is 11 cases, and I checked that it fails when either bug is put back — one failure for the escaping, six for the hard-coding.
The section patterns still target English conventions. Those regexes match Section IV.15 and Article IV. Even where the OCR output is clean, the retriever never learned to look for PASAL or AYAT, so the exact path does not currently fire for the Indonesian documents the OCR pipeline was built for. The two halves were developed against different sample documents and never met.
The OCR clean-up corrupts the token it was written to protect. The normalization pass in easyocr_implementation.py applies a substitution table before the PASAL spacing fix, and two of its entries are global: every S becomes 5 and every l becomes 1, written for digits but applied to the whole string. So PASAL 5 leaves the OCR layer as PA5AL 5, the spacing fix that follows never matches, and the structure pass that inserts a line break before PASAL never fires for it — AYAT and BAB survive only because they contain neither letter. The fix is a digit-context rule for those two substitutions; it is not in the repo yet.
The hybrid path is not the default wiring. Retriever.get_retriever() sets _use_chromadb = True and, whenever ChromaDB comes up, returns a ChromaDBEnhancedRetriever whose query is a plain similarity_search(query, k=4). The merge, the consensus bonus, and the exact-match fallback described above all live in SimpleRetriever, which that wrapper only calls when ChromaDB throws or returns nothing. In a normal run the hybrid retrieval is the fallback, not the path a question takes first. Routing section-bearing queries to it is the obvious next change.
Retrieval quality is not measured. There is an evaluation harness, and it does record real numbers — 9 of 9 questions answered, 15.5s average response, context utilization at 0.35. But the RAGAS metrics in the committed results are all zero because the runs had include_ragas: false, so faithfulness and context precision are unmeasured rather than good. Everything above is an argument from design, not from measurement. The weighting is 0.7/0.3 and the consensus bonus is 0.1 because those were reasonable starting values, not because a sweep said so.
That last gap matters most. The architecture separates OCR corruption from retrieval mismatch, but until RAGAS is enabled and the weights are tuned against measured results, the tuning remains engineering judgment rather than evidence.
Code: github.com/fadhlillah2/llama-docs-auditor — retrieval in rag/retriever.py, OCR in rag/easyocr_implementation.py, and the fix described above in commit f47b6c4 with its tests in tests/test_section_references.py.