PAKGOV-RAG
In progressA bilingual Urdu-English retrieval-augmented generation system for Pakistani government and legal documents: ingestion, chunking, multilingual embeddings, BM25 plus dense retrieval, reciprocal rank fusion, grounded generation with citations, and failure analysis.
- IngestPDFs to text with PyMuPDF
- Chunk6,711 chunks with page and article metadata
- RetrieveBM25, dense (FAISS) and hybrid RRF, k = 60
- AnswerCited answers, or "insufficient evidence"
Measured results
One 50-question set shared by every retriever: 15 English, 15 Urdu, 20 mixed-language. Values as reported in my 30 September 2026 write-up; the evaluation files are in the repository. The set is small, so treat the numbers as indicative, not as a benchmark.
| Retriever | R@1 | R@3 | R@5 | R@10 | MRR |
|---|---|---|---|---|---|
| BM25 | 0.360 | 0.580 | 0.640 | 0.680 | 0.469 |
| Dense (multilingual MiniLM-L12-v2, 384-d) | 0.440 | 0.600 | 0.620 | 0.740 | 0.535 |
| Hybrid RRF | 0.500 | 0.660 | 0.740 | 0.820 | 0.605 |
- Hybrid R@5 gain over BM25
- +0.100
- Hybrid R@5 gain over dense
- +0.120
- Answers with citations
- 37 / 50
- Declined: insufficient evidence
- 13 / 50
Earlier pilot (Version 0): a 20-question synthetic test scored Hit@1 20% and Hit@5 25%. I keep that result because it shows what the first version could not do.
Limits, stated plainly
- The evaluation set is small, so one retriever beating another by a few points may not hold on a larger set.
- The corpus covers a limited set of official documents, and the system gives no legal advice.
- Next: article-aware chunking, reranking, larger natural-question sets, broader government coverage.
- Python
- sentence-transformers
- FAISS
- BM25
- RRF
- PyMuPDF
- Streamlit