PDF Question Answering System

What you'll build

Build a Retrieval-Augmented Generation (RAG) system that lets a user upload one or more PDF documents and ask questions about their content. The system must chunk the PDF, embed the chunks, retrieve the most relevant ones for a question, and then ask a Large Language Model (LLM) to answer using only those chunks. Every answer must show its sources (page number + short snippet) so the user can verify it.

What goes in, what comes out

{
  "question": "What was the company's total revenue in 2024 and how did it change versus 2023?",
  "retrieved_chunks": [
    {"page": 4,  "score": 0.91, "text": "Total revenue for fiscal year 2024 was $2.34B, up 18% versus $1.98B in fiscal 2023."},
    {"page": 12, "score": 0.74, "text": "The growth was primarily driven by the enterprise SaaS segment, which contributed $890M."},
    {"page": 31, "score": 0.71, "text": "In contrast, hardware revenue declined 4% year over year due to inventory normalisation."}
  ]
}

Answer:
  Total revenue in fiscal 2024 was $2.34B, an 18% increase versus
  $1.98B in fiscal 2023.

Sources:
  [page 4]  "Total revenue for fiscal year 2024 was $2.34B, up 18%
             versus $1.98B in fiscal 2023."
  [page 12] "The growth was primarily driven by the enterprise SaaS
             segment..."

Datasets

Tools you'll need

These are suggestions, not requirements. If your team is more comfortable with a different library, model, or framework that achieves the same goal, use it — and briefly explain the choice in your README.

Python: Python 3.10 or newer. Compute: A laptop is enough. The embedding model is small; the LLM runs on a hosting provider.

• pypdf — Extracts text from PDFs page by page. Simple and works for most documents.
• pdfplumber — Better for PDFs with tables or multi-column layouts. Slower but more accurate.

• sentence-transformers — Local sentence embeddings. all-MiniLM-L6-v2 is small and fast; BAAI/bge-small-en-v1.5 is a bit better.

• chromadb — Tiny embedded vector database. Runs in-process, no server.
• faiss-cpu — Alternative; faster but no metadata filtering out of the box.

• openai / anthropic / groq — A hosted LLM to generate the grounded answer.

• streamlit — PDF upload widget, chat-style question/answer, citation rendering — all easy.

• tiktoken — Count tokens before sending to the LLM so you stay under the context window.

How to approach it

One reasonable path through the project. Specific tools (UMAP, HDBSCAN, BERTopic, etc.) are examples — feel free to swap them for alternatives you know better.

1. Parse. Extract text from the PDF page by page. Keep the page number with every chunk — you need it for citations.
2. Chunk. Split each page into chunks of ~500 tokens with ~50 tokens of overlap. Try not to break in the middle of a sentence.
3. Embed. Run every chunk through a sentence-embedding model. Store the embedding + chunk text + page number in the vector store.
4. Retrieve. When the user asks a question, embed it the same way and find the top-k (k=5 is a good default) most similar chunks.
5. Generate. Build the prompt: system instruction ("Answer using only the provided context. If the context does not contain the answer, say so."), the retrieved chunks, and the question. Call the LLM.
6. Render. Show the answer plus the cited chunks (page number + snippet) so the user can verify.
7. Evaluate. For each question in the eval set, check (a) did retrieval surface the correct page, (b) is the generated answer correct, (c) did the system cite the right page.

What to deliver

• A Streamlit app where the user uploads a PDF and asks questions.
• An evaluation report on the team's eval set: retrieval@5 (did the right page show up in the top 5?), answer accuracy, citation accuracy.
• A short README documenting the chunking strategy, the embedding model, the system prompt, and the team's observations.

References

• LangChain RAG tutorial
• ChromaDB documentation
• Sentence-Transformers documentation
• pypdf documentation
• Original RAG paper (Lewis et al., 2020)