Multimodal and Domain-Specific NLP

What you'll build

Build a Visual Question Answering (VQA) system: given an image and a natural-language question about it, the system returns a short answer. The team will combine a pretrained vision-language model (BLIP-2 or LLaVA) with a small fine-tune on a domain-specific QA dataset (e.g., chart QA, document QA, or biomedical QA — pick one). Compare zero-shot vs fine-tuned on the same eval set and report accuracy plus error patterns.

What goes in, what comes out

[
  {
    "image": "charts/2017-bar-revenue.png",
    "question": "What was the revenue in 2016?",
    "answer": "4.2B"
  },
  {
    "image": "charts/2017-bar-revenue.png",
    "question": "In which year was the revenue highest?",
    "answer": "2018"
  },
  {
    "image": "charts/europe-energy-pie.png",
    "question": "What share of energy came from renewables?",
    "answer": "37%"
  }
]

Approach                      Exact-match  Numeric (±5%)
--------------------------    -----------  -------------
BLIP-2 zero-shot                 0.42          0.55
BLIP-2 + LoRA fine-tune          0.61          0.74

Per question type (fine-tuned):
  Read a single value           0.81
  Compare two values            0.66
  Compute a sum / difference    0.42  ← still hard
  Identify a trend              0.71

Datasets

Tools you'll need

These are suggestions, not requirements. If your team is more comfortable with a different library, model, or framework that achieves the same goal, use it — and briefly explain the choice in your README.

Python: Python 3.10 or newer. Compute: A 16 GB GPU is comfortable. Load BLIP-2 in 8-bit; LoRA fine-tuning needs another few GB on top.

• transformers — Provides BLIP-2, LLaVA, and the image processors.
• torchvision — Image transforms.
• pillow — Image I/O.
• bitsandbytes — 8-bit loading so you can fit the model + LoRA on 16 GB VRAM.

• peft — LoRA on the vision-language model. Cheaper than full fine-tune.
• accelerate — Device placement and mixed precision.

• evaluate — Exact-match and other simple metrics.
• datasets — Streams the QA pairs.

How to approach it

One reasonable path through the project. Specific tools (UMAP, HDBSCAN, BERTopic, etc.) are examples — feel free to swap them for alternatives you know better.

1. Pick a domain. Choose one of ChartQA, DocVQA, PathVQA, or a custom domain. Different domains demand different reasoning.
2. Load the data. Inspect the question types. Pick the metric that matches: exact match for short answers, numeric tolerance for numbers, ANLS for documents.
3. Zero-shot baseline. Load Salesforce/blip2-opt-2.7b (or llava-hf/llava-1.5-7b-hf) in 8-bit. For each val example, build the prompt "Question: ... Answer:" and generate. Record accuracy.
4. LoRA fine-tune. Attach LoRA adapters to the language-model side of the vision-language model. Fine-tune for 1–2 epochs on the train set.
5. Generate on val. With the fine-tuned model. Record accuracy.
6. Break down by question type. Classify val questions into categories (single value, comparison, calculation, trend). Report per-type accuracy for both systems.
7. Error analysis. Pick 20 wrong answers from the fine-tuned model. Group by failure mode.

What to deliver

• A notebook with both zero-shot and fine-tuned VQA pipelines.
• A results table with overall and per-question-type accuracy.
• An error analysis: 20 failure cases categorised by type (misread chart axis, wrong arithmetic, OCR error, etc).
• A short README explaining the domain choice and what the team learned.

References

• BLIP-2 paper (Li et al., 2023)
• LLaVA paper (Liu et al., 2023)
• ChartQA dataset card
• DocVQA dataset
• PEFT for vision-language models