arxiv Python package to fetch papers, any sentence embedding model (e.g. all-MiniLM-L6-v2), and a clustering method of your choice (BERTopic, HDBSCAN, k-means). You're free to pick the labeling strategy, and the visualisation library. Going further (optional). Build a small Streamlit / Gradio dashboard for the demo, or turn it into an agent that watches arXiv weekly and writes a short LLM-generated summary of the week's rising topics.What you'll build
Build a small system that ingests recent research abstracts from arXiv, groups them into coherent topics, gives each topic a human-readable name, and shows which topics are growing month by month. The output is a short ranked list of fast-rising topics plus a simple dashboard a researcher could browse to see what the field is excited about right now.
What goes in, what comes out
Input
Around 2,000–8,000 arXiv abstracts from the chosen categories, over a 2–10 month time window.
Output
A ranked list of around 20 named topics, monthly trajectories per topic, and the top 10 fastest-rising topics — viewable in a Streamlit or Gradio dashboard.
data/abstracts.jsonl){
"paper_id": "2509.12345",
"title": "Self-Refining Agents for Long-Horizon Tool Use",
"abstract": "We introduce a self-refining agent framework that combines tool use with periodic reflection. The agent maintains a scratchpad summarising what worked and what failed, and periodically rewrites its plan...",
"categories": ["cs.CL", "cs.LG"],
"submitted": "2025-09-14",
"authors": ["A. Doe", "B. Smith"]
}rank topic_label papers (last 3mo) growth
---- ------------------------------------------- ----------------- ------
1 Self-refining and reflective LLM agents 412 +218 %
2 Long-context retrieval with reranking 287 +154 %
3 Speculative decoding for inference speedup 203 +120 %
4 Mixture-of-experts routing and load balance 188 +96 %
5 Synthetic data for instruction tuning 174 +71 %Datasets
arXiv abstracts (chosen categories, last 2–10 months) ↗
Each record contains a title, an abstract (~150 words), one or more category tags, and a submission date.
How to get it: Use the arxiv Python package. Query one category at a time, respect the 3-second-per-request rate limit, save to data/abstracts.jsonl.
Kaggle arXiv metadata dump (optional, larger) ↗
A snapshot of 2M+ arXiv records as a single JSON file. Use it if you want a bigger dataset without the API.
How to get it: Download the JSON (~3 GB), filter to the categories and date range you care about, save to CSV or parquet.
Tools you'll need
These are suggestions, not requirements. If your team is more comfortable with a different library, model, or framework that achieves the same goal, use it — and briefly explain the choice in your README.
Python: Python 3.10 or newer. Compute: A laptop CPU is enough. No GPU required. Around 2 GB of free disk for cached abstracts and embeddings.
Embeddings: BAAI/bge-small-en-v1.5. LLM (optional): any hosted option (e.g. GPT-4o-mini or GPT-4.1-mini (OpenAI)).
pandas— Storing and slicing the abstracts (one row per paper).numpy— Embedding matrix handling, monthly counts.tqdm— Progress bars while you fetch and embed thousands of abstracts.
arxiv— Friendly client for the arXiv API. Handles pagination and rate limits.requests— For pulling the optional Kaggle metadata dump or any extra URLs.
sentence-transformers— Embeds each abstract.all-MiniLM-L6-v2is the standard small choice.bertopic— Wraps embeddings + UMAP + HDBSCAN + keyword labels in one object. Fastest baseline. (UMAP and HDBSCAN are defined below.)umap-learn— UMAP stands for Uniform Manifold Approximation and Projection. It compresses the high-dimensional (384-dim) embeddings down to a few dimensions so the clusterer can find structure.hdbscan— HDBSCAN stands for Hierarchical Density-Based Spatial Clustering of Applications with Noise. It groups points by density, finds clusters of variable size, and labels outliers as noise.scikit-learn— Used here for TF-IDF (Term Frequency–Inverse Document Frequency) baselines and small helpers.
openai / anthropic / groq— Hosted Large Language Models (LLMs) for naming clusters. The model only sees the keywords plus 3 sample titles per cluster.transformers— Optional, if you want to run a small local language model for labelling.
plotly— Time-series and stacked-area charts.streamlit— Quickest path to a clickable demo (~50 lines of Python).gradio— Alternative dashboard, slightly nicer for ML demos.
How to approach it
One reasonable path through the project. Specific tools (UMAP, HDBSCAN, BERTopic, etc.) are examples — feel free to swap them for alternatives you know better.
- Fetch. Pull abstracts for the chosen categories and time window. Cache to
data/abstracts.jsonl. - Clean. Drop abstracts shorter than 20 words, strip LaTeX, normalise whitespace.
- Embed. Encode each abstract with
bge-small-en-v1.5. Save the matrix todata/embeddings.npy. - Reduce. Use UMAP (Uniform Manifold Approximation and Projection) to compress the embeddings from 384 dimensions down to 5.
- Cluster. Run HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) on the reduced vectors. Expect 20–40 clusters plus a noise bucket.
- Label. Extract the top 8 keywords per cluster using c-TF-IDF (class-based Term Frequency–Inverse Document Frequency: TF-IDF computed per cluster instead of per document). Optionally ask a language model to name each cluster from the keywords plus 3 sample titles.
- Trend. Bin per month, count per topic per month, compute a growth-rate score.
- Present. Build a Streamlit dashboard: top-rising table, monthly trajectories, sample papers per topic.
What to deliver
- A reproducible script or notebook that takes a date range and produces the topic map.
- A Streamlit (or Gradio) dashboard (~100 lines) showing rising topics, monthly trajectories, and example papers per topic.
- A short README explaining how to run the pipeline and what the team found.