How it works
- Retrieval: we fetch technology headlines from NewsAPI and preprocess them with NLTK (tokenize, drop stopwords, lemmatize).
- Search, from scratch: a TF-IDF index with cosine similarity, with no search library. Weighting title words 3× noticeably improved which articles came back first, and ignoring terms that appear in fewer than 2% or more than 70% of articles filtered out the noise.
- Answers: the top 5 articles go to Qwen2.5-1.5B-Instruct, running on the GPU rather than through a hosted API, and the answer shows which articles it came from.
- Topics: terms that appear together across at least two articles are grouped into 5 topic clusters, each with 5 sub-trends, and shown as charts.
Limits
The free NewsAPI tier caps how many articles you get (about 70–100 per fetch), and a 1.5B-parameter model sometimes gives generic answers. A larger model and more sources were the obvious next steps.