Personal Learning Platform with RAG Evaluation

Any organisation that wants an assistant to answer from its own documents (a support desk, a policy library, a training programme) meets the same gap: a general language model answers fluently, but not from that material, and it cannot show where an answer came from. Retrieval augmented generation closes the gap by fetching the relevant passages before the model writes, but it adds choices that decide quality: how to split the documents, which encoder, how many passages, which model writes, and how to tell a correct answer from a plausible one. In learning there is one more requirement, because an answer read is not an answer learned, so the learner needs a way to check what stuck. This project builds that system over one semester, in three steps, for someone learning to invest from scratch.

The first step fixes how output is judged: the same prompts run through three hosted models and every answer is scored on one rubric. The second is a benchmark harness for retrieval: one configuration file drives interchangeable pipelines behind a single interface, a plain one and one that rewrites the question and fetches more when its confidence is low, swept across encoder sizes and retrieval depth and scored by exact match, F1 and a language model acting as judge. The third is the portal: a curated library is cleaned, split on sentence boundaries and encoded once into a local vector index; each question fetches the five closest passages, a hosted model answers from them with the sources shown beside the answer, and a second call writes a multiple choice quiz that is filed for the learner to take later. A fixed set of questions with written answers runs through the same chain to score it.

• Learning

  A fixed evaluation set before any tuning

  A retrieval system has many settings, and without questions whose answers are known every change is a matter of taste. The benchmark made that concrete: fetching ten passages instead of one lifted F1 from 0.49 to 0.63, while the rewrite gated by confidence changed only about one answer in ten. A frozen question set, scored the same way on every run, is what lets a team change the encoder, the chunking or the model in production and know whether it got better.

• Learning

  Sources as data, not as prose

  The portal shows its sources from what retrieval returned, each passage carrying the title, type and author recorded for its document, rather than trusting the model to cite. That keeps the library the authority and makes every wrong answer traceable to one of two places: the passages fetched or the text written from them. An answer that cannot be traced to a document cannot be audited or corrected, which is what makes grounded answers usable where accuracy matters.

• Learning

  Index once, answer per question

  Cleaning, splitting and encoding the library happen once, offline, into an index kept on disk; a question only encodes itself, searches and asks the model to write. The encoder is an open model that runs locally, so rebuilding the index costs nothing, and only the writing step is paid per call. Separating the two paths is what lets the library grow and the writing model change without the other side noticing, with cost that follows questions rather than documents.