The need
Students in the specialization work across signal processing, machine learning and perception. Their course material is spread over many Moodle pages in PDFs, slides and annotated diagrams, and much of the meaning sits in the figures. Text-only RAG misses those figures, and keyword search does not help a student find an explanation.
What I owned
The whole system, from the literature review to the ingestion pipeline, the agent graph, the API, the web interface and the deployment.
The solution
The assistant treats documents and figures as equal sources. When a relevant visual exists, the interface shows it first, followed by an answer based only on retrieved evidence, ending in a Sources section with the PDF name and page.
Key modules
Multimodal ingestion
Separates text from figures, normalises text into Markdown, and uses a vision model to keep useful visuals and drop logos and decorations. Kept figures get a detailed caption, and both are embedded into one Pinecone index under two namespaces.
Agent graph
Six LangGraph nodes: a router with guardrails, a query reformatter, hybrid retrieval, an image relevance gate, answer synthesis and a final format check.
Web interface
A React chat with session history, talking to a FastAPI service in Docker through asynchronous submit and poll calls, exposed over HTTPS without opening inbound ports.
How it was built
The project started from a review of text RAG, multimodal RAG and vector databases, which showed that these systems are rarely built as complete deployments for visually rich course material.
Filtering visuals before indexing was a key decision, because decorative images pollute the knowledge base and hurt retrieval. A second gate at query time checks whether a retrieved image really fits the question before it is displayed. Program-level questions skip retrieval altogether, so the system does not waste time searching for general orientation answers.
The demonstrator indexes a few dozen course PDFs uploaded through Google Drive, because automated Moodle acquisition needs an officially provisioned token.
Results
Evaluated qualitatively on representative queries through the deployed interface:
- Course questions route to retrieval and return a grounded answer with the right figure and its source page.
- General program questions skip retrieval and correctly show “Sources: none”.
- When content is missing from the index, the assistant says so and does not invent an answer.