Building a university RAG LLM Middleware

This talk covers building and deploying a university RAG system using Python FastAPI, hybrid retrieval, performance tuning, and real-world integration challenges.

Overview

I will showcase a RAG system I built and deployed at FAU, which serves as an AI-powered assistant over internal and public university data for students and staff.

The talk will cover:

Technical architecture: Python FastAPI server, hybrid retrieval (BM25 + dense embeddings via OpenAI + custom weight regression), and LLM generation pipeline.

Performance tuning: Techniques used to reduce latency (asynchronous calls, caching), improve retrieval quality, and handle generation accuracy challenges in a real environment.

Deployment learnings: Integrating with university systems, managing security and authentication, and monitoring user queries for iterative improvement.

Live Demo Plan:

1- Query the assistant live with questions like “What is the process for thesis submission?” or “What events are happening this month?”.
2- Display generation output in real-time.
3- Walk through relevant parts of the code (retriever logic, query pipeline).

Tech stack