Retrieval, measured like a lab

Every answer Syla gives your assistant is only as good as the passages it retrieves from your own coursework. So retrieval is not vibe-checked — it is measured, continuously, on a real student's corpus, with the same metrics retrieval research uses. Last measured August 25, 2026.

100%

Success@3

a correct source in the top three, every judged question

0.978

MRR

the first result is the right one, almost every time

0.873

nDCG@10

graded ranking quality across the whole list

0.877

Recall@10

share of all relevant sources surfaced

0.998

AUROC

knows when it does not know: refusal calibration

100%

paraphrase consistency

reworded questions find the same sources

Against the baselines

Same corpus, same judged questions, one pipeline stage removed at a time. The row that matters: vector-only search — which is what generic “chat with your PDF” retrieval does — and keyword search, which is what most tools actually ship. “Noise” counts forbidden results: a past exam ranked above the professor's own lecture.

configuration	nDCG@10	Recall@10	MRR	paraphrase	noise
Syla, full pipeline	0.849	0.866	0.956	100%	0
Vector-only (naive RAG — what “upload the PDF” does)	0.839	0.871	0.944	100%	0
Keyword-only (classic search)	0.516	0.618	0.625	67%	6

The part no benchmark shows

Deadline, grade, and status questions are never answered by any of the retrieval above. They are routed to structured Canvas records, because a semantic search that returns a confident, plausible, wrong due date is the one failure a student never forgives. The measurement for that path is simpler: intent routing and exam-scope accuracy are held at 100% by the same test suite, and a wrong answer there fails the build.

Method, honestly

Real corpus. Six real courses, roughly 4,900 indexed passages from actual lecture slides, syllabi, homework, and announcements — not a synthetic dataset.
Hand-marked truth. Every gold passage was found by literal text search over the corpus, the way a person would Ctrl-F the PDF. None came from running the ranker — labels derived from the system under test always flatter it.
Refusal is graded too. Ten questions have no answer in the corpus, and the correct behavior is saying so. Empty-answer correctness is currently 100%.
Regressions fail the build. The suite runs in CI against a recorded baseline; a metric that drops blocks the release.

Numbers produced by the committed evaluation harness. Questions about the methodology: founder@usesyla.app