How the Models Were Chosen

This page shows how the embedding and reranking models running this demo were selected, what was measured, and — the part most write-ups skip — what the evidence does not support. Every number below is either measured against the live endpoints or a published benchmark with its source named. The two are never mixed.

The finding worth three orders of magnitude

Qwen3-Reranker scores through its chat template. Testing that wrapping applied client-side against posting raw query/document text — same model, same endpoint, same documents:

Both calls hit the same live endpoint, which at the time of this July 2026 test served Qwen3-Reranker-4B-seq-cls, 329 prompt tokens, three documents and one query:

What the demo showed, and why it was not a test

Six plain-English questions fired at the live index — 254 files, 605 chunks, health green. Kept here because they are what the demo actually showed, including the one it got wrong.

Five hit, one missed — and the miss is the useful one. "What happens when a user checks out" returned cookie settings and health checks: nothing to do with checkout. Rewording found the boundary:

The corpus has no "checkout" concept — it models the same idea as orders. The retrieval wasn't broken; the vocabulary was. Naming the domain concept fixes it instantly, and no amount of reranking would have.

The keyword leg, tested separately

Exact identifiers should not need semantics, and they don't: EmailSender → EmailSender.cs, IEmailSender.cs, EmailSenderExtensions.cs · OrderService → OrderService.cs, IOrderService.cs · IRepository → IRepository.cs. PaymentMethods returned exactly one result — Buyer.cs — rather than padding the list with near-misses.

Six rerankers evaluated

The incumbent was not bad — it remains strong on multilingual text (MIRACL 69.32, best in its table). It was bad specifically on code, at roughly half the achievable nDCG.

Why the headline benchmark was thrown out

The widely-quoted figure for this comparison is MTEB-Code 73.42 vs 41.38. It did not survive checking: that number is the vendor scoring its own model on its own retrieved candidates. Not independent.

So the claim was verified from a primary source instead — arXiv 2509.25085 v3, Table 2, published by a rival lab:

The gap is genuine, independent, and same-pipeline. Worth noting: two of the nine research agents reported that this table did not exist — they had read an earlier revision of the paper. Pinning the revision caught it.

Two calls that went against the obvious answer

Staging was inverted — embedder first, reranker second

The instinct is to swap the reranker first, since its benchmark gap is larger. Rejected: the reranker is the risky integration — chat-template handling plus yes/no logit extraction, and both vLLM and llama.cpp had shipped silent-wrong-score bugs. The embedder is deterministic, parity-testable, and a drop-in dimension. Do the verifiable one first.

A cost claim was corrected by hand

One research pass asserted the new reranker cost "+35% FLOPs." Working the embedding-table math directly — the incumbent's XLM-R-large table is ~250,002 × 1024 ≈ 256M of its 568M parameters and carries no FLOPs, leaving ~312M non-embedding, against Qwen3-0.6B's ~440M — gives ≈1.41×. The full-vocab lm_head at only the last position is ~0.3 GFLOP, negligible. "+35%" was true only for a naive all-position export.

Speed decided as much as accuracy

Cold start, not accuracy, was the real product risk. A visitor arriving after an idle period gets a first query that can take two minutes. Someone who assumes it's broken and leaves at second 20 never sees the retrieval quality at all. That produced the warm-up retry, the heartbeat, and honest "waking up the server…" messaging instead of a spinner that lies.

Speed also drove the hardware. Sized then for the 4B, which fits a cheaper L4, A10G was chosen for ~2× the memory bandwidth and better latency. Always-on serving was costed at ~$1,150/month; scale-to-zero brings real demo usage to a few dollars a day.

Measured on this corpus

Everything above this point is published benchmarks plus one live prompt-format test. This section is different: it is this system, this corpus, measured in September 2026. Both experiments are reported, including the one that was wrong.

A question set built to discriminate, not to demo

The original six demo questions each contained a word from their own answer's filename, so plain keyword matching could win them outright and no embedder was ever really under test. The replacement set has fourteen questions with zero token overlap between question and answer filename, verified per question at run time, plus four exact class names as a lexical control and two unanswerable questions to check abstention.

Experiment 1: the chunk header. Wrong.

Every indexed chunk carried a context header naming the file, namespace, class and fields. Measured across 605 chunks it was 33.3% of all embedded text, and 36% of chunks carried it twice. The theory was that this collapsed a file's chunks toward one vector describing the header rather than the code. It was stripped, the corpus fully reindexed, and the identical battery rerun.

It got worse, and not one of the six questions that both stacks missed moved. The header was not inert noise: // Class: RegisterModel was the only occurrence of "register" in that chunk, so Register.cshtml.cs fell from rank 1 to absent. Class and file names are real signal for natural-language questions. The change was reverted.

Experiment 2: is a code-specialized embedder worth it?

The comparison that had never been run. Local e5-large-v2 (general-purpose, 1024-dim, ONNX on CPU) against the hosted code embedder (1536-dim, GPU). Both corpora verified identical first: 605 chunks, 605 vectors, 351 carrying the header, on each side. The measurement is the exact cosine rank of the correct file across all 605 chunks, with no index approximation, no fusion, no reranker and no score floor, so nothing but the embedder is being compared.

The code-specialized embedder wins four of the six hard questions, and on one of them the margin is large. It still does not make them retrievable. Its best rank anywhere in that set is 15 of 605, so every one of these questions still misses a top-5 result. And it is substantially worse on the two questions the general-purpose model answers at rank 1 and rank 2.

The honest conclusion is that neither embedder can answer these six, that the two are good at different questions rather than one being better, and that the remaining gap is not something a better embedding model fixes. Chunking strategy is the open candidate.

Experiment 3: which reranker, on identical input

The experiment that should have come first. A cross-encoder is a pure function of (query, documents), so handing two of them the same candidate list removes the embedder, the index, the fusion weights and the score floors from the comparison entirely. Nothing is left varying but the model. Shortlists came from one retrieval call per question at the same pool depth the server reranks at.

Qwen3-Reranker-0.6B takes six questions to bge’s two, with five ties, and lifts MRR from 0.452 to 0.772 against bge’s 0.618. Same shortlists, same corpus, same questions.

This is where the retrieval quality on this corpus actually lives. Changing the embedding model moved nothing that mattered: no question became retrievable and both controls got substantially worse. Changing the reranker, on byte-identical input, moves MRR by 71%.

The clearest cases are the ones the first stage nearly lost. GetMyOrdersHandler.cs came out of retrieval at rank 27 and CatalogFilterPaginatedSpecification.cs at 28, both effectively invisible to a user, and Qwen3 pulled them to 4. ExceptionMiddleware.cs sat at 5 and went to 1, where bge left it at 5 and never moved it at all.

bge is not useless. It still adds 37% over raw retrieval and wins two questions outright, including rescuing IdentityTokenClaimService.cs from 18 to 2 where Qwen3 only managed 5. Across the set it is simply the weaker model. And this settles the question the correction at the top of this page raises: the 0.6B was deployed over the 4B for latency, but on measured retrieval quality it also beats the reranker this project previously shipped.

Latency is deliberately not scored here. The two models ran on different hardware, roughly 250 ms per query against roughly 140 s, which measures an A10G against a laptop CPU and says nothing about either model.

Experiment 4: does the embedder earn its place at all?

A cross-encoder never reads a vector. It reads the query and the document text together, so what it needs is a candidate list, not embeddings. Full-text search can produce one. That makes “full-text, then rerank” a complete architecture with no embedding model, no vector column, no index, and no reindex when models change. Given Experiment 3, that deserved a measurement rather than an assumption.

Same reranker on both sides. The only difference is how the shortlist was built.

The embedder contributes recall, and nothing else. Where full-text finds the answer but buries it, the reranker erases the difference completely:

Rank 25 out of retrieval, rank 1 after reranking, identical to the hybrid run that started at 6. So the embedder is not improving the ordering of anything once a strong cross-encoder is in play. What it does is put three answers in front of the reranker that full-text never surfaces at all: IdentityTokenClaimService.cs, GetMyOrdersHandler.cs and CatalogFilterPaginatedSpecification.cs. Those are pure recall, and no reranker recovers a document it was never shown.

The honest trade, then: the embedding endpoint buys about 17% more answerable questions and zero ranking improvement. That is a real gain and a real cost, and which way it goes depends on the corpus and the budget rather than on anything a benchmark table can tell you.

What this evidence does not support

The model selection is defensible on independent benchmarks plus a decisive live prompt-format measurement. The end-to-end retrieval-quality number is the one thing still missing, and it is gated on labelling that golden set. Stating that plainly is more useful than a chart that implies otherwise.

← Try the search this all feeds