Choosing the Embedder

This demo has shipped bge-code-v1 since launch. In October 2026 it was measured against two alternatives on this corpus, with the chunking, retrieval and reranker held identical so that only the embedding model varied. It lost.

The two constraints that come before quality

Most published comparisons of code embedding models are not usable here, because two constraints disqualify candidates before accuracy is considered.

SQL Server caps a vector at 1998 dimensions

Vectors in this system live in a SQL Server VECTOR column, which is limited to 1998 dimensions. That removes two of the most frequently recommended code embedders outright: nomic-embed-code at 3584 dimensions and Salesforce/SFR-Embedding-Code-2B_R at 2304. Neither fits without truncation, and truncation is a change that would itself need measuring.

A dedicated vector store does not have this limit. Qdrant accepts 2304, 3584 and 4096, verified directly rather than assumed. Moving the vectors out of SQL Server would take the vector leg out of the fusion procedure and require rebuilding reciprocal rank fusion in application code, so it is an option held in reserve rather than a default.

License

Several strong candidates are CC-BY-NC-4.0, which makes them unusable in a commercial product regardless of how they score. They remain useful as measurement references.

Method

Three databases were built from the same 254 files and 605 chunks of eShopOnWeb. Chunk text was copied verbatim between them, so the chunker, the corpus, the hybrid retrieval procedure, the fusion weights, the score floors and the reranker are identical across all three. The embedding model is the only variable.

Each model was given its own correct query convention, because they are not interchangeable: bge-code-v1 takes an <instruct>/<query> wrapper, Qwen3 takes an Instruct:/Query: prefix, and the Jina model takes its documented nl2code prefixes. They also differ on whether an end-of-sequence token is appended before pooling. Getting any of this wrong produces vectors that look reasonable and retrieve badly, so each configuration was verified on a relevant-versus-irrelevant ordering check before a database was built.

The questions

Twenty questions taken from this demo's own usage log, which is to say questions real people actually typed, not questions written to be answered. Four natural negative controls came with them, queries the corpus cannot answer at all. The same twenty real questions, labelled by hand at file level with graded relevance (59 labels), provide nDCG, recall and MRR.

For the unlabelled questions the cross-encoder reranker scores the results. It is independent of the embedding model, which is what makes it usable as a judge, but it is still a model's opinion rather than ground truth.

Results

Both models as the inference stack actually serves them, on identical corpus, chunking, retrieval, fusion weights, score floors and reranker.

The third candidate, jina-code-embeddings-1.5b, is competitive on relevance but scores 0.0596 on the four negative controls, effectively tied with Qwen3 and well above the shipping model's 0.0072 on that small set. It is CC-BY-NC in any case, so it was only ever a measurement reference.

What was decided

The embedder was switched to Qwen3-Embedding-0.6B. It went against one metric, the negative controls, and that is worth stating plainly rather than presenting the measurement as unanimous, even though that metric turned out to overstate the cost.

What it buys: better top-1, better mean top-5, better separation, and better recall, nDCG and MRR on the labelled set, at 0.6B parameters against 1.54B, 1024 dimensions against 1536, and 25.2 chunks per second against 16.8. Cheaper to run, cheaper to store, and it finds more of the answers.

What it costs: on the four negative controls the mean top score is roughly eight times worse, 0.0593 against 0.0072. That overstates it. Almost all of the rise is one question: "parity check" now matches an Equals implementation at 0.23, which is arguably a fair hit. Re-measured on 38 verified-unanswerable questions through the production relevance gate, both models let the same 28 through under the score floors then in place. Under the floors re-derived since, bge-code-v1 would let 7 through and Qwen3 lets 11. That is a small real cost, not eight times.

On the labelled set the ranking gap is real at this sample size. nDCG@5 0.5942 against 0.4309: paired question by question, Qwen3 wins 7, bge wins 2, and 11 tie, with a 95% bootstrap interval of +0.05 to +0.29. MRR 0.8167 against 0.6000, interval +0.05 to +0.40. An earlier version of this page, scored on six template queries, had bge ahead on both, 0.5587 against 0.5412.

The reasoning for taking that trade: recall and ranking are what a user notices. The score floors have since been re-derived on this model, on 40 answerable and 38 unanswerable questions. Unanswerable questions that return results fell from 28 of 38 to 11, and no answerable question lost its answer.

This page previously recommended keeping the old model. That recommendation was based on the same numbers, read more conservatively.

What this evidence does not support

The hand-labelled set this page used to name as outstanding now exists, and the labelled rows above come from it. A larger one would tighten every interval here.