Databricks RAG: Low-Dim Qwen3 + Rerank for 89% Recall@10
Minimize embedding dims to 256 with Qwen3 MRL (self-managed path), set num_results=50, always rerank ANN top-50 candidates for +15pts recall@10 over 74% baseline.
Minimize Dimensions and Tune Queries to Cut Latency Without Losing Recall
Higher-dim embeddings (1024-1536) increase ANN scan costs, memory use, and slow throughput—test empirically to pick the lowest dim preserving recall@10, like 384 over 1024 if equivalent. Limit num_results to 10-100 (default 50 with reranker, 10 without) since HNSW scales linearly and excess slows queries without better answers. Match endpoint SKU to scale: Standard for <2M 768-dim vectors (low latency), Storage-Optimized for <1B vectors (cheaper, higher latency, dims divisible by 16, triggered sync only). Add metadata filters (e.g., {"document_type": "manual"}) to Delta tables for scoped ANN scans, boosting precision/speed. Stick to ANN for semantic queries (highest QPS); hybrid (ANN+BM25) only for exact terms like SKUs or ISO 13849-1.
Self-Manage Qwen3 MRL Embeddings to Hit Target Dims Like 256
Fixed-dim models like databricks-gte-large-en (always 1024) force re-embedding for size changes. Qwen3-Embedding-0.6B uses Matryoshka Representation Learning (MRL) to pack signal into early dims, enabling safe truncation to any power-of-2 (32-1024). Managed Delta sync ignores dimensions param, always outputs 1024—use self-managed: pre-compute with API ({"input": [text], "dimensions": 256}), UDF to Delta table (chunk_embedding), then index with embedding_vector_column and embedding_dimension=256. Query same way: embed query at 256, pass vector to similarity_search. For prod scale, swap UDF for ai_query() batch inference.
Rerank Top-50 ANN Hits for 15pt Recall Gain Over Vector Distance Alone
ANN cosine similarity doesn't guarantee query relevance—close vectors (e.g., "sensor calibration" vs. "actuator recalibration") rank by distance, not utility. Databricks Reranker re-scores top-50 with query-aware model: 74% ANN-only recall@10 jumps to 89% (+15pts), beating cloud rivals by 10pts. Enable via reranker={"model": "databricks_reranker", "parameters": {"columns_to_rerank": ["chunk", "doc_summary"]}} (first 2000 chars, richest first; order matters). Adds ~1.5s latency—skip only for <200ms needs, >5 QPS unscaled, or non-RAG search. Production stack: Qwen3@256dims (self-managed), ANN HNSW, triggered Delta sync, rerank metadata.