#llm
Every summary, chronological. Filter by category, tag, or source from the rail.
Optimizing Transformer Inference with FlashNorm
FlashNorm accelerates transformer inference by folding RMS norm gains into projection weights and parallelizing normalization and matrix multiplication via custom CUDA kernels.
AI EngineerDebugging Silent Failures in Stateful LLM Inference
When stateful models like Jamba produce silent errors, they often stem from state cache mismanagement. Debugging requires logprob forensics, threading request IDs through kernels, and identifying how memory pressure triggers hidden architectural flaws.
Optimizing Inference for Agentic Workflows
Agentic inference requires shifting focus from individual request latency to end-to-end task completion, utilizing prefix caching and agent-aware scheduling to reduce costs and improve performance.
Scaling Small Open Source Models for Production
Small models often outperform frontier models on specific tasks. By moving from top-down routing to a decentralized, queue-based architecture, you can double cluster throughput and drastically reduce latency and costs.
Advances in Data Center Inference Engineering
Inference engineering is shifting from post-training optimization to a cycle where dedicated training processes—specifically in quantization, KV compaction, and speculative decoding—are essential for production performance.
Optimizing Inference Platforms for Trillion-Parameter Workloads
Inference platforms must prioritize KV cache locality and intelligent workload scheduling to manage the high cost of prefill, treating heterogeneous GPU capacity like a game of Tetris to balance real-time agentic traffic with overnight batch processing.
Why LLM Performance Benchmarks Often Lie
Common benchmark harnesses often fail to generate requested load, inflate latency, or use unrealistic settings, leading to misleading results. Reliable benchmarking requires multi-process load generation, client-side observability, and standardized, production-representative workloads.
Operating Distributed Inference Systems at Scale
Inference at scale is no longer a model problem; it is an orchestration problem. Reliability and efficiency now depend on a unified control plane that manages GPU state, KV cache, and distributed request routing.
LLM-as-an-Improver: Iterative Candidate Refinement
Instead of using LLMs only for verification, the 'LLM-as-an-Improver' framework uses feedback from verifiers to iteratively refine and improve candidate outputs, significantly increasing success rates in complex reasoning tasks.
Risks of Agent-Mediated Hiring: Access and Recurrence Bias
Multi-agent résumé screening systems can inadvertently amplify hiring biases, creating 'recurrence' where specific candidate profiles are consistently excluded due to agent-to-agent feedback loops.
Characterizing Web Search by Conversational LLM Agents
This research analyzes how conversational LLM agents navigate web search, identifying distinct search strategies and the relationship between query formulation, search results, and final response quality.
Mitigating LLM Tool Hallucination via Closed-World Resolution
To prevent LLM agents from hallucinating non-existent tools, implement a closed-world resolution framework that strictly validates tool calls against a predefined, verifiable schema before execution.
Mapping the Design of LLM Benchmarks
Current LLM benchmarks often lack transparency in design, leading to misaligned evaluations. This research provides a taxonomy to categorize benchmark construction, helping developers better understand what models are actually being tested for.
Detecting LLM Harm via Latent States
Rather than relying on output filtering, this research proposes monitoring internal latent states of LLMs to detect harmful intent before it manifests in generated text.
Moving Beyond LLMs: Jev and the Rise of Calibrated Decision Models
Jev is a new transformer-based model that replaces text generation with calibrated probability outputs, offering a faster, cheaper, and hallucination-free alternative for software automation tasks.
Harness Engineering: Building Reliable AI Agents
AI agents are composed of a frozen reasoning model and a controllable 'harness.' Harness engineering focuses on building the data, memory, and tool layers to turn nondeterministic model outputs into reliable, repeatable workflows.
Navigating the Open vs. Proprietary AI Trade-off
Choosing between open and closed AI models is a critical business decision that impacts margins, infrastructure, and defensibility. The most effective strategy often involves a hybrid approach rather than a binary choice.
Frontier AI Pacing, IBM Granite 4.2, and Meta's Muse
The panel discusses the industry-wide debate on slowing down frontier AI development, IBM's release of the reasoning-focused Granite 4.2 models, and Meta's vision for personal, agentic AI.
The Inference Engineering Pareto Atlas: Optimizing LLM Performance
The paper provides a systematic framework for navigating the trade-offs between cost, quality, and latency in LLM inference, identifying which optimization techniques dominate the performance frontier.
OBC-Prune: Outcome-Based Calibration for Reasoning Model Pruning
OBC-Prune introduces a calibration method for pruning large reasoning models that prioritizes final output accuracy over intermediate token probability, preserving complex reasoning capabilities during compression.
PrismML's Ternary Compression for On-Device LLMs
PrismML is shrinking high-performance LLMs to fit on consumer hardware by using 'ternary' weight compression, achieving 98% benchmark parity with original models.
Making Global Data AI-Ready: The UN System Data Commons
The UN is migrating its global statistics to a new platform built on Google's Data Commons to improve AI accuracy and enable direct data retrieval via the Model Context Protocol (MCP).
Standardizing Safety for Open-Weight AI Models
Baseten, Hugging Face, and Goodfire have partnered to create a transparent, built-in safety infrastructure for open-weight models to combat risks like model 'abliteration'.
Building Reliable Multi-Agent Systems with ADK 2.0 Workflows
Stop relying on complex system prompts for agent coordination. Use deterministic workflow primitives—sequential, parallel, and loops—to structure AI behavior and ensure reliability.
Observability for AI Agents: Tracing and Evaluation with MLflow
Traditional monitoring fails to capture the complexity of multi-agent AI systems. MLflow provides OpenTelemetry-compatible tracing and LLM-as-a-judge evaluation to identify silent failures, latency bottlenecks, and non-deterministic behavior in production.
CLEAR: Cross-Source Evidence Adjudication for Medical LLMs
The CLEAR framework improves medical LLM reliability by systematically adjudicating conflicting evidence across multiple sources, reducing hallucinations and improving clinical accuracy.
Safe Error Correction for LLMs via Frozen-Base Adjustment
The paper introduces a method to correct specific model errors by adjusting parameters while keeping the base model frozen, ensuring targeted fixes without degrading general capabilities.
Optimizing Disaggregated LLM Serving via Calibrated Routing
Efficient LLM serving in disaggregated clusters requires a two-stage approach: calibrating model latency predictions to account for system noise, followed by intelligent request routing to minimize total cluster latency.
Optimizing KV Cache Placement for Long-Lived LLM Sessions
To handle long-lived LLM sessions, move the KV cache beyond GPU VRAM into CPU memory and SSDs using tiered placement policies to balance latency and memory capacity.
Multiscale Indexing: Solving the Fixed Chunk Size Trap
Fixed chunk sizes are a form of lossy compression that creates a 20-40% recall gap. Instead of tuning chunk sizes, index data at multiple scales and use Reciprocal Rank Fusion (RRF) to merge results for significantly higher accuracy.
AI EngineerShowing 30 of 1372