News Feed

Building better AI benchmarks: How many raters are enough?

Google Research · 2026-03-31 · Evals

Algorithms & Theory

Agent Evaluation Readiness Checklist

LangChain Blog · 2026-03-27 · Evals

A practical checklist for agent evaluation: error analysis, dataset construction, grader design, offline & online evals, and production readiness.

How Kensho built a multi-agent framework with LangGraph to solve trusted financial data retrieval

LangChain Blog · 2026-03-26 · Evals

Discover how Kensho, S&P Global’s AI innovation engine, leveraged LangGraph to create its Grounding framework–a unified agentic access layer solving fragmented financial data retrieval at enterprise scale.

How we build evals for Deep Agents

LangChain Blog · 2026-03-26 · Evals

💡TLDR: The best agent evals directly measure an agent behavior we care about. Here's how we source data, create metrics, and run well-scoped, targeted experiments over time to make agents more...

Show HN: Knitting – shared-memory function calls for JavaScript workers

Hacker News · 2026-03-24 · Evals

What it isKnitting is a from-scratch concurrency framework for JavaScript built on shared memory. Instead of manually wiring message protocols, you just call functions across threads.Why I built...

A New Framework for Evaluating Voice Agents (EVA)

Hugging Face · 2026-03-24 · Evals

Testing and Refining Claude Code Skills with MLflow

MLflow Blog · 2026-03-23 · Evals

How to test Claude Code skills using MLflow tracing and LLM judges, and create a self-improvement loop where Claude Code refines its own skills.

Your Agents Need an AI Platform

MLflow Blog · 2026-03-18 · Evals

Making your agents work reliably in production requires observability, evaluation, version control, and governance. Learn how MLflow brings all four together as the only complete open source AI platform.

SOTA Embedding Model for Agentic Workflows Now in Public Preview

Databricks Blog · 2026-03-17 · Evals

Retrieval underpins modern AI systems, and the quality of the embedding model determines...

Benchmarking LLMs at the Game of Science (Eleusis) [video]

Hacker News · 2026-03-16 · Evals

Article URL: https://www.youtube.com/watch?v=tz5wALHhhds Comments URL: https://news.ycombinator.com/item?id=47396503 Points: 1 # Comments: 0

Beyond Semantic Similarity: Introducing NVIDIA NeMo Retriever’s Generalizable Agentic Retrieval Pipeline

Hugging Face · 2026-03-13 · Evals

AI is supercharging fake work

Hacker News · 2026-03-12 · Evals

As anyone with an internet connection knows, there’s been a lot of buzz about how AI is going to reshape the workforce for the past 3 years and layoffs due to “AI” have already started, the most...

Databricks acquires Quotient AI to power AI agent evaluations

Databricks Blog · 2026-03-11 · Evals

Databricks is excited to announce the acquisition of Quotient AI, an innovator in...

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

Hacker News · 2026-03-08 · Evals

Article URL: https://arxiv.org/abs/2603.03823 Comments URL: https://news.ycombinator.com/item?id=47295537 Points: 8 # Comments: 1

Show HN: Go LLM inference with a Vulkan GPU back end that beats Ollama's CUDA

Hacker News · 2026-03-08 · Evals

dlgo is an LLM inference engine written in Go. CPU path has zero dependencies beyond the standard library. GPU path uses Vulkan compute — no CUDA required.I benchmarked it against Ollama using the...

Ask HN: Has finding more competitors ever made you more confident?

Hacker News · 2026-03-08 · Evals

I was researching the personal finance market and initially found only a few obvious companies.After digging more, I found a much longer tail of startups I’d never heard of, including products...

Eval awareness in Claude Opus 4.6’s BrowseComp performance

Anthropic Engineering · 2026-03-06 · Evals

Evaluating Opus 4.6 on BrowseComp, we found cases where the model recognized the test, then found and decrypted answers to it—raising questions about eval integrity in web-enabled environments.

Evaluating Skills

LangChain Blog · 2026-03-05 · Evals

By Robert XuRecently at LangChain we’ve been building skills to help coding agents like Codex, Claude Code, and Deep Agents CLI work with our ecosystem: namely, LangChain and LangSmith. This is...

Agent Trace Evaluation with TruLens Scorers in MLflow

MLflow Blog · 2026-03-04 · Evals

Score agent plans, tool calls, and reasoning with TruLens GPA framework through mlflow.genai.evaluate().

LangSmith CLI & Skills

LangChain Blog · 2026-03-04 · Evals

We’re releasing a CLI along with our first set of skills to give AI coding agents expertise in the LangSmith ecosystem. This includes adding tracing to agents, understanding their execution,...

LangChain Skills

LangChain Blog · 2026-03-04 · Evals

We’re releasing our first set of skills to give AI coding agents expertise in the open source LangChain ecosystem. This includes building agents with LangChain, LangGraph, and Deep Agents. On our...

Benchmark Your Way to Better RAG and Agents:Tuning Vector Search with MLflow

MLflow Blog · 2026-03-02 · Evals

High-level summary: problems, approaches, and takeways for better RAG with MLflow

Evals Skills for Coding Agents

Hamel Husain's Blog · 2026-03-02 · Evals

Evals Skills for Coding Agents

You don’t know what your agent will do until it’s in production

LangChain Blog · 2026-02-26 · Evals

You can't monitor agents like traditional software. Inputs are infinite, behavior is non-deterministic, and quality lives in the conversations themselves. This article explains what to monitor,...

Multi-turn Evaluation & Simulation: Enhancing AI Observability with MLflow for Chatbots

MLflow Blog · 2026-02-24 · Evals

MLflow 3.10 introduces multi-turn evaluation and conversation simulation so you can score entire conversations, test agent changes with reproducible scenarios, and catch failures that only surface...

Agent Observability Powers Agent Evaluation

LangChain Blog · 2026-02-22 · Evals

You can't build reliable agents without understanding how they reason, and you can't validate improvements without systematic evaluation.

monday Service + LangSmith: Building a Code-First Evaluation Strategy from Day 1

LangChain Blog · 2026-02-18 · Evals

Learn how monday Service developed an eval-driven development framework for their customer-facing service agents.

OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments

Hugging Face · 2026-02-12 · Evals

MemAlign: Building Better LLM Judges From Human Feedback With Scalable Memory

MLflow Blog · 2026-02-03 · Evals

Introducing MemAlign, a new framework that aligns LLMs with human feedback via a lightweight dual-memory system, achieving competitive or better quality than state-of-the-art prompt optimizers, at...

Introducing DeepEval, RAGAS, and Phoenix Judges in MLflow

MLflow Blog · 2026-01-29 · Evals

Improve your agents using MLflow's extensive, industry-leading suite of high-quality LLM judges.

Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs

Hugging Face · 2026-01-27 · Evals

AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality

Hugging Face · 2026-01-21 · Evals

Designing AI-resistant technical evaluations

Anthropic Engineering · 2026-01-21 · Evals

What we learned from three iterations of a performance engineering take-home that Claude keeps beating.

LLM Evals: Everything You Need to Know

Hamel Husain's Blog · 2026-01-15 · Evals

LLM Evals: Everything You Need to Know

Demystifying evals for AI agents

Anthropic Engineering · 2026-01-09 · Evals

The capabilities that make agents useful also make them difficult to evaluate. The strategies that work across deployments combine techniques to match the complexity of the systems they measure. \n

Rapidly Prototype and Evaluate Agents with Claude Agent SDK and MLflow

MLflow Blog · 2025-10-15 · Evals

How to quickly prototype an agent using the Claude Agent SDK then instrument and evaluate it with MLflow

Selecting The Right AI Evals Tool

Hamel Husain's Blog · 2025-10-01 · Evals

Selecting The Right AI Evals Tool

AfriMed-QA: Benchmarking large language models for global health

Google Research · 2025-09-24 · Evals

Generative AI

Democratizing AI Safety with RiskRubric.ai

Hugging Face · 2025-09-18 · Evals

Beyond Manually Crafted LLM Judges: Automate Building Domain-Specific Evaluators with MLflow

MLflow Blog · 2025-09-15 · Evals

How to easily create custom evaluators that understand the semantics of your domain and automatically align with human experts

Writing effective tools for agents — with agents

Anthropic Engineering · 2025-09-11 · Evals

Agents are only as effective as the tools we give them. We share how to write high-quality tools and evaluations, and how you can boost performance by using Claude to optimize its tools for itself.

Building and Managing an LLM-based OCR System with MLflow

MLflow Blog · 2025-08-30 · Evals

Building GenAI tools presents a unique set of challenges. As we evaluate accuracy, iterate on prompts, and enable collaboration, we often encounter bottlenecks that slow down our progress toward...

A scalable framework for evaluating health language models

Google Research · 2025-08-26 · Evals

Generative AI

Assessment-focused UIs in MLflow

MLflow Blog · 2025-08-11 · Evals

In MLflow 3.2, we introduced the concept of assessments, which are the quality evaluations and trace annotations that are crucial for understanding and improving your AI applications. With the...