Introducing the Dynamic News Search Benchmark

Today we are releasing the Dynamic News Search Benchmark, a daily evaluation of search providers on time-sensitive news retrieval. The benchmark has been running internally at Seltz for several months. We plan to publish the evaluation data and the prompts that generate it, so anyone can reproduce the methodology against their own search system or any provider they want to test.

We are leading with the methodology rather than the leaderboard for a specific reason. Seltz performs well on this benchmark; that is part of why we decided to publish it now. But the methodology is the thing we want the technical community to look at first. Unlike other benchmarks, DNSB is built so that we have no control over what events are tested, what queries are generated, or which answers count as correct.

Why is that important? Benchmarks should be impossible to bend toward one providers strengths. To achieve this, we gave up control over the inputs and designed the system to be tested on new data every day.

The rest of this post is about how the benchmark is built, why each design decision matters, and what running it daily has revealed. Full competitive results, with named providers, will follow in a separate post once we are confident the evaluation has stabilized.

The problem with benchmarks

Every major search provider has published its own benchmark. Tavily, Exa, Firecrawl, and others all have their own standard for how they measure up against competitors.

Benchmarking is an important tool. But a problem arises when the methodology is designed around a provider’s strengths rather than concrete agent behavior. Even when the methodology is documented and the intent is honest, the evaluation can drift toward what the vendor’s system happens to be good at. That changes how results translate to how these systems perform in production.

There are two ways this drift shows up.

Static benchmarks become optimization targets

A static benchmark, in machine learning terminology, becomes something to overfit. After a benchmark has been public for a few months, companies (and academic researchers) start optimizing their systems against it. They are not training on the evaluation set; the data is held out. But the published numbers are the target, and over enough iterations, systems converge. Recent benchmarks of search APIs for agents have found the top providers statistically indistinguishable on aggregate scores. That is what overfitting to the evaluation data looks like in practice.

Data drifts toward vendor strengths

The provider selected the queries. The provider defined what counts as a relevant result. For a knowledge provider serving AI agents (where the relevance question is whether a returned document helps the agent answer correctly, not whether a human would click on it) that selection control matters more than it does for general-purpose search. An agent’s downstream task is concrete and verifiable. A benchmark that drifts from that task stops measuring the thing the user actually cares about.

We built our benchmark to address both problems at the structural level. The next section walks through how.

How the benchmark is built

The construction pipeline runs once a day and produces an evaluation set for that day.

The pipeline has six stages: news fetching, question generation, query generation, expected answer extraction, evaluation sample compilation, and search and scoring.

Four of those choices do most of the work.

Where the news comes from

The first stage fetches recent news, using Wikipedia’s Current Events portal as the source. Each day, Wikipedia contributors post ten to thirty items summarizing events as they break, organized into ten categories.  Each item is a short complete sentence in present tense, with one or more links to the news articles it references.

We picked Wikipedia for three properties of the source itself:

1. Content is human-curated, which means no AI-generated summaries in the source data.

2. Posting policies are strict; items have to be event-focused, sourced to reliable reporting, and free of opinion.

3. The license is permissive (CC BY-SA 4.0), so we can use the content without restriction.

There is a fourth property that does most of the work for our purposes: the contributor pool is open. Anyone can post items to Current Events, which means we have no control over what events make it in or how they’re written.

The trade-off is coverage. Ten to thirty items per day is small relative to the full news cycle. The category restrictions filter out some kinds of stories. But the coverage cost is the unpredictability we want. We’ll return to this point in the next section.

From news to evaluation data

A news excerpt is not yet something you can run a search benchmark against. To turn it into a query, we generate three things from each excerpt:

• a natural language question whose answer is contained in the excerpt

• a web search query that an agent would actually send to retrieve documents that answer the question

• the expected answer extracted from the excerpt itself.

The three artifacts are not generated in parallel. They have a specific dependency order, and the order is what makes the queries representative of what real agents produce.

We start by generating the question. An LLM, given the news excerpt, produces a natural language question that a user might ask and whose answer is contained in the excerpt. The question is then handed to a second LLM acting as an agent — one that is forced to call a search engine before it can answer. The query sent by the agent to the search engine is the query the benchmark uses. The expected answer is extracted from the excerpt by a third LLM, given both the excerpt and the question, so the extracted answer is by construction the answer to that specific question.

The forcing function is the methodological move worth pausing on. The query is not a text transformation of the question; it is the query an agent actually produces when it has to retrieve information from the web before answering. Different questions produce different queries because real agents shape queries to the search engine, with the right named entities and the structure retrieval responds to. That is the kind of query a production agent issues, which is the kind of query we want to evaluate search providers against.

The output of these stages is a tuple: <question, query, expected answer, gold URLs>.

The gold URLs are the news articles the original Wikipedia excerpt linked to. Those URLs matter for one thing only: they tell us which sources the Wikipedia editor considered authoritative for that event. They are not used for scoring. Scoring is based on whether a provider’s returned documents support the expected answer, regardless of which specific URL the document comes from.

How relevance gets graded

This is the design decision that differentiates our methodology from most providers. A search result is graded relevant if and only if an LLM, given the question and the search result as context, can produce the expected answer. If the model returns the wrong answer or abstains, the result is graded non-relevant. Binary grading is what makes daily refresh feasible.

The traditional way to build a search benchmark is to send the queries to several search engines, pull the results, and have human annotators label each one on a graded scale: very relevant, somewhat relevant, not relevant. To control for annotator bias, you usually want at least three annotators per result. Doing that at web scale, every day, on a fresh set of queries, is unfeasible unless you are maybe Google.

Binary, automatable grading also has a property that graded human relevance does not: it ties the relevance label directly to the downstream task. A result is relevant if it lets the agent answer correctly, and not relevant if it does not. We are measuring task success, which is what the user of an agent actually cares about, rather than a proxy for it.

The grading is verifiable in a way graded relevance is not. The expected answers are short and specific: usually a name, a number, a date, a place. It’s hard to mark a wrong answer correct or a right answer wrong by accident. There is one failure mode we have spent a lot of time on: the LLM guessing answers from topic familiarity rather than from the document in front of it. We describe that and the mitigations we have in place in the limitations section below.

When we send queries to the search providers under evaluation, we filter out Wikipedia results from the response. Otherwise, Wikipedia’s own page about the event would trivially answer every question, and every provider would score perfectly on the items where Wikipedia happens to be in their results. The filter applies to the Current Events pages and the per-day Wikipedia pages that the questions are generated from. Whether to filter all of Wikipedia is something we are still working through.

What we measure and what we don’t

Once we have relevance labels for each result, we compute four retrieval metrics per query and one end-to-end metric. The retrieval metrics are Precision@1, Hit Rate@5, Hit Rate@10, and Reciprocal Rank.

Precision@1 asks whether the first result is relevant. For a fact-seeking question, the first result is what an agent will reach for first, and a relevant first result means the agent does not have to read further to answer correctly.

Hit Rate@5 and Hit Rate@10 ask whether at least one relevant result appears in the top five or top ten. They tell us how aggressively a provider’s results have to be filtered before the agent finds something useful.

Reciprocal Rank is the inverse of the rank position of the first relevant result. It is a more granular signal than Hit Rate, sensitive to where the first relevant document lands in the ranking.

The end-to-end metric is Accuracy: the fraction of questions an agent answers correctly when given all the search results from a provider as context. This is the metric closest to what an agent operator actually cares about.

We deliberately do not use Recall or NDCG. Both metrics need information we do not have at web scale. Recall requires knowing how many relevant documents exist in total for a given query. At web scale, we cannot enumerate them. NDCG requires both the total count of relevant documents and a graded relevance label for each one, neither of which we have. These metrics are the right choice for evaluating a closed retrieval set where you control the document pool. They are not the right choice for evaluating what an agent retrieves from the open web.

Each of these four choices was made for technical or operational reasons we just walked through. Together, they have a structural consequence that does not depend on us being honest about it. We will look at that next.

What makes it fair

Fairness in a benchmark is usually treated as a question about intent: did the authors try to be fair, did they avoid cherry-picking, did they document the methodology. Those are reasonable questions, but the answers are not verifiable, and a benchmark built in good faith can still drift toward what its authors are good at. Structural fairness is different. It does not depend on intent. It is a property of how the benchmark is constructed, and it can be checked by anyone who reads the methodology. The Dynamic News Search Benchmark has three structural properties that produce fairness without us having to vouch for it.

Unpredictability. The evaluation data refreshes every day from events nobody can predict. A court ruling, an attack, an election result, a natural disaster: we cannot know what tomorrow’s news will be before Wikipedia’s contributors post about it. An item enters today’s evaluation set the same day the underlying event happens. There is no public test set that has been out for two years for systems to optimize against, and no fixed target to optimize toward.

This does not mean a system cannot do well on this benchmark. It means doing well requires actually retrieving and ranking news from the live web, not pattern-matching against a known evaluation set. You can still do well on a predictable benchmark. It is just harder to do well on an unpredictable one through any path other than genuinely solving the underlying retrieval problem.

Independence. No component of the construction pipeline touches Seltz’s system. The news source is Wikipedia. The questions, queries, and expected answers are generated by a third-party LLM. The relevance grading is done by a third-party LLM as well. The gold URLs the Wikipedia editors linked to point at news articles on the open web; sometimes they point at sources we do not crawl. I found one item where the linked article was an Italian local newspaper that no one at Seltz had heard of. It may not even be in our index. The benchmark surfaces sources we do not control and does not penalize us for it; it just measures whether providers find documents that answer the question.

Independence has a less obvious dimension. We do not write the excerpts. Whatever phrasing patterns or stylistic conventions exist in them (Wikipedia's editorial guidelines produce a fairly consistent house style) exist independently of Seltz. There is no authorial signal in the source that we could optimize against even if we wanted to, because we have no editorial control over how excerpts are written.

Automation. The benchmark is fully automated end to end. No human reviews the items, writes the questions, grades the results, or chooses what to include. This is what makes daily refresh feasible. If grading required human annotators, daily refresh would be impossible at this scale, and without daily refresh, unpredictability collapses back into a fixed target.

This is the part that connects the three properties. Automation is not just an operational convenience. It is what makes unpredictability possible in the first place. A benchmark that has to be hand-graded can be unpredictable in principle, but in practice it will not be refreshed often enough to stay ahead of optimization. Daily refresh is what keeps the target moving, and binary grading is what makes daily refresh cheap enough to do.

These three properties are visible in the methodology. Anyone who reads the construction pipeline can verify them; the reader does not have to take the fairness claim on trust.

Of course, a benchmark only matters if running it teaches you something. The next section is about what the daily run actually shows.

What the benchmark reveals

We have been running the benchmark internally for several months. Most of what it has been useful for is not ranking competitors but finding gaps in our own system.

When the benchmark grades a Seltz result non-relevant for a question we should be able to answer, it surfaces a specific failure: an article we crawl from a domain we crawl that our retrieval did not return for that query. Sometimes the article is in our index but ranked too low. Sometimes the article never made it into the index at all because of an issue further upstream in our pipeline. Either way, the failure is precisely diagnosable. We can trace it back through our system and fix the underlying cause. The benchmark, used this way, is a useful diagnostic tool for issues that would otherwise be hidden.

Running it against other search providers has shown a few things worth naming, even with names anonymized in this post.

The first is that some providers do not appear to ingest news in the way the benchmark requires. They return zero relevant results for most queries on most days, not because their retrieval is poor but because the articles the questions are asking about are not in their index at all. This is a coverage gap, not a ranking gap, and the benchmark distinguishes between the two cleanly: a provider with poor retrieval will still find some relevant documents some of the time; a provider that is not ingesting fresh news will return nothing.

The second is that retrieval latency varies far more across providers than the public benchmarks suggest. The fastest and slowest providers in our evaluation are separated by up to 30x. For a single query that gap is small in absolute terms. For an agent that issues multiple search calls per task, the gap compounds quickly into the response time the end user actually sees.

We chose to publish the methodology now in part because the results we get on it are good. A vendor only publishes a benchmark when the numbers favor publication, and we are no exception. The check on that bias is everything we walked through in the previous section: the methodology was built before we ran it, the data refreshes from a source we do not control, and the prompts and evaluation set will be public. Whether or not Seltz happens to be on top on any given day, the benchmark itself is not engineered around our strengths. The full competitive results, with named providers and longer-running data, are coming in a separate post.

What the benchmark can’t measure, and what’s next

The current version has its limits.

The sample size is small. Ten to thirty items per day, drawn from Wikipedia’s editorial pool, leaves obvious topic gaps and means a single day’s score can move noticeably on a thin sample. The queries are simple and fact-seeking: questions with a name, a number, or a date as the answer. They are not the open-ended queries an agent doing deeper research would issue.

The biggest methodological limit is one we have spent real time on. The LLM that grades relevance can guess answers from topic familiarity rather than from the document in front of it. If the news for a given week is largely about the same ongoing story, the parties and dates involved appear in many documents. The grader, asked whether a specific document supports a specific answer, can sometimes return a plausible answer that did not actually come from that document. We mitigate this with prompt instructions that push the model toward abstaining when it is not certain (“better to say I don’t know than to guess”), and by anchoring the questions to specific dates so that a document about an adjacent day’s events does not match. It is better than it was when we started. It is still not perfect.

The next version of the benchmark addresses several of these limits at once. We are working on adding sources beyond Wikipedia, possibly with random source selection per day to push unpredictability further. With more sources, we can generate more evaluation data per day, closer to one hundred items than to twenty. We are also moving toward harder queries, ones where the answer is broader than a single fact and where the agent has to do more work. Verifying broader answers is genuinely harder than verifying short ones, and we do not yet have a clean approach to it.

This is the 0.1 version. We are publishing it because the methodology holds up to scrutiny and because the results it produces are useful enough internally that we think they will be useful to others. Once we release the prompts and the evaluation data, anyone will be able to run it against their own search system or any provider they want to test. We will keep iterating on it in public.