What to Look for When You Choose a Web Search Vendor

The architectural question behind every web search API decision

If your AI agents need web search, you are probably comparing vendors on the dimensions that show up in their docs: latency, coverage, pricing, freshness, ranking quality.

Those dimensions matter, but they're not what determines whether you regret your choice in eighteen months.

Every web search vendor sits on one of two architectural foundations. Some wrap an upstream SERP API, with large providers like Google's index doing most of the work behind the scenes. Others built their own crawler, their own index, and their own retrieval models from scratch. That choice bounds what they can offer you on every dimension that affects your agent downstream.

Data residency, retention policies, deployment surface, terms-of-service exposure, the compliance posture you can pass through to your own customers: these are not features your vendor can add later if you ask. They are properties of the architecture your vendor chose.

A vendor that wrapped has a fixed list of capabilities outside their control, and therefore outside yours; a vendor that built keeps those capabilities on the table. When a customer asks whether your AI product can run in their private cloud or on-prem, a wrap-based vendor cannot say yes at any price. Their retrieval depends on a SaaS endpoint they do not control. A built-from-scratch vendor can.

The choice between wrap-based vendors and built-from-scratch vendors is less a pricing comparison than a question about which conversations stay open for you as your product scales. The graphic above sketches the comparison. The rest of this post explains why this fork is the question behind every other question you should be asking the vendor.

What wrap-based providers can and can't offer you

Tavily, Exa, SerpApi, and Firecrawl all have components that fallback on large, traditional, search engines. We'll use Google as our example for simplicity.

They add a ton of value along the way: SDKs, ranking adjustments, agent-friendly response formats, integration ergonomics.

The reason their products ship fast is the same reason they pass certain constraints through to you: they do not control the index.

They are translating between your AI product and someone else's search infrastructure. That translation has three main consequences.

Data control. When you query a wrap-based provider, your queries travel through the upstream SERP API to reach the underlying index. Google sees them. Google's retention policies apply to that traffic. Google's logs include it. Your search vendor cannot offer you stronger data controls than Google offers them, because the data has to leave your vendor's stack to get to the index. If a customer of yours asks where their queries are stored and for how long, your wrap-based vendor's honest answer involves a third party they do not control.

Deployment surface. Wrap-based providers are cloud-only by definition. Their architecture is "call our SaaS endpoint, which calls another SaaS endpoint." They can't implement private cloud and on-prem deployments. Their retrieval depends on a third-party SaaS endpoint outside their control. When a procurement team asks whether your product can be deployed inside their network, "we are working on it" is the only answer a wrap-based vendor can give you, and the answer will most likely not change.

Terms-of-service exposure. When you build on a wrap-based provider, you have an indirect dependency on the upstream search engine's terms of service. The cleanest current example is Google v. SerpApi, filed in December 2025: Google is suing SerpApi for circumventing technological protection measures around the SERP. The case is unresolved. Why should you care? This category of risk is now made real. If you depend on a vendor that depends on Google's tolerance, you are exposed to whatever Google decides about that tolerance.

These are not adversarial observations. They are the structural consequence of one architectural choice. Wrap-based vendors make different products possible, and they will likely remain the right choice for some applications. They may not be right choice if you think customers will eventually ask any of these three questions.

What built-from-scratch providers can offer you

A vendor that built their own crawler, their own index, and their own retrieval models has the opposite shape. They paid the engineering cost of owning every layer of the stack, and the property they get in exchange is control. They can configure the system end-to-end because there is no upstream that gets to make decisions for them. The capabilities a wrap-based vendor passes through from upstream are the same capabilities a built-from-scratch vendor configures directly.

The same three categories, mirrored.

Data control. Queries stay in the vendor's stack. There is no upstream SERP API that has to see your traffic before the vendor can return results. Retention windows are a configuration the vendor can offer you, not a default they inherited. Zero data retention becomes a deployment option rather than a sales conversation that escalates to a different team. Per-customer retention windows are possible because the vendor controls the data layer end-to-end.

Deployment surface. Private cloud, on-prem, and controlled deployments are all options because the stack is the vendor's to deploy. On-prem and private cloud are technically distinct, and a vendor that built can offer the right one for the customer's compliance posture. The procurement question that ends with "we are working on it" from a wrap-based vendor ends with a deployment plan from a built-from-scratch one.

Compliance posture. A vendor that owns their stack controls the audit surface. SOC 2, HIPAA, and zero-data-retention deployments are architectural properties of a vendor in this position, because the vendor can answer questions about every layer of their system without pointing at someone else's documentation. The compliance posture you can pass through to your own customers is bounded by what your vendor can defend in an audit. Wrap-based vendors can defend their layer; built-from-scratch vendors can defend the whole stack.

This is not abstract. Victor Riparbelli, CEO of Synthesia, describes the choice as follows:

"As we evaluate solutions, data ownership and enterprise requirements are critical. Seltz's focus on building real web-knowledge infrastructure enables this, rather than simply wrapping technology around a search engine."

The shape of the choice, summarized:

Why this matters for any AI product

The case for caring about your search vendor's architecture has two angles. The first is about your customers. The second is about your unit economics. They converge.

Your customers' requirements are bounded by your vendor's architecture. Every B2B AI product eventually has a customer who cares about something on the capability list. Maybe a startup's first major enterprise contract requires zero data retention. Maybe a SaaS grows into a market with data localization rules. Maybe a team's largest customer shifts their procurement bar overnight after an internal security review surfaces a question nobody had asked before. These are not regulated-vertical edge cases. They are the normal trajectory of a B2B product that scales beyond its initial design center.

When that conversation happens, your answer is bounded by what your search vendor can offer you. If your vendor wrapped, the answers their upstream allows are the answers you can give your customer. If your vendor built, the answer surface is wider. The architectural decision your vendor made before you became their customer determines which conversations stay open for you. The Gauss Algorithmi piece on enterprise RAG architecture makes a related observation from a different angle: when a customer's data requirements are non-negotiable, a wrapper architecture rarely satisfies them, because the chain of custody breaks the moment the prompt leaves your vendor's network.

Margin compounds at scale. The query volume profile of AI products is structurally different from human search. A human searches a few times a day. An AI agent makes hundreds or thousands of retrieval calls per task. The cost-per-query math and the latency math both bend in directions that wrap economics weren't designed for. Every additional query is a marginal cost the wrap-based vendor pays to upstream and passes through to you. Every additional latency hop compounds across your agent's flow.

The architectural argument and the economic argument converge: wrap-based vendors are the right choice when query volume is small and procurement requirements are simple. Both of those conditions stop holding the moment your AI product gets traction.

How to evaluate a web search vendor

When you are evaluating a web search API for your AI product, the first question to ask the vendor is whether they wrapped an upstream SERP API or built their own index. Their answer to that question predicts the answers to most of the other questions you are going to ask: about retention, deployment, compliance posture, terms-of-service exposure, and the customer conversations you will have downstream of the integration.

The wrap-or-build answer is not the only thing that matters. Latency, ranking quality, coverage, pricing: these still matter. But they are evaluated against the architectural foundation, not in isolation from it. A fast wrapper is still a wrapper. A built-from-scratch system with worse coverage in your specific vertical may still be the right choice if your roadmap involves customers who will eventually ask the questions in this post.

Seltz is the built-from-scratch path. We are a knowledge provider for AI agents — we wrote the crawler, the index, and the retrieval models ourselves, in Rust, because the architectural property the rest of this post argues for is the property we wanted to be able to offer customers. If that aligns with the conversations you expect to have, check out the docs here.