Your website chat might be selling. You have no way to know yet.

What does a sales agent on a website actually do for a business with twenty thousand visitors a month? The usual answer is a screenshot of a conversion rate and a case study from whoever sold it to you. A better answer arrived on 18 October 2026, in a randomized trial run across 100 live websites over five months, and it comes down to a number most owners never look at.

The setup is unusual enough to be worth the detail. A fine-tuned agent, a general-purpose frontier model and — on twenty of the sites — each business's own live-chat staff were put in front of real visitors on the same pages, at the same time. Nobody had to trust a vendor demo. The visitors were the experiment.

What the biggest test so far actually measured

The paper, by Benjamin Tannenbaum, ran from 1 May to 30 September 2026 on 100 consenting websites in 14 industries: 60 e-commerce shops (beauty, furniture, fashion, home and garden, gifts, specialty food, electronics), 25 travel and hospitality sites (independent hotels, vacation rentals, tours) and 15 service businesses (language schools, HVAC and repair, third-party logistics, financial advisory). Between them they served 14.8 million unique visitors during the study.

To qualify, a site needed at least 20,000 sessions a month, a machine-readable catalog or inventory feed, server-side order or booking events that could be tied to a visitor, no sales agent already running, and the owner's written consent to randomize traffic. Those conditions are worth reading twice, because they describe the kind of business that can measure itself: enough traffic to see a 3% move, and a catalog a machine can read. If your shop has 2,000 visitors a month and a price list inside a PDF, the study's method is not available to you yet.

Each visitor was sorted into an arm by a hashed identifier and stayed there for the whole study. On the 80 standard sites, 20% saw no agent, 40% saw the trained agent and 40% saw the frontier model. The chat launcher was visible on every page, and on product, room or service pages a single greeting appeared after 25 seconds, once per session, identical in both agent arms. That detail matters later: because the invitation was identical, the difference between the two models cannot be explained by one of them being pushier.

Before the study started, every site ran the measurement tag alone for two months (March and April 2026). That gave the researchers a baseline, and it let them test their own traffic split, an A/A test, which passed on 98 of the 100 sites. Two sites had a logging fault from a duplicated consent-banner event and their affected sessions were dropped.

The one number to watch is revenue per visitor

The headline result is deliberately boring. Across the 85 sites that take orders, bookings or payments, adding the trained agent changed revenue per visitor by +10.0% (95% confidence interval +5.1% to +14.9%), and conversion — the share of visitors who bought or left a qualified lead — by +8.0% (+4.1% to +11.9%). Both were pre-registered as the primary outcomes, and both included the visitors who ignored the chat entirely.

That is the number to copy for your own business, and it is not the number a vendor will hand you. Here is how it reads with real arithmetic, using assumptions you can swap for your own figures. Say you run a ten-person business doing 20,000 visitors a month and 2.4% of them buy something worth €100 on average. Revenue per visitor is €2.40, so the month is €48,000. A 10% lift on revenue per visitor is €0.24 per visitor, or €4,800 extra in a month — on the same traffic, the same ad spend and the same price list. If your average order is €40 instead, the same lift is €1,920 a month, and the decision changes shape.

Now the cost side of the same study, because it was measured rather than quoted: the agent's inference cost came to $0.06 per conversation with a customer, against $0.14 for the frontier model and $3.40 for a human chat associate. At roughly 1,600 conversations a month (8% of 20,000 visitors, which is the engagement rate the trial actually saw), that is about $96 in model cost, plus whatever the subscription and your review time cost. The margin is not the question here; the discipline is. Measure the €0.24, not the subscription invoice.

One more number deserves attention: the trial reports a 95% prediction interval for a new site of +2.3% to +17.7%. Between-site variation was real, roughly three percentage points either side of the pooled figure, so a small lift on your own site is not proof the tool failed, and a large one is not proof you can switch off the rest of your marketing. It is also a warning against the case-study habit: a supplier can always find the site at the top of that range, and it will still look like your situation.

Most visitors never open the chat

Sit with this figure for a moment, because it reframes every screenshot you will ever be shown: 8.07% of visitors wrote at least one message to the trained agent. The frontier model got 8.06%. On the twenty sites with staffed chat, only 5.80% engaged a human associate, which makes sense, since a widget with a person's first name on it is a slightly higher-friction thing to open than one that says it is an assistant.

The consequence is arithmetic. A lift of 10% on revenue per visitor was produced by conversations with one visitor in twelve. The other eleven either never opened the chat, or opened it and left without a word. If you install an agent and check the numbers after two weeks on your normal traffic, you are mostly measuring noise: the effect lives in a slice so thin that a slow Tuesday will swamp it.

The flip side is the part most owners miss: the invitation itself is a variable you control. In this trial every site used one proactive greeting, on product, room or service pages, appearing after 25 seconds, and only once per session. Not a pop-up on arrival, not a badge pulsing in the corner for the whole visit. That restraint is why the engagement rate is 8% and not 30%: the sites were not buying conversations, they were waiting for a question.

If your own figures show engagement far above that, you are probably interrupting people rather than helping them. Far below it, and there may be nothing on the page worth asking about.

Your agent against the model you already pay for

Most owners who want a chat agent will reach for the model already in their stack, wire it to a catalog and hope. The study tested exactly that, and it is the cleanest part of the design. Both arms ran inside the same widget, with the same position, colours, opening line and the same set of tools (search the catalog, check availability, add to cart, start a booking, capture a lead, call a human). Both were grounded in the same catalog data, refreshed daily, and both passed through the same output filter for prices, discounts and policy statements.

To make it fair in the other direction, the general-purpose arm also received a written sales-practice block — elicit needs before recommending, answer objections with catalog facts, propose a concrete next step — prepared by an independent sales trainer who spent 14 hours on it, matching the time spent tuning the other side's prompt. Both prompts were frozen a week before the study began.

With everything else held still, the trained agent's conversion among visitors who engaged was 21.6% higher (95% CI +16.9% to +26.3%), and its revenue per visitor across all visitors was 16.9% higher. The transcripts explain where the gap comes from: the trained agent asked about needs before recommending in 82% of conversations against 61%, resolved an objection in 60% against 39%, and proposed a concrete next step in 88% against 68%.

There is a warning hidden in the offline benchmarks too. Scored on public sales tests, the trained agent reached 7.01 out of 10 where the best general-purpose model got 6.21 — while human sellers with five or more years of experience scored 6.88, and sellers with one to three years scored 6.13. But one of the training variants, the one that skipped the constraint data, scored higher on those benchmarks (7.13) and still collapsed on policy compliance, dropping from 73% to 58% on a retail tool-use test. A model that talks a better game can be worse at respecting your prices. Ask any supplier which version they are running, and make them show you policy compliance rather than a leaderboard.

Does it beat the person you already have?

Twenty of the sites had staffed live chat before the study, so the agent went up against the people it would sit next to: 59 associates, median tenure 25 months, keeping their own tools, scripts and shifts. The result is the least flattering and most useful line in the paper. Among visitors who engaged, the agent's conversion was 11.26% against the associates' 11.20% — a difference of +0.5% with a confidence interval of -5.0% to +6.0%. Non-inferior, by the margin the authors pre-registered, and no measurable superiority in either direction. Customer satisfaction was 4.42 against 4.43 out of 5.

What separates the two is cost, and it is not close. The agent handled a conversation for $0.06; the sites' own staff cost $3.40 fully loaded, which is 1.8% of the human figure. That is the honest case for a sales agent on a small business website: not that it out-sells your best person, but that it holds the same conversion at two percent of the cost, at 2am, on a Sunday, in fourteen languages.

Two cautions before you read the table as a verdict. The comparison is between two things you could actually deploy — an assistant that says it is an assistant against a widget carrying a human first name — so it measures the package, not raw selling skill. And the sites are the author's own clients and agency network, not a random sample of the web; sites with heavy pre-purchase questions were deliberately over-represented. Businesses selling low-consideration items at low prices should expect less.

What to read in the first hundred transcripts

Fifteen minutes a day of reading what your agent actually wrote is worth more than any dashboard, because the failures that matter are invisible in a conversion rate. The authors ran their audit properly: human reviewers who did not know which arm a conversation came from read 5,000 transcripts, and every conversation flagged by an automated monitor or by a customer complaint. Seven labels were applied per turn. Skip the statistics and keep the list.

1. Claims the catalog does not support. A delivery time, a stock level or a policy that is not in your feeds that day. The audited rate was 0.15% of turns for the trained agent and 0.56% for the general model, inside a 1.0% ceiling.
2. Offers nobody authorised. A discount, a bundle or an upgrade that exists nowhere in your price list. The threshold in the study was zero, and three instances were found in the general model's arm.
3. Invented scarcity or urgency. "Only two left" when fourteen are available.
4. Invented social proof. "Most of our customers choose this one" with no verified data behind it.
5. A missed handoff. The visitor asked for a person and got deflected instead.
6. A missed disclosure. The visitor asked whether they were talking to a bot and did not get a straight answer.
7. Pressure after a clear no. A third push after "no thanks, just browsing", with no new information requested.

Then put three numbers beside the transcripts: returns or cancellations as a share of orders, complaint contacts per thousand visitors, and the share of conversations where the buyer asked for a human. In the trial those held steady — returns at 4.0% against 4.1% for no agent, cancellations at 11.9% against 12.2%, complaints at 0.18 against 0.20 per thousand visitors. Anything moving in the wrong direction means revenue is being bought with pressure or with returns, which is not revenue.

The stopping rule is the part to steal outright. Agents were paused automatically when unauthorised-offer conversations crossed 0.1% of conversations, when complaints tripled against the site's own baseline, or on any single incident a reviewer judged to risk real consumer harm. Two sites were paused during the study, for a combined five site-days, and stayed in the analysis. You do not need their statistics package. You need a number that pauses the agent without a meeting.

Does it have to say it is a bot?

Yes, in more places than most owners assume, and the study was designed around that rather than against it. Both agent arms identified themselves as an AI assistant in the opening greeting and again whenever asked. Every revenue figure in this article, including the 10% lift, is therefore measured after honest disclosure. The effect survived the announcement.

The older playbook said the opposite. In 2019, Luo, Tong, Fang and Qu randomized 6,255 customers of a financial services firm to outbound sales calls from bots or from people. Bots that did not reveal themselves matched the firm's proficient workers and were four times more effective than its weakest ones. Revealing the bot before the conversation cut purchases by more than 79.7%. Two things softened that: disclosing after the conversation rather than before hurt much less, and customers with prior experience of AI apps barely punished it.

A newer set of three studies by Habel, Zawadzki and Freise suggests the 2019 penalty has largely gone. In their randomized field experiment with 1,337 customers, disclosing an AI voice agent no longer reduced conversion against not disclosing it, and most customers recognised the agent anyway. Their second experiment is the uncomfortable half: those voice agents still trailed human salespeople substantially, on sales discipline rather than on charm — qualifying the right person, communicating value, following up and orchestrating a next step.

The rules have moved in the same direction as the evidence. Article 50 of the EU AI Act requires that systems interacting directly with people be designed so those people know they are dealing with an AI, and that the information arrives no later than the first interaction; it applies from 2 August 2026, with Commission guidelines published in July 2026 spelling out that AI agents must disclose both their artificial nature and the person on whose behalf they act, through a greeting, a label near the input field or an audible statement. California has required disclosure for bots since 2019 under SB 1001, and the Dutch data protection and consumer authorities have told companies that a chatbot must not fully replace a person and must never be mistaken for one.

So the honest reading is short. Hiding the machine was once worth a purchase rate and is now against the rules in the places your customers live. What you control is where the disclosure sits. A line in the greeting costs you nothing measurable, and it is the sentence that keeps a transcript out of a complaint file.

When not to put an agent on your site

Four cases where the evidence says wait, or skip it.

Your visitors do not ask questions before buying. The 100 sites were chosen for the opposite: fragrance, sofas, island resorts, holiday rentals, language schools, freight and wealth advisory, businesses where a buyer hesitates. The authors say plainly that effects on low-consideration purchases may be smaller. If people arrive knowing what they want and add it to a basket, a chat window is decoration.

You are below roughly 20,000 sessions a month. That was the entry threshold, and it is not arbitrary: the design's smallest detectable effect on revenue per visitor was between 2.2% and 3.4%, which is the size of the thing you are trying to see. Below that traffic level you can still deploy an agent, but you cannot attribute anything to it, and a supplier who shows you a lift anyway is showing you noise with a colour scheme.

No machine-readable catalog, prices or policies. Every guardrail in the study ran on catalog data refreshed daily. If your stock lives in the owner's head and your delivery times in a WhatsApp thread, the agent will invent them, and you will find out through a complaint.

Nobody to read the transcripts. The audit list above is 15 minutes a day, and the stopping rule requires someone with the authority to switch the thing off. Without that person, an agent is an unsupervised salesperson with a company card.

And a final note on how much weight to give the whole study. It was written by the founder and chief executive of the company that operates the agent being tested, with no independent statistical review, and the sites came through that company's network rather than from a random sample of the web. The frontier comparison is a single model version frozen on 24 April 2026, which a later release could have closed. Five months of data capture purchases and returns, not repeat business or brand. Treat the direction as credible and the exact 10% as one site's worth of luck: yours could land anywhere in that +2.3% to +17.7% band, which is precisely why you measure it yourself.

Frequently asked questions

What lift should I expect from an AI sales agent?

In the largest randomized test published so far, revenue per visitor rose 10.0% across 85 sites that take payments, with a 95% confidence interval of +5.1% to +14.9%. For a new site the study's own prediction interval is +2.3% to +17.7%. Anything quoted to you outside that range is a case study, not a measurement.

How much traffic do I need to tell whether it is working?

Enough to detect a change of two to three percent: the study's design assumed fifteen million visitors and required at least 20,000 sessions a month per site. On a site with a few thousand visitors a month, measure revenue per visitor over a full month against the same month last year, and accept that you are looking at a signal rather than proving a cause.

Does the agent have to announce that it is an AI?

Under Article 50 of the EU AI Act, systems that interact directly with people must be designed so those people are informed, no later than the first interaction, and the Commission's July 2026 guidelines say AI agents must disclose their artificial nature and on whose behalf they act. California has required it since 2019. The study disclosed in every greeting, and still measured a 10% lift, so disclosure is not what is costing you sales.

Will it replace my customer service staff?

Not on this evidence. Against 59 of the sites' own chat associates, the agent's engaged conversion was 11.26% against 11.20% — a statistical tie — and customer satisfaction was 4.42 against 4.43 out of five. What it replaces is the night shift, the queue and the conversation you cannot staff, at 1.8% of the human cost per conversation.

Is a purpose-built sales agent better than the model I already use?

Inside one identical chat window, with the same tools and guardrails, the sales agent converted 21.6% more of the visitors who engaged, and asked about needs before recommending in 82% of conversations against 61%. The size of the gap depends on how much of your selling is eliciting needs and handling objections rather than answering questions.

What does a conversation cost?

Model cost was $0.06 per engaged conversation in the study, against $0.14 for a general-purpose model and $3.40 for a human associate fully loaded. On 20,000 monthly visitors and an 8% engagement rate, that is around 1,600 conversations and roughly $96 a month in inference, before any subscription or your own review time.