Deep dive

How to measure chatbot ROI on your website, honestly

Measuring chatbot ROI honestly means counting bookings, leads and staff time against real costs, then proving the gain with a split test. Here's how.

Key takeaways
  • Build a monthly ledger with three value lines, bookings, qualified leads and team hours freed, and subtract the full cost of the tool plus your own upkeep time.
  • Use a split test that shows the assistant to a random half of eligible sessions, because before-and-after comparisons mostly measure the season.
  • Count a conversation as value only when it produces a booking, a qualified lead or a resolved question that would otherwise have cost staff time.

The most common chatbot ROI calculation looks something like this: “We had 900 conversations last month. At five minutes each, that’s 75 hours saved. The bot paid for itself twice over.” It’s an appealing sum, and it is almost always wrong. A conversation isn’t a saved hour. Many of those 900 people would never have emailed you at all, and some received an answer that sent them away annoyed. Honest ROI starts by refusing to count conversations as value.

This piece sets out a method for measuring what a website assistant is actually worth. It covers the ledger, the attribution problem, the split test that separates the assistant’s effect from everything else that changed that month, and the metrics that look impressive but mislead. The worked numbers are illustrative, for a small clinic, but the method applies to almost any service business.

Start with a chatbot ROI ledger, not a dashboard

A dashboard shows you activity. A ledger shows you money. The difference is that a ledger has two sides: value created and cost incurred. Keep one per month, and keep it simple enough that you’ll actually update it.

On the value side, three lines usually cover it:

  • Bookings created through the assistant. Count the appointments the assistant booked, then multiply by the margin on each one, not the gross price. A $200 appointment with a $60 margin is worth $60 to you, not $200.
  • Qualified leads. People who gave contact details and match your criteria for a sale. Multiply by your close rate and the margin on what they buy. Be careful here, because a lead is only worth something once it converts.
  • Team time freed. Questions the assistant resolved that would otherwise have reached a person. Count the hours, multiply by what those hours cost you, and only claim the saving when you can show the time was actually redeployed or avoided.

On the cost side, include everything: the subscription, the setup hours, the time you spend each month maintaining content, and the time your team spends handling hand-offs. Many ledgers omit the upkeep and the hand-off time, which are often the largest costs after the first month.

Ledger line How to measure it Common mistake
Bookings from chat Tag each booking with its source when it’s created Counting all bookings during chat hours, whether or not chat was involved
Qualified leads Contact details plus a fit check against your criteria Counting every email address as a lead
Team time freed Hours of avoided work, times the true hourly cost Claiming hours saved that nobody can point to on a schedule
Tool cost Subscription plus setup and monthly upkeep hours Leaving out upkeep and hand-off time

A worked example

Consider a physiotherapy clinic with a busy website and a two-person front desk. In a typical month, the assistant holds 300 conversations. Here is how the numbers might work, all illustrative:

  • Appointments. 36 of the 300 conversations (12 per cent) end in a booking. At a $32 margin per appointment, that’s $1,152 of contribution.
  • Leads. 40 conversations produce contact details that match the clinic’s criteria for a course of treatment. About 15 per cent become paying clients, so six clients. At roughly $60 of margin each, that’s $360.
  • Team time. Around 150 of the conversations are routine questions that would otherwise have been emailed to the front desk and answered at about four minutes each. That’s ten hours. At a loaded cost of $25 an hour, it’s $250, and the clinic can show the time was redeployed to rebooking follow-ups.
  • Costs. A monthly subscription of $150, plus about two hours of upkeep, roughly $50 at the same rate. That’s $200 in total.

Net value after costs: $1,152 plus $360 plus $250, minus $200, which comes to $1,562 for the month.

The figure is useful, but the method is what matters. Notice that the team-time line is the most open to challenge, because it only counts if the saved time turned into something real. Notice too that the leads line depends on a close rate you have to measure, not assume. Each line has a weak point, and the honest ledger says where it is.

Monthly ledger, illustrative clinic $1,152 $360 $250 $1,562 Appointments Leads Team time Net after costs
An illustrative monthly ledger. Each value line carries its own assumption, and the net figure is what remains after the tool and upkeep costs.

Attribution: the booking that came later

Attribution is where most chatbot numbers go wrong. A visitor chats with the assistant on Monday, thinks about it, and books by phone on Thursday. Did the chat cause the booking? Possibly. Did it merely help? Maybe. You can’t always tell.

Handle this with a range rather than a single number. Set two thresholds. The lower bound counts only bookings where the chat session is recorded as the source: the visitor booked inside the assistant itself. The upper bound adds bookings made by people who chatted within a set window before booking, say seven days. Report both. If the true number sits somewhere between them, you’ll know roughly how much to trust the headline.

Tag each booking with its channel at the moment it’s created. Reconstructing it later from memory, or from a staff member’s guess about where the call came from, is unreliable. A simple “how did you hear about us” field on the booking form, with a chat option, costs almost nothing and improves the data a great deal.

Applied to the clinic example, the 36 bookings were tagged to a chat session, which gives $1,152 of appointment contribution. If the clinic also counts phone bookings from people who chatted within the previous seven days, the upper bound might be around 44 bookings, or $1,408. Show both figures in the ledger. A decision to keep the tool should still make sense at the lower number. If the case only works at the upper one, collect more data before deciding.

Run a split test, not a before-and-after

The most tempting comparison is between last month and this month. Bookings went up after the chatbot launched, so the chatbot must have worked. The trouble is that a dozen other things changed too: the season, a campaign, a competitor closing, a price tweak, a holiday. A before-and-after comparison mostly measures those changes.

A split test isolates the effect. Randomly assign eligible sessions to two groups for several weeks. The treatment group sees the assistant; the control group doesn’t. Because assignment is random, the groups are alike in everything else, and any difference in outcomes can reasonably be credited to the assistant.

Here is an illustrative result. Over six weeks, the clinic has 9,000 eligible sessions, split evenly, 4,500 in each group. The control group produces 162 bookings, a rate of 3.6 per cent. The treatment group produces 185, a rate of 4.1 per cent. The difference is 23 extra bookings, a relative lift of about 14 per cent.

Before you believe that, run an A/A check. Split identical traffic into two groups with no assistant at all, and measure how much the rates differ by chance. If the two identical groups routinely differ by a percentage point or more, a half-point gap in your test is noise. Only a difference bigger than the normal wobble is evidence, and that’s the test to apply to any split result, including one you’ve run yourself.

Tag the source at creation: record the channel the instant a booking or lead is made. Data captured at the time is trustworthy; data reconstructed a month later from memory is not, and it's the first thing an accountant or partner will challenge.

Metrics that look good and mislead

Some numbers flatter the assistant. Use them as diagnostics, not as proof of value.

Metric Why it misleads A better version
Conversations started Counts curiosity as much as need Conversations that end in a booking, lead or resolved question
Containment rate (no human needed) A high rate can mean visitors gave up Containment plus a 48-hour check: did they book, or come back?
Messages per conversation Shorter isn’t always better Time to a complete answer, and whether it was right
Thumbs-up ratings Only a self-selected few click them A random sample of conversations, read in full
Hours saved Only counts if the schedule actually changed Hours moved to other work, or a hire you didn’t need to make

The containment rate deserves particular caution. A visitor who types a question, gets a confident but incomplete answer, and leaves has been “contained” in the metric and lost in reality. Always pair containment with a follow-up check.

Don't book hours saved as money until someone can point to it: a deflected email only saves cash if the schedule changed, a shift was cut, or a hire was deferred. Otherwise it's reclaimed time, and it belongs in the notes, not the profit line.

Count the costs you forget

The cost side of the ledger is where honest accounts get uncomfortable. Three costs usually go missing.

Setup time is the first. A few hours of writing content, mapping rules and testing feels small until you add up the hours of the person who did it. Spread it across the expected life of the tool and include it.

Upkeep is the second. Prices change, hours shift, policies get rewritten. Someone has to update the assistant’s source material, and that time recurs every month. In the clinic example it was two hours, which is a small but real line.

The third is the cost of a bad answer: a visitor who was told something wrong, a complaint, a booking that went sideways. These are hard to put a number on, but you can track them. Count complaints that trace back to an assistant reply each month, and review the worst one in detail. A single costly error can outweigh several months of small gains, and the ledger should make that visible.

A monthly review routine

Once the ledger exists, a short monthly routine keeps it honest. It takes about an hour once it’s established.

  1. Update the value lines from tagged data, and note the attribution range for bookings.
  2. Refresh the upkeep and hand-off hours, and recalculate the net.
  3. Read a random sample of ten or so conversations in full, looking for wrong answers and missed opportunities.
  4. Check the split test, if one is running, and compare the result with the A/A baseline.
  5. Decide one change to make next month, whether to content, rules, or the handoff, and write down what you expect it to do.

Writing down the expected effect is an underrated habit. It turns the monthly review into a learning loop rather than a reporting ritual, and it protects you from crediting every change with success after the fact.

What good looks like after six months

A realistic picture after six months looks less dramatic than the pitch. The assistant handles a steady share of routine questions and books a meaningful number of appointments. Some visitors use it; many don’t. The team spends a little less time on repeat questions, and the time is used on rebooking and follow-up. The ledger is positive but not spectacular, and the split test shows a lift you can defend rather than one you have to argue for.

That’s a good outcome. It’s also one you can only be sure of if you measured from the start, and if you were willing to count the costs and the failures alongside the wins. A chatbot that makes your site more useful, and that you can prove does so, is worth keeping. One that merely looks busy in a dashboard is worth questioning.

If you want to run this method on your own site, Voxy’s free 14-day trial lets you start with a single page and see what the assistant does with real questions, no card required. You can start the trial from the homepage, review how the pricing is structured, and read more about the conversion side of the story in how instant answers lift website conversion. Whatever you choose, measure it the same way: a ledger, a split test, and a monthly review.