This is a real report from a real hands-on test of a hosted AI customer-support product: one unpaid 70-minute session on its public trial, run before this service existed. It shows the format and the depth of a findings report; a paid audit runs the same method over about two weeks, with many more runs of the scenario. The vendor and product names, AI agent names, legal entity, hostnames, account and object IDs, exact prices, distinctive interface wording and the tester's details have been removed or generalized. Findings, severities and timestamps are as recorded. Screenshots are referenced by number but not included here.
Report
Vendor, product
Vendor A, Product A. A hosted AI customer-support platform: an AI agent answers from a knowledge base, then hands off to a human teammate in a shared inbox.
Session
About 70 minutes, 9:01 PM to 10:10 PM local time, on the self-serve trial (top plan, time-limited). One admin seat.
Test data
A fictional coffee roaster with a made-up FAQ covering shipping, returns and hours, with no real customer data and no real channels connected.
Customer side
The vendor’s hosted web chat page, reset to a brand-new visitor for each clean run.
Evidence
Numbered screenshots (01 to 25) and a minute-by-minute raw timeline. The tester’s visitor IP, which appears in the agent view, is redacted throughout.
Summary
The main scenario works, but only after four undocumented configuration steps, and every misstep along the way reaches the customer as silence with no warning to the admin. The AI consent notice promised in the privacy policy never appeared. When the AI does answer, its answers are accurate and grounded. 3 High, 4 Medium, 1 Low.
Severity key.High: breaks the main scenario, or contradicts a public commitment such as the privacy policy, in normal use. Medium: degrades the experience or reliability; a workaround exists. Low: cosmetic or minor friction.
Scenario result
Intended flow: the customer asks a question, the AI agent answers from the knowledge base, and a human teammate takes over when the AI can’t help or the customer asks for a person.
It does work, but only after four non-obvious configuration steps:
Insert an “assign AI agent” step in front of the default human step.
Let the human step accept chats already assigned to an AI agent, not just unassigned chats.
Add “talk-to-a-human button clicked” as a second trigger inside the same automation, wired to the human step.
Keep everything in one automation. A second automation silently stops the first.
With that setup the AI answered 6 of 8 questions the FAQ covers. When it handed off, or the customer pressed the button, the chat reached the human inbox.
High F1. Failed routing leaves the customer in silence, with no warning to the admin
What happens: five separate configuration states left the customer with no reply, no “please wait” and no handoff (9:16 PM, 9:18 PM, 9:26 PM, 9:30 PM/9:31 PM, 9:51 PM).
Cause (a), confirmed: the AI step’s fallback, “continue to the next step”, does nothing if the next human step only accepts unassigned chats, which is how the default human step is set. A chat held by the AI counts as assigned to an AI agent, so the human step never picks it up. Adding that status to the human step fixed it (changed 9:37 PM, confirmed 9:38 PM).
Reproduce:
In the default automation, insert an AI agent step before the default human step, with its fallback set to continue to the next step.
Leave the human step on its default filter (unassigned chats only).
Open the hosted chat as a new visitor and ask something the knowledge base doesn’t cover.
Expected: the AI hands the chat to a human. Actual: no reply of any kind, and the chat stays in the AI queue.
Cause (b), confirmed: enabling a second automation placed above the default stopped the default from running at all. The 9:51 PM chat sat unassigned, with the agent view reporting that the conversation was not assigned (screenshot 19). Moving the default back to the top restored routing (9:53 PM).
Reproduce:
With routing working, create a second automation, enable it and place it above the default.
Start a chat as a new visitor.
Actual: the chat sits unassigned, and neither the AI nor a human answers.
Warning to the admin: the builder gives none in either case.
Evidence: screenshots 10 to 13 and 19; timeline 9:16 PM to 9:31 PM and 9:51 PM.
High F2. The talk-to-a-human button does nothing on its own
What happens: enabling the AI step’s “show a talk-to-a-human button” option adds a live-support link to the chat window. Clicking it sends a POST to the vendor’s transfer-to-human API endpoint, which returns 200, but the chat stays with the AI and the customer sees no change. Reproduced 4 times: with the human agent set to Invisible, set to Online, and with a separate “button clicked” automation placed below the default.
Reproduce:
Enable the button on the AI step, leaving the rest of the default automation unchanged.
Open the hosted chat as a new visitor, with browser DevTools open on the Network tab.
Click the live-support link.
Observe: the transfer request returns 200. The chat stays with the AI, and nothing changes for the customer.
What makes it work: adding the same “button clicked” trigger inside the default automation, wired to the human step. It worked on the first try (10:00 PM, screenshot 22).
Warning to the admin: nothing tells the admin the button needs this extra wiring.
Evidence: screenshots 17, 21 and 22; DevTools capture of the 200 response.
High F3. AI consent and disclosure don’t match the privacy policy
The policy says: AI features run on a named third-party model provider. Conversation messages, files and AI history are sent to that provider. A clear consent notice, with explicit consent, comes before first use, and consent can be withdrawn from the Settings page.
Observed: no consent notice appeared at any point while creating two AI agents, using the builder preview or running live chats. The Settings menu has no AI or consent option (checked 10:06 PM).
End customers: the chat widget shows only the AI agent’s display name. Nothing tells the customer that replies come from an AI, or that their messages go to a third-party model provider.
Product UI: no model or provider is named anywhere in it.
Reproduce:
In a new trial workspace, create an AI agent and attach a knowledge base.
Use the builder preview, then run a live chat from the hosted chat page.
Watch for a consent prompt at each stage, then look for an AI or consent option under Settings.
Evidence: the policy’s AI section, quoted verbatim in the client version; timeline 10:06 PM.
Medium F4. The handoff experience is poor for the customer
The AI doesn’t wait for an answer: it asks whether the customer would like to be connected with a teammate, then transfers without waiting for a reply (9:38 PM).
No message on transfer: the widget header silently changes from the AI agent’s name to the admin’s role title, and the live-support link disappears (screenshots 24 and 25).
Silence while waiting: after the transfer, customer messages get no reply until a human answers (9:39 PM to 9:43 PM). There is no auto-acknowledgement and no expected wait time.
Unnamed human: by default the human teammate appears under the role title with a blank avatar, even when signing replies by name (screenshot 15).
Positive: the human’s reply reached the customer within 4 seconds, and the inbox flags chats awaiting a first response in red (screenshot 14).
Medium F5. Knowledge retrieval is inconsistent
Hit rate: with the second AI agent (from 9:38 PM), 6 of 8 live questions the FAQ covers were answered correctly. Both misses were the first message from a fresh visitor, with no other factors involved.
The same question both ways: the shipping-time question failed at 9:53 PM and was answered correctly at 10:09 PM. The returns question was answered at 9:38 PM and failed at 9:59 PM.
Inconsistent misses: on a miss, the chat sometimes went straight to the human with no AI message at all (9:53 PM and 9:59 PM; screenshot 20 shows assignment and transfer in the same second). A question outside the FAQ at 9:38 PM got an “I’m not sure” reply first.
Retracted: my earlier hypothesis that Q&A entries are ignored was wrong. A Q&A-only knowledge base answered correctly 5 minutes after creation (10:08 PM, screenshot 23). The first AI agent went 0 of 2 before a document was added, but that sample is too small to blame Q&A.
When it does answer: replies are accurate, short and grounded. The returns answer, for example, gave the FAQ’s exact window and exchange rule. I saw no invented prices or policies.
Medium F6. The builder preview doesn’t match live behavior
At 9:26:10 PM the agent preview said it was not sure about shipping costs, one second after the live chat answered the same topic correctly. The preview also took about 15 seconds per reply.
Reproduce: ask the same FAQ question in the builder preview and in a live chat within the same minute, and compare the answers.
Medium F7. Automation management traps
Created switched off: new automations are saved disabled, with no notice (9:50 PM, screenshot 18).
Order matters, undocumented: the order decides which single automation runs (F1, cause b), and the interface doesn’t explain this.
Orphaned chats: removing an AI agent from the flow leaves its open chats parked in the AI queue. The first agent’s two conversations stayed there, with no prompt to reassign them.
Low F8. Interface defects
Knowledge base name: limited to 30 characters and silently truncated (9:07 PM, screenshot 03). An ampersand in the name breaks the page URL and breadcrumb (10:03 PM).
Translation glitches: garbled helper text in the Q&A editor, a label with a number fused into the words, and a non-English list separator in saved condition lists.
No confirmation on destructive edits: clicking a workflow link deletes it instantly, and Escape discards an unsaved panel.
Unreliable first interactions: the first click after a page load is often ignored. The create-knowledge-base dialog closed itself on first interaction 3 out of 4 times. The chat page loaded blank once (it rendered on reload).
Message shown twice: one customer message appears twice (9:26 PM and 9:27 PM). Not yet attributed: it may have been resent by hand during testing.
Data handling and hosting facts
Recorded as observed. Anything that could not be verified is marked “not asserted”.
Chat widget naming: the widget’s iframe name, its local storage keys and its visitor cookie all carry the name of a different customer-service platform, based in another jurisdiction. The team profile includes a field for a messaging app popular in that region. Not asserted: the corporate relationship, or where data is hosted.
Hosts seen from the customer-facing chat page: the vendor’s hosted chat domain; the vendor’s static asset domain for the widget; an API gateway on a third, differently named domain, which carries the transfer-to-human call; an icon font from a third-party CDN in another jurisdiction; a web analytics beacon from a large CDN provider.
Agent view: shows each visitor’s IP address, city, OS and browser by default.
Retention: the privacy policy gives no retention periods.
Pricing and limits
From the vendor’s public pricing page, read at 10:05 PM. Exact prices and allowances are removed from this sample; the client version lists them.
AI tier: AI agents need the third of four plans, about four times the price of the entry paid plan. It includes a monthly allowance of AI replies, with extra replies sold in blocks.
Lower tiers: the free plan (capped at a handful of conversations) and the entry paid plan include no AI at all.
Integrations: webhooks and API access are on the top plan only, at a custom price.
Trial: a few days at the top plan, with a small gifted AI message allowance and a conversation cap. Usage against these limits could not be verified, because the order page did not open.
Workspace state left behind
AI agents: the second agent is active. The first is out of the flow but not deleted.
Automations: the default automation now runs AI then human, with two triggers. A separate test automation for the button exists and is switched off.
Knowledge bases: two test knowledge bases, one with Q&A entries plus a document and one with Q&A entries only.
Status: the admin seat is left set to Online.
Questions for the vendor
Which model and provider answer customers, and do customer messages leave your infrastructure? Where is conversation data hosted, and for how long is it kept?
Where is the AI consent notice your privacy policy describes, and where is the Settings option to withdraw consent?
Is “only the first matching automation runs” intended? Why does the talk-to-a-human button need a separate trigger?
What is your relationship to the platform whose name appears in the widget’s code and cookies?
Want a report like this on your product?
Tell me the product and the scenario, and I quote a fixed price for the same kind of report.
