How an audit runs.

You tell me the product and the one scenario your customers run most. I sign up like a stranger, set it up from the docs alone, run the scenario from a clean state until it breaks or holds, and keep a network capture running throughout. About two weeks later you have a findings report, a short list of questions a careful buyer would ask you, and one retest round for whatever you fix.

We agree the scenario in one paragraph, written the way a customer would run it. We agree the account I will use, the dates, the price and what is out of scope. That paragraph is what the report is measured against, so it is worth ten minutes of argument now.

I also read your public docs and privacy policy before I touch the product, and note the claims the scenario can test: where data goes, what the trial includes, what the onboarding promises.

I create the account the way a new customer would and set the product up from the docs alone, without asking you for help. Steps the docs skip, defaults I have to change and dialogs that close on their own go in the log, each with a timestamp.

The business I set up is made up. Its FAQ, its customers and its data are synthetic, so the test puts no real customer data into your product or the report.

I run the scenario from a clean state, again and again, changing one thing at a time until it works, or until I can show where it fails and how to reproduce it. The browser’s network log stays open on the customer-facing side for the whole session.

When something fails I keep going until I can reproduce it and isolate the conditions that trigger it. In the sample test the handoff failed four times in about twenty minutes before the cause turned out to be a default filter on the human step, then failed a fifth time later for a different reason. Tracking those two causes down is the most useful page of the report.

The report opens with the scenario result, whether it held or broke and where, then lists each finding with a severity, what happens, the confirmed cause, the steps to reproduce it, and the screenshots and timeline range that back it up. Data-handling facts and the pricing and limits I observed get their own sections.

It closes with questions for your team: the ones a careful buyer or a security reviewer would ask after reading it. You answer them before a customer asks.

You fix what you choose to fix. Inside the agreed window I rerun those items from a clean state with the original steps, take fresh screenshots, and mark each one fixed, partly fixed or still open. One round comes with each pre-launch audit.

The timeline behind the sample report, with the vendor removed. The times are the real times; the finding numbers match the report.

That test was one unpaid session on a public trial, run before this service existed. A paid audit uses the same method over about two weeks, with many more runs of your scenario from a clean state.

• 9:01 PMTrial account created. Blank workspace, docs open in the next tab.
• 9:07 PMKnowledge base built from the vendor’s own FAQ format.
• 9:16 PMFirst customer question goes unanswered: no AI reply, no handoff, no “please wait”.F1
• 9:26 PMThe same question answered once the FAQ was re-uploaded as a document.
• 9:37 PMHuman step changed to accept chats the AI already holds. Nothing in the builder suggested it.
• 9:38 PMHandoff works. The AI transfers without waiting for the customer’s yes.F4
• 9:51 PMA second automation placed above the default stops all routing.F1b
• 10:00 PMTalk-to-a-human button finally works, as a second trigger inside the default automation.F2
• 10:06 PMSettings checked for the consent option the privacy policy promises. There is none.F3
• 10:10 PMSession closed. 8 findings, 25 screenshots, 5 hosts in the capture.

1. The network log.Each host the customer-facing side contacts, when it was first seen, what it carries, and whether your privacy policy names it. Request paths and responses for anything that explains a finding.
2. Numbered screenshots.Taken at each step that matters, redacted of anything that would identify a real person, and referenced by number in the findings.
3. A raw timeline.Minute by minute, including the dead ends. The report quotes it; you get the whole thing.
4. The configuration at each step.Which settings were on, which automation ran first, which filter the default step carried. This is what makes a finding reproducible on your side.
5. The workspace state left behind.So your team knows what to clean up, and so a retest can start from the same place.

Anything that works well is written down too. A report that lists only failures gives a lopsided picture of the product, and your engineers will trust the rest of it less.

I test from a customer’s seat and stop where a customer would: no exploit attempts, no probing past what a customer can reach, and no source code review. I also give no legal opinion on your policy. When the policy says one thing and the product does another, the report records both and leaves the legal reading to your counsel.