What does your AI product do when a stranger sets it up?

You tell me the product and the one scenario your customers run most. I sign up like a stranger, set it up from the docs alone, run the scenario from a clean state until it breaks or holds, and keep a network capture running throughout. About two weeks later you have a findings report, a short list of questions a careful buyer would ask you, and one retest round for whatever you fix.

The test behind the sample report: one unpaid 70-minute session on a vendor’s public trial. A paid audit runs the same method over about two weeks, with many more runs of your scenario from a clean state.

• 9:01 PMTrial account created. Blank workspace, docs open in the next tab.
• 9:07 PMKnowledge base built from the vendor’s own FAQ format.
• 9:16 PMFirst customer question goes unanswered: no AI reply, no handoff, no “please wait”.F1
• 9:26 PMThe same question answered once the FAQ was re-uploaded as a document.
• 9:37 PMHuman step changed to accept chats the AI already holds. Nothing in the builder suggested it.
• 9:38 PMHandoff works. The AI transfers without waiting for the customer’s yes.F4
• 9:51 PMA second automation placed above the default stops all routing.F1b
• 10:00 PMTalk-to-a-human button finally works, as a second trigger inside the default automation.F2
• 10:06 PMSettings checked for the consent option the privacy policy promises. There is none.F3
• 10:10 PMSession closed. 8 findings, 25 screenshots, 5 hosts in the capture.

Each pre-launch audit includes one retest round for the items you fix. Further rounds are quoted separately.

1. Separate from OpenSourcesAI.Paying for an audit buys no coverage, placement or ranking on opensourcesai.com. Before a buyer hires me to test a vendor’s product, I tell them about any paid work for that vendor.
2. Findings come with evidence.Each one has a timestamp and a screenshot. Anything I could not verify is marked as not verified.
3. Your customers’ data stays out.I work with a made-up business and test accounts. Real customer data does not go into the product during an audit.
4. One agreed scenario.Keeping to one scenario lets your team rerun the same steps and compare the result.
5. Fixed price.The number in the quote is what you pay. I do not bill by the hour or add charges partway through.

Is this a penetration test?

No. I test from a customer’s account and stay within what a customer can reach, so there are no exploit attempts, no source code review and no legal opinion. If you need a penetration test, hire a pen tester as well; the two cover different ground.

What do you need from us?

The product, the one scenario you care about most, and a trial or test account. If the trial is too short for the scenario, a test workspace with a longer clock.

How long does it take?

About two weeks from kickoff to report. A retest takes a few days. The clock starts when the scope is signed and the first half is paid.

Will you publish anything?

No. The findings are yours and stay private. If you agree in writing, one anonymized line can go on the track record page, without the product’s name, the vendor’s name or any other detail that would identify either.