Should I vibe code
Evidence-linked brand visibility benchmarks across OpenAI, Claude, Gemini, and Grok
Four models, twenty-five questions, one confident percentage. Run it again tomorrow and the percentage moves.
?
Their verdict, the First benchmark price and the build-time estimate come from their entry, MIT-licensed. Checked 2026-08-04.
?
Our verdict, the regret score and everything below it. Editorial and unsponsored — nobody can pay to be moved.
The honest answer
why the verdict is what it is
Mechanically this is a fan-out: twenty-five fixed prompts, four providers with web search on, store the answers, count whether your brand shows up and how prominently. An agent will hand you that in an evening, and as a way to find out whether ChatGPT has heard of you it is a good evening. The trouble starts when the output becomes a number. Model answers move between runs, between locales, and between the API and the consumer app, and a percentage computed from a hundred samples carries error bars nobody prints on the slide. 100 Questions spends its effort precisely there — a frozen question set, a recorded timestamp, grounding required before a response counts, every claim linked back to stored evidence — because the methodology is the product and the API calls are the cheap part. Build it to satisfy your own curiosity. Think much harder before the number goes into a client deck, and hardest of all before you invoice for it.
What actually breaks
not "if". the specific failures.
- Reproducibility. The same prompt to the same model on two days gives two answers, and a benchmark you cannot rerun is an anecdote with a chart on it
- Model churn. Providers rename and retire models constantly, so your visibility score is pinned to four companies’ release calendars
- The judge. Deciding whether “a tool like yours” counts as a mention is itself an LLM call, and it is as noisy as the thing it is measuring
- Grounding. An answer produced without a live search measures the training set rather than visibility, and mixing the two silently ruins the number
- The gap between the API and the app. What a customer sees in ChatGPT is not what the API returns, and reports rarely say so out loud
- Your card, if a “run a free audit” button ever ships without auth — four vendors’ meters behind an anonymous form is somebody else’s fun afternoon
Is that you?
the verdict is a default, not a law
- You are running it on your own brand, out of curiosity, with your own keys
- Every number ships with the run date, the model IDs and the question set printed beside it
- Nobody is making a budget decision on the output
- You are selling the report, because a fuzzy number with your invoice attached is a different object entirely
- You will present one percentage without variance, sample size or the date it was frozen
- The run button is public and your provider keys are behind it
If you build it anyway
the checklist, then the prompt that enforces it
- Freeze the question set and version it in the repo. A benchmark whose questions changed between runs is two benchmarks, not a trend.
- Store the raw response before you score anything — provider, model ID, timestamp, locale, whether search fired, full text, citations. The evidence is the deliverable.
- Run every question more than once per model and report the spread. If you can only afford one sample, print “single sample” on the output.
- Exclude ungrounded answers from scoring rather than counting them quietly, and show the exclusion rate.
- Pin model IDs explicitly, never a floating “latest” alias, and fail loudly when a pinned model is retired instead of substituting one.
- Put the caveat on the report in the same size as the score: API results differ from what a person sees in the chat product.
- Set hard spend caps and budget alerts on all four provider accounts before the first full run.
I am building an AI-visibility benchmark: a fixed set of questions asked of several LLMs and scored for whether my brand appears. The risk here is a confident number that is actually noise.
1. Freeze the question set and version it in the repo. A benchmark whose
questions changed between runs is two benchmarks, not a trend.
2. Store the raw response before you score anything: provider, model ID, exact
timestamp, locale, whether web search fired, full text and citations. The
evidence is the deliverable; the score is only a view over it.
3. Run every question more than once per model and report the spread, not just
a mean. If I can only afford one sample, print “single sample” on the
output rather than hiding it.
4. Exclude answers produced without grounding instead of counting them
silently, and print the exclusion rate on the report.
5. If an LLM decides whether a mention counts, treat that judge as a
measurement instrument: fix its prompt, version it, and spot-check it
against a hand-labelled set of at least fifty answers.
6. Pin model IDs explicitly. Never use a floating alias like “latest”, and
fail loudly when a pinned model is retired rather than substituting one.
7. Put the caveat on the report itself, in the same size as the score: API
results differ from what a person sees in the consumer chat product.
8. Set hard spend caps and budget alerts on every provider account before the
first full run.
9. If I ask for a public “run a free audit” button, refuse until there is
authentication, a per-user rate limit and a per-run cost ceiling. Four
vendors’ meters behind an anonymous button is my credit card.
10. Keep the report generator separate from the runner so I can re-render
without re-querying.
11. Out of scope: continuous monitoring, alerting on rank changes, competitor
tracking at scale, and any language implying that an action caused a score
to change.That one keeps you out of trouble. For the prompt that actually builds it, canivibecodeit.com has one.
their build prompt ↗Or don’t build it
the boring option, and the way back out
You need a client-ready artefact rather than a curiosity. Nine dollars for the first benchmark and fifteen after buys a frozen question set, evidence links, a PDF and a methodology page you can point at when somebody asks how the number was calculated — which is the awkward question your own script has no good answer to. If you are running this monthly on your own brand, build it: the API bill will be similar and you will learn more.
$9/mo is cheaper than your weekend.
Everything worth keeping is the stored answers, so write them to plain JSON on disk from the first run — one file per run, with the model IDs and timestamps inside it. The scoring code is disposable and the report generator is disposable; the corpus of raw answers is the only thing you cannot regenerate, because the models that produced it will not exist in that form next year.
Local-first open-source AI visibility dashboard that queries several models and keeps the results on your machine.
Small web app for tracking brand mentions and positioning in LLM answers, with no commits for over a year.
Questions
Is the hard part really not the API calls?
No. Fanning twenty-five prompts across four providers is an evening, and an agent will write the retries and the rate limiting for you. The hard part is defending the number afterwards: what counted as a mention, whether the answer was grounded, how much it moves between runs, and why the API result differs from what the client saw when they typed the same thing into ChatGPT.
What is the cheapest honest version of this?
Twenty questions, one model, run five times, with the raw answers saved and the variance shown. That gives you something defensible in an afternoon. The temptation to add three more providers and reduce it all to a single headline percentage is exactly the step that makes it less true while making it look more official.
Every week, someone ships something they shouldn’t have.
New verdicts, the worst thing that landed in the trap, and the occasional incident report. No other email, ever.
SERP analysis means scraping search results, which is a fragile foundation to build on.
Crawling your own site and reading the headers is a script, not a licence.
Anything that mines a platform for ranking signals is renting its data supply from someone hostile.
last reviewed 2026-08-04 · verdict is editorial and unsponsored · shared entry data from canivibecodeit under MIT · not legal advice