Should I vibe code
Crawl an owned site, track selected keywords, and organize remediation tasks
The open-source rank tracker everyone points at last shipped in 2022. That is the whole feasibility study.
?
Their verdict, the Premium price and the build-time estimate come from their entry, MIT-licensed. Checked 2026-08-04.
?
Our verdict, the regret score and everything below it. Editorial and unsponsored — nobody can pay to be moved.
The honest answer
why the verdict is what it is
The crawler is not the product and it never was. Fetching your own pages, tabulating status codes, titles, canonicals and broken links is a script — we said as much on the Screaming Frog page and rated it ship it. What Seobility charges €49.90 for is the half that is not a crawl: three hundred keywords checked against Google, every day, for a year. There is no polite API for that. Google's own Search Console gives you averaged, sampled positions for queries you already rank for and will never tell you where a competitor sits, so a rank tracker means either scraping the SERP yourself — an arms race against rate limiting, CAPTCHAs, per-datacentre IP reputation and a results page that changes shape every quarter — or buying SERP queries from someone who already fights it. Three hundred keywords daily is nine thousand queries a month, and at commercial SERP-API rates that bill lands above the subscription before you have written a line. So the honest shape of this project is: build the crawler in a weekend, enjoy it, use it, and treat the rank-tracking half as a thing you buy or a thing you go without. The evidence is sitting in the prior art below — the best-known open-source rank tracker was archived in 2022, which is roughly how long that fight lasts.
What actually breaks
not "if". the specific failures.
- Rank tracking, permanently. Scraping the SERP is a maintained adversarial relationship, not a feature — proxies, CAPTCHAs, layout changes and a results page that differs by location, device and personalisation
- The bill, if you take the sane route instead. Three hundred keywords daily is nine thousand SERP-API calls a month, which costs more than the subscription you were avoiding
- The crawler's manners, the first time it runs unattended on a schedule. Sitebulb runs on your desktop where you can see it; a cron job at 25,000 URLs is a load test somebody else finds out about
- Crawl traps — faceted navigation, session IDs in query strings, infinite calendars — which will happily generate a million URLs on a site that has four thousand pages
- JavaScript rendering, the moment you add a headless browser. Each page becomes twenty requests, fires your own analytics beacons, and pollutes the data you are collecting
- The findings list, which is the thing you were actually paying for. Four thousand rows of true observations sorted by nothing is not an audit, and the ordering is judgement rather than code
- Backlink data, which you cannot produce at all. Nobody builds a web-scale link index over a weekend, and the commercial tools are reselling one
- Historical continuity, which is the only reason to track anything. Change your extraction logic in month four and every chart before it becomes a different measurement wearing the same axis
Is that you?
the verdict is a default, not a law
- You are crawling sites you own, on demand, with a concurrency limit you chose deliberately
- The output is a table you read yourself rather than a score you show a client
- Rank tracking is either absent or bought from an API, with a hard monthly query cap
- You want the specific checks nobody else runs — your own conventions, your own templates, your own definition of a broken page
- The crawler runs on a schedule against sites you do not own, unattended, and nobody is watching the target's error rate
- You are planning to scrape Google directly and have not priced the proxies, the CAPTCHAs or the weekend the layout changes
- Somebody is being invoiced for the report, because then the judgement layer is the deliverable and you have not built one
- The plan includes a backlink index
- You need the numbers to be comparable across a year and have not frozen how each one is calculated
If you build it anyway
the checklist, then the prompt that enforces it
- Respect robots.txt and send a real user agent with a contact URL, even when crawling your own site. Especially when crawling your own site — you will forget which config points where.
- Set concurrency and requests-per-second explicitly, default them low, and make the crawler back off on 429s and 5xxs rather than retrying harder.
- Cap total URLs per crawl and deduplicate aggressively on query parameters, or a faceted-navigation trap will eat the run and the target's origin.
- Buy SERP data rather than scraping it, and put a hard monthly query budget in code so a scheduling bug cannot spend a month's allowance overnight.
- Freeze how every metric is computed before you collect a second data point. A chart of a definition that changed is worse than no chart.
- Store raw crawl responses, not just parsed findings, so a new check can be run over old data instead of requiring a new crawl.
- Order findings by impact from the start, even if the ordering is a hand-written weight table. An unsorted list of true things is the failure mode of every homemade auditor.
I want a technical SEO auditor: crawl a site, extract on-page signals, report
issues. Build the crawler properly and be honest about what is out of reach.
1. Before any crawling, build the politeness layer: robots.txt parsing, a
configurable requests-per-second that defaults low, a concurrency cap, and
exponential backoff on 429 and 5xx. Show me the config before the parser.
2. Identify honestly. A descriptive user agent with a contact URL, no browser
impersonation, no rotating identities.
3. Add a hard URL cap and query-parameter normalisation next, and explain crawl
traps to me — faceted navigation, session IDs, infinite calendars — before
the first real run.
4. Store raw responses on disk keyed by URL and timestamp. Parsing is derived
and must be re-runnable over stored data without recrawling.
5. Then extraction: status, redirect chain, title, meta description, canonical,
h1s, hreflang, indexability, internal link graph. Plain tables first, no
scoring.
6. Only after that, findings — and every finding must name the exact URLs that
prove it and carry a hand-set impact weight. An unsorted list of four
thousand true observations is not an audit; say so if I ask for one.
7. If I ask for rank tracking, stop. Tell me Search Console gives averaged
sampled positions and nothing about competitors, that scraping Google is an
ongoing adversarial project, and price a SERP API for my keyword count
before writing anything. Then put a hard monthly query budget in code, and
refuse outright to build proxy rotation or CAPTCHA solving.
8. If I add JavaScript rendering, warn me first: every page becomes many
requests, third-party beacons fire, and my own analytics gets polluted.
9. Freeze metric definitions before collecting history, and record the
definition version alongside every data point.
10. Out of scope and say so: backlink indexes, competitor content scoring, and
anything crawling a domain I do not own. Then tell me Seobility Premium is
€49.90 a month and that the keyword half is most of what that buys.That one keeps you out of trouble. For the prompt that actually builds it, canivibecodeit.com has one.
their build prompt ↗Or don’t build it
the boring option, and the way back out
Buy the moment you want positions over time, or a second site, or a report anyone else reads. The €49.90 is not paying for the crawler — you can have that for free, and Screaming Frog's page here says so — it is paying for three hundred daily SERP lookups you would otherwise buy at a higher unit price, plus a few hundred codified checks ordered by someone who has seen which ones matter. Build your own if what you want is the odd check nobody sells: your internal URL conventions, your template rules, your definition of a page that is broken.
$49.9/mo is cheaper than your weekend.
Keep the raw crawl on disk — one stored response per URL per run, with headers — and treat every finding, score and chart as derived. Do that and switching tools is trivial, because what you own is the history rather than the software. Export findings as CSV with the URL, the check name and the definition version, and if you did buy SERP data, keep the raw position rows too: those are the only numbers in this project you cannot regenerate, since nobody can tell you where you ranked last March after the fact.
Self-hosted technical SEO auditor with a crawler and a codified issue list, actively developed.
Open-source rank tracker, archived with no commits since early 2022 — the clearest available evidence of how that fight ends.
Questions
Why can't I just scrape Google for rankings?
You can, for a while. What you are signing up for is not a scraper but an ongoing adversarial relationship: rate limiting, CAPTCHA challenges, IP reputation that degrades as soon as you use a datacentre address, and a results page whose markup and shape change several times a year. Positions also vary by location, device, language and personalisation, so a single number is already a simplification you have to defend. The open-source rank tracker in the prior art below fought that fight and stopped in 2022.
Screaming Frog is rated SHIP IT and Sitebulb DEMO ONLY. Where does this sit?
Right next to Sitebulb, for a related but distinct reason. Sitebulb's page is about judgement — knowing which of four thousand findings is worth a Tuesday. Seobility's is about the keyword half, which is not a judgement problem but a data-acquisition one: the tracking that makes up most of the subscription is something you either buy or do without. The crawler in all three cases is the same weekend, and that weekend is genuinely worth having.
What is the least obvious thing that goes wrong?
Changing how you calculate something in month four. Rank tracking and audits are only useful as trends, and a trend needs the measurement to hold still. Redefine "indexable", switch from desktop to mobile SERPs, or start following one more redirect hop, and every chart quietly becomes two different measurements sharing an axis. Version your metric definitions and store the version with each data point, or the history you spent a year collecting is not comparable to itself.
Every week, someone ships something they shouldn’t have.
New verdicts, the worst thing that landed in the trap, and the occasional incident report. No other email, ever.
Anyone can write a crawler. The product is knowing which of the 4,000 findings is worth a Tuesday.
Crawling your own site and reading the headers is a script, not a licence.
SERP analysis means scraping search results, which is a fragile foundation to build on.
last reviewed 2026-08-05 · verdict is editorial and unsponsored · shared entry data from canivibecodeit under MIT · not legal advice