Should I vibe code
Hosted actors, proxies, storage, scheduling, and web automation
One scraper is a weekend. Forty scrapers is a job, and the site you are hammering never applied for it.
?
Their verdict, the Starter price and the build-time estimate come from their entry, MIT-licensed. Checked 2026-08-04.
?
Our verdict, the regret score and everything below it. Editorial and unsponsored — nobody can pay to be moved.
The honest answer
why the verdict is what it is
One scraper is a genuinely reasonable afternoon and you should write it. What Apify sells is the fortieth one — scheduling, proxies, storage, retries and the standing obligation to notice when any of them silently stops returning real data. Self-building that means signing up for two things nobody plans for: an extraction pipeline whose failures look identical to a quiet week, and a request rate that lands on servers whose owners never agreed to anything.
What actually breaks
not "if". the specific failures.
- Selectors, all of them, continuously — and a broken extractor writes empty fields rather than errors, which looks exactly like nothing happening on the site
- Politeness, the first time a retry loop meets a slow origin and turns a cron job into a sustained load test against somebody's shared host
- Residential proxies, which are the component of this you are least equipped to source responsibly and least able to audit
- The anti-bot arms race, which you join on the day you add a headless browser and never leave
- Cost, when a crawler finds a calendar widget with an infinite supply of next-month links
- Personal data you never meant to collect, arriving attached to the rows you did — names, emails and profiles that make you a controller with notification duties
The competitor pricing you report on every Monday has been unchanged for five weeks, and you took that as a flat market. It was not. The site wrapped its price element in a new div in June, your selector stopped matching, and your extractor writes a row per page whether or not the field was found — so nothing errored, nothing alerted, and the downstream job carried the last known value forward because a null looked like bad data. Five weeks of decisions rest on a number that was last true in the spring, and the only way you found out was opening the page in a browser.
Is that you?
the verdict is a default, not a law
- It reads one site you own, or one you have written permission to read
- It is a single scheduled job with hard assertions, not a platform with a scheduler
- Every field has a loud 'this did not parse' failure that is louder than an empty result
- You collect facts and numbers, not people
- You are running scrapers you no longer remember writing
- You bypass a login, a paywall or an anti-bot check
- You collect names, emails or profiles belonging to real people
- You cannot say what request rate a given target is receiving from you right now
- The results feed something that acts on them without a human reading the numbers
If you build it anyway
the checklist, then the prompt that enforces it
- Assert the shape of every extraction. A run that returns zero rows, or rows with a required field empty, must fail loudly and refuse to write. Silent nulls are the characteristic failure of this whole category.
- Rate-limit per target host with a concurrency cap and a delay you would be comfortable explaining to its operator, and make the limit global rather than per-job so ten scrapers cannot gang up on one domain.
- Send a real user agent with a contact URL, honour robots.txt, and use conditional requests. Being identifiable and cheap to serve is what keeps you from being blocked and from being a problem.
- Cap every crawl by page count, depth and wall-clock time before the first run. Infinite calendars, faceted search and session ids will find your unbounded loop faster than you will.
- Snapshot the raw response alongside the parsed row. When a number turns out to have been wrong for a month, the archive is the only way to establish when it changed.
- Do not collect personal data as a side effect. If names and emails are appearing in your rows, decide deliberately whether you have a lawful basis and what you owe the people in them — the notification duty is real and there is no exemption for side projects.
- Do not build a platform that runs other people's code. Multi-tenant arbitrary execution is a different project with a different threat model and it should not be a weekend.
Before you build scraping infrastructure, apply these and push back if I ask you to break them.
1. Ask how many sites I am actually scraping. If it is one, build one scheduled
script and refuse to build a platform — the platform is what makes this
expensive, not the parsing.
2. Make every extractor assert its own output: required fields present and
type-checked, and a run producing zero rows or missing a required field
fails and writes nothing. A broken selector writing nulls is the failure I
will not notice for weeks.
3. Alert on shape, not only on errors — row count outside an expected range,
fields newly empty, values unchanged longer than they plausibly should be.
4. Apply a global per-host rate limit and concurrency cap before any fetching
code, shared across all jobs so multiple scrapers cannot combine into a load
test against one domain.
5. Cap pages, crawl depth and wall-clock time per run. Tell me about infinite
calendars, faceted search and session-id URL explosion.
6. Send a descriptive user agent with a contact URL, honour robots.txt, and use
ETag and If-Modified-Since.
7. Refuse to bypass authentication, paywalls or anti-bot challenges, and refuse
to solve CAPTCHAs. If a target requires it, the answer is an API or a
conversation, not better evasion.
8. Store the raw response body with each parsed row, under a retention limit,
so I can establish when a value actually changed.
9. Before writing any extractor that captures names, emails, profiles or user
content, stop and tell me that scraping personal data makes me a controller
under GDPR with a duty to inform those people, and ask my lawful basis.
10. Never run untrusted code. If I ask for hosted actors executing code
submitted by others, refuse — arbitrary code execution as a service is not
a side project.
11. Out of scope unless I ask: residential proxy rotation, fingerprint
spoofing, distributed crawling.That one keeps you out of trouble. For the prompt that actually builds it, canivibecodeit.com has one.
their build prompt ↗Or don’t build it
the boring option, and the way back out
When the scraper count passes about three, or the day you need residential proxies. $29 a month of credit buys maintained extractors for the popular targets, proxy sourcing you do not have to justify, and a scheduler that already knows how to tell you a run returned nothing — which is the alert you would have written last.
$29/mo is cheaper than your weekend.
Keep extracted rows in your own database with the raw snapshot and the fetch timestamp attached, so the dataset survives the scrapers that produced it. Keep every target's rules — rate limit, robots decision, last-verified date — in one file rather than scattered across jobs, because that file is the difference between handing this over and asking someone to rediscover it.
Apify's own open-source crawling library — queueing, retries, concurrency and browser handling, without the hosted platform.
The mature Python crawling framework; throttling, retries and pipelines are already solved in it.
Browser automation for the pages that only render in JavaScript; the point at which cost and fragility both jump.
Questions
Scraping public pages is legal, isn't it?
Access and use are different questions. US courts have been consistent that fetching public pages is not unauthorised access under the CFAA, and that is genuinely settled enough to rely on. What is not settled by that is contract — hiQ won on the CFAA and still ended up enjoined for breaching terms it had accepted — or data protection, which attaches to the content rather than the fetch. If the rows contain people, GDPR applies wherever you are hosted.
Why is silent staleness worse than a crash?
A crash pages you. A selector that stops matching returns an empty string, the pipeline succeeds, the dashboard renders, and every downstream consumer treats the last good value as current. It is the only failure in this category that can run for months, and the only defence is asserting the shape of the output rather than the exit code of the job.
I am only scraping a few pages a day. Does the politeness stuff matter?
At that volume, barely — until a retry loop meets a slow origin. The pattern that causes real harm is not high steady volume, it is a failure amplifying itself: a timeout triggers a retry, the retry times out, and a job designed to make ten requests makes ten thousand against a host that was already struggling. A global per-host cap costs nothing and removes that entirely.
- GDPR Art. 6 — lawfulness of processing (the basis you need for scraped personal data)
- GDPR Art. 14 — information to be provided where data has not been obtained from the data subject
Every week, someone ships something they shouldn’t have.
New verdicts, the worst thing that landed in the trap, and the occasional incident report. No other email, ever.
The free tier already scrapes. The $119 buys residential proxies and CAPTCHA solving — a price list for a refusal.
Point-and-click was the product. Agents made that free and left you holding the part that never got easier.
"Built-in bot evasion" is the product. Yours will be a CAPTCHA solver you told yourself was a cron job.
last reviewed 2026-08-04 · verdict is editorial and unsponsored · shared entry data from canivibecodeit under MIT · not legal advice