A soft block is an informal scraping term for a response that looks successful at the HTTP layer but does not contain the requested data — for example, a challenge page, login shell, empty result template, or deliberately incomplete payload returned with a plausible status such as 200 OK.
A scraper that treats every 2xx response as valid can quietly write a challenge page into an HTML archive or replace real records with an empty template. Soft blocks are operationally dangerous because they resemble successful fetches in dashboards while corrupting the dataset downstream. They are broader than soft 404s, which specifically describe not-found content served under the wrong status.
Not every challenge returns 200, and not every empty response is bot detection. Authentication expiry can legitimately redirect to a login page; a search can genuinely have zero results; an application can fail while still returning HTML. The label describes a false-success symptom, so diagnosis needs evidence from the response rather than the status alone.
Check the final URL after redirects, the status, Content-Type, expected schema, required fields, and positive content markers together. For a product JSON endpoint, that might mean a stable product ID and title; for an HTML page, a known main-content selector. Also record block markers such as challenge titles or vendor headers. Cloudflare challenge responses, for example, set cf-mitigated: challenge and use text/html even when the requested resource had another media type.
Positive validation is safer than maintaining only a blacklist of block-page phrases. Challenge markup changes, while the fields that make a response useful to your pipeline are usually explicit. Keep a small sample of rejected payloads for diagnosis, but do not let an unvalidated response advance the job's success counters.
Quarantine an invalid payload before it reaches the main dataset. Record the URL, request identity, response metadata, and validation reason; then retry a bounded number of times with backoff or a fresh approved request context. An unlimited retry loop can amplify the same detection signal and waste quota without producing better data.
When repeated responses fail the same semantic checks, stop that item for review rather than converting the failure into an empty record. Compare the URL in an ordinary browser and inspect whether the outcome is a bot challenge, an expired session, a real empty state, or a changed schema before choosing the next fix.
How Crawlora handles this
Crawlora's platform endpoints return documented structured fields, which gives consumers concrete schema and required-field checks before accepting a response. Its docs also describe API errors separately, so callers can keep invalid payloads out of their datasets instead of relying on status alone.
Related reading
Glossary
FAQ
No. A 200 only says the HTTP request succeeded; it does not prove the body contains the requested resource. Validate the final URL, media type, schema, and expected content before committing it.
No. A soft 404 is specifically a not-found page served with a misleading status. A soft block is the broader scraping failure where an apparently successful response withholds expected data, often because of a challenge or session gate.
No. The source may genuinely have no matches, your session may have expired, or the response schema may have changed. Compare positive markers, response metadata, and a normal browser result before classifying it as a block.
Beyond Soft Block, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.