Crawlora Docs
Create a monitor once — a page or a sitemap — and get a signed webhook the moment a change is detected. No polling loop, no diffing pipeline to maintain.
Monitors use the same authentication as every other Crawlora endpoint — no separate signup or product key.
POST /monitors with a url and how often to check it. Creating, listing, updating, pausing, and deleting monitors is always free — only a completed check run costs credits.
curl -X POST https://api.crawlora.net/api/v1/monitors \
-H "x-api-key: $CRAWLORA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/pricing",
"cadence_minutes": 60,
"notification": {
"webhook_url": "https://example.com/webhooks/crawlora",
"webhook_secret": "whsec_...",
"events": ["change.detected"]
}
}'cadence_minutes accepts 5 minutes to 10080 (7 days) and defaults to 60. A disabled monitor (enabled: false) is never checked until re-enabled.
target_type defaults to page. Set it to sitemap to track a whole sitemap's URL set instead of one page's content.
Scrapes urland diffs an exact SHA-256 fingerprint of the page's content against the previous check. Reports a compact line-level diff when it changes.
Fetches the sitemap at url — a <urlset>, or a one-level <sitemapindex> whose child sitemaps are fetched and merged — and diffs the tracked URL set for additions and removals. Use sitemap.include_patterns/exclude_patterns (shell-style globs matched against each URL's path) to scope what's tracked, and sitemap.max_urls (default 5000, hard cap 10000) to bound it.
Changing target_type or sitemap on an existing monitor (via PATCH /monitors/{id}) resets its stored baseline, so the next check starts fresh instead of comparing against a now-meaningless prior state.
Two independent events, both opt-in via notification.events (defaults to change.detected only, for back-compat).
Every delivery carries X-Crawlora-Signature: t=<unix>,v1=<hmac-hex> — an HMAC-SHA256 digest of the string "{t}.{rawBody}" using your webhook_secret. Recompute the digest, compare it in constant time, and reject any timestamp older than a few minutes to reject stale or replayed deliveries.
import hmac, hashlib, time
def verify(secret: str, signature_header: str, raw_body: bytes, max_age_seconds: int = 300) -> bool:
parts = dict(p.split("=", 1) for p in signature_header.split(","))
t, v1 = parts["t"], parts["v1"]
if abs(time.time() - int(t)) > max_age_seconds:
return False
expected = hmac.new(secret.encode(), f"{t}.".encode() + raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, v1)Management is free. Only a completed check run is billed — 1 credit per run, regardless of target type or whether the run found a change.
GET /monitors/{id}/checks returns the monitor's recent runs, most recent first.
Each check reports whether it changed, the diff (page target) or added_urls/removed_urls (sitemap target), whether it was billed, and a webhook_deliveries entry per event this run sent — each with a status of delivered, rejected, failed, or skipped_unsafe_url.
Crawlora is designed for responsible structured public web data workflows. Customers are responsible for using Crawlora in compliance with applicable laws, third-party rights, target-platform rules, and Crawlora terms.
Read Crawlora termsYes — /monitors accepts the same x-api-key used by every other Crawlora endpoint (not just a dashboard session), and create/list/get/update/delete/check-history are each exposed as MCP tools, so an agent can manage its own monitors.
The check is skipped (recorded with skipped: true, and never billed) and the monitor keeps its normal cadence — it isn't paused or put into an exponential-backoff failure state, since insufficient credits isn't a target-site error.
An exact SHA-256 fingerprint of the scraped page's content, compared to the fingerprint from the previous check. Any byte-level difference counts — there is no semantic/LLM-judged change detection yet.
No — one monitor watches one url (a page, or a sitemap). Create one monitor per target.
Delivery retries on 5xx/429/network errors with capped attempts (respecting a numeric Retry-After), then records the run's webhook_deliveries entry as failed. Check GET /monitors/{id}/checks for delivery history.
Open the console to create a monitor from the dashboard, or call POST /monitors directly with your API key.
Need API-key basics first? Start with Getting Started.