Crawlora Changelog

Launched a suite of nine no-signup tools at /tools, each backed by the live Crawlora API with a one-click path from a free sample to a ready-to-run API call.

• Free Web Scraper turns any URL into clean Markdown, with a "Works with" wall of 180+ verified sites and clear blocked-vs-unreachable errors.
• "Can I Scrape This Site?" anti-bot checker reports the protection a domain uses and how hard it is to collect, framed around authorized public data.
• Free SERP Checker runs multi-engine searches with shareable, cached result permalinks.
• Email & Contact Finder, Keyword Research, YouTube Transcript Extractor, App Review Analyzer, TikTok Ad & Trend Finder, and Google Trends Dashboard round out the set.
• Every result shows a "Get this via the API" snippet prefilled with your inputs, plus a soft rate-limit upgrade prompt, so the jump from sample to integration is one copy-paste.
• Results are shareable via per-result permalinks with their own social preview cards.

Added result permalinks with per-result Open Graph cards across the free tools, plus a share bar on blog posts.

• Every free-tool run can be shared as a replayable permalink that re-renders the same result.
• Per-result Open Graph cards are generated server-side, so shared links unfurl with a result-specific preview image.
• SERP-checker results are cached and shareable, with a visit counter on shared links.
• Blog posts gained a footer share bar with an "Explore with AI" option.

Added six new platforms — Polymarket, Kalshi, Metaculus, IMDb, Rotten Tomatoes, and Box Office Mojo — plus an expanded Reddit API, all returning normalized JSON.

• Polymarket: events, markets, outcome prices, order books, midpoints, spreads, price history, tags, sports, leaderboards, and trade activity.
• Kalshi: events, markets, series, recent trades, order books, exchange schedule, and historical market data.
• Metaculus: forecasting questions, community forecasts and history, metadata, options, and comment feeds.
• IMDb: title and name detail, credits, awards, episodes, reviews, keywords, trivia, and technical specs.
• Rotten Tomatoes scores and reviews for movies, series, seasons, and episodes; Box Office Mojo lifetime grosses, yearly and weekend charts, releases, franchises, and genres.
• Expanded the Reddit API with subreddit about and comments, multi-subreddit and domain posts, user posts and comments, and trends.

Regenerated all six official SDKs from the public API contract to cover the new platforms — now 491 public operations — and published v1.7.0-sdk.1 to every language registry.

• Added grouped helpers for Polymarket, Kalshi, Metaculus, IMDb, Rotten Tomatoes, and Box Office Mojo, plus the expanded Reddit API.
• Published v1.7.0-sdk.1 to npm, PyPI, RubyGems, Maven Central, Packagist, and the Go module proxy.
• The public contract grew to 491 operations, and the hosted MCP server now exposes 439 tools.

Published official beta SDKs for Java, PHP, and Ruby across Maven Central, Packagist, and RubyGems, and documented all six SDKs on the SDK and examples pages.

• Added an official Java SDK to Maven Central as net.crawlora:crawlora-sdk — a dependency-free client built on java.net.http for Java 17+.
• Added an official PHP SDK to Packagist as crawlora/sdk for PHP 8.1+, installable with composer require crawlora/sdk:^1.5@dev.
• Added an official Ruby SDK to RubyGems as the crawlora gem for Ruby 3.0+, installable with gem install crawlora --pre.
• All three share grouped endpoint helpers, API-key and JWT auth, automatic retries with backoff, numeric and cursor pagination, middleware hooks, and client-side rate limiting.
• Promoted every official SDK to the v1.5.0-sdk.3 tag and refreshed the SDK hub, guide pages, and developer examples to cover all six languages.

Shipped a new Brand Data API and surfaced it across the public catalog, platform discovery, docs, Playground, and a use case.

• Added the GET /brand/retrieve endpoint, which fetches a domain's homepage and returns a normalized brand profile as structured JSON.
• Extracted fields include title, description, slogan, brand colors, logos and icons, social profiles, footer links, and schema.org organization data such as address and stock ticker.
• Added a Brand Data API platform landing page with docs, Playground, and related discovery links.
• Supported query options for domain input, a faster maxSpeed mode, a cache-freshness window, and an upstream fetch timeout.
• Wired the Brand Data API into the company data enrichment use case alongside LinkedIn, SimilarWeb, and Product Hunt.

Expanded the hosted MCP server to expose the public endpoint catalog as callable tools, with API-key authentication and enforced REST/MCP parity.

• Exposed the public endpoint catalog as hosted MCP tools so agents can search, research, and enrich without bespoke integrations.
• Added API-key authentication using the x-api-key header for MCP tool calls.
• Enforced REST/MCP coverage so the available tools track the public REST endpoints.
• Covered families include search and maps, Google and Yahoo finance and trends, YouTube, TikTok and TikTok Top Ads, reviews, marketplaces, travel, geocoding, podcasts, datasets, Reddit, and the new Brand Data API.
• Exposed usage endpoints over MCP so agents can check consumption while they work.

Published the Reddit endpoint family across the public catalog, platform discovery, docs, Playground, and a use case, with reliability improvements to the Reddit collection path.

• Added Reddit search, post detail, comments, and subreddit posts endpoints that return normalized public post and comment fields.
• Added a Reddit platform landing page with docs, Playground, and related discovery links.
• Added a Reddit social listening use case for brand mentions, sentiment, and community trend research.
• Improved reliability of the Reddit collection path for more consistent public results.
• Kept Reddit copy scoped to public posts and comments with responsible-use guidance.

Rolled out a sitewide visual refresh and expanded the public content footprint across blog, comparison, and discovery surfaces.

• Introduced a new emerald design system with a refined palette and self-hosted typography across marketing, console, Playground, and docs pages.
• Refactored and curated the site footer for clearer navigation and fewer columns with "see all" links.
• Added cornerstone blog posts and topic archive pages for organic discovery.
• Expanded comparison coverage and SERP-focused landing pages for high-intent queries.
• Added social preview (Open Graph and Twitter) images sitewide and tightened sitemap hygiene.
• Added a new contact page and refreshed the brand icon set with an installable web app manifest.

Improved front-end performance and added automated quality and Core Web Vitals monitoring for the public site.

• Reduced page weight with self-hosted fonts, bundle budgets, and catalog caching.
• Split the Playground into lazy, on-demand tabs so initial loads stay light.
• Added Core Web Vitals reporting through site analytics for real-user performance.
• Fixed accessibility issues including focus rings and label-in-name compliance.
• Added an automated Lighthouse quality guardrail to the build pipeline.

Hardened the API request path, expanded observability, and shipped reliability-focused SDK upgrades.

• Added a Redis-backed circuit breaker with fail-open rate limiting so the API stays available during dependency hiccups.
• Added availability cache and hedge metrics plus optional distributed tracing for clearer operational insight.
• Added a dependency security watch and upgraded the Go runtime for the API.
• Added SDK middleware, idempotency, client-side rate limiting, and connection pooling.
• Included pagination metadata in SDK responses and added bearer authorization support for MCP access.

Published the Shopify endpoint family across the frontend catalog, platform discovery, pricing examples, use-case pages, Playground, sitemap, and changelog.

• Added Shopify store, products, product detail, recommendations, collections, collection products, pages, page detail, search suggestions, child sitemaps, and sitemap URL endpoints to public discovery.
• Added a Shopify platform landing page with docs, Playground, pricing, related commerce links, and official-API caveats.
• Added a Shopify storefront intelligence use-case page for catalog monitoring, collection research, page discovery, predictive search analysis, and sitemap audits.
• Synced frontend endpoint credit examples with backend Shopify billing weights.
• Kept Shopify copy scoped to public storefront data and directed private merchant, order, checkout, inventory, and admin workflows to official Shopify APIs.

Published the Shop.app endpoint family across the frontend catalog, platform discovery, pricing examples, use-case pages, Playground, sitemap, and changelog.

• Added Shop.app search, product, product reviews, product variants, related products, product-to-shop, shop profile, shop products, collection products, shop locations, shop reviews, shop typeahead, categories, suggestions, and analysis endpoints to public discovery.
• Added a Shop.app platform landing page with docs, Playground, pricing, related marketplace links, and official-API caveats.
• Added e-commerce product intelligence use-case coverage for Shop.app, Amazon, and eBay workflows.
• Synced frontend endpoint credit examples with backend Shop.app billing weights.
• Included Shop.app docs, Playground, platform, and use-case routes in sitemap-backed discovery.

Published machine-readable discovery surfaces for agent workflows and improved markdown negotiation for clients that prefer structured text responses.

• Added public agent card, skill index, and skill markdown routes for capability discovery.
• Published MCP server card and API catalog linkset metadata for client-side integration flows.
• Added authorization server and protected-resource metadata with canonical public origins.
• Improved markdown response negotiation so agent clients can request readable page content.
• Published Content Signals and DNS-AID records to make site policy and identity metadata easier to discover.

Expanded the YouTube transcript showcase library and improved the generated metadata, page performance, and browse experience for long-form transcript examples.

• Added more real transcript showcase pages across AI, business, documentaries, interviews, and research topics.
• Published accurate video publish dates in showcase page metadata and visible page details.
• Made showcase pages static so generated examples stay fast and consistent for visitors.
• Reduced runtime memory needs for the growing showcase library.
• Added pagination to the YouTube showcase index so the growing library stays easy to scan.
• Included paginated showcase URLs in the public sitemap so crawlers can discover deeper pages.

Refined SDK documentation, public OpenAPI export behavior, and example guidance for teams integrating Crawlora APIs.

• Added a public OpenAPI export path that excludes deprecated surfaces from SDK generation.
• Documented SDK maintenance and release checks so generated clients stay easier to verify.
• Updated public SDK links and examples across developer documentation.
• Kept private account, billing, and deprecated endpoint details out of public SDK discovery.
• Improved developer-facing guidance for typed models, enum handling, and example smoke checks.

Hardened YouTube video metadata extraction and related availability checks used by transcript and video research workflows.

• Improved fallback behavior for YouTube video metadata when upstream responses vary.
• Added retry coverage for automated checks that confirm video availability.
• Aligned metadata reliability work with transcript showcase publish-date verification.
• Preserved clear error behavior for unsupported or unavailable upstream video pages.
• Kept reliability documentation current for future YouTube endpoint maintenance.

Published public discovery paths for stored Google Maps business datasets backed by structured search and validation behavior.

• Added a first-class Datasets platform surface for stored Google Maps business records.
• Published list, search, nearby, facets, and item endpoints in Docs and Playground.
• Aligned dataset read pricing and calculator examples with backend billing weights.
• Validated filtering, pagination, geo radius, and result-window behavior for stored dataset reads.
• Clarified that stored dataset search returns indexed records without triggering live scraping.

Synced the public frontend catalog and marketing paths with newly active CoinGecko endpoints from backend Swagger.

• Added CoinGecko as a first-class Crypto & Digital Assets platform surface.
• Published markets, coin profile, chart analysis, trending, search, category, chain, exchange, NFT, unlock schedule, treasury, news, and Learn article workflows.
• Aligned pricing examples and calculator scenarios with backend billing weights for the new endpoint family.
• Updated homepage, product, feature, platform, footer, getting-started, and Playground surfaces to reflect the active public catalog.
• Kept crypto copy focused on research workflows, not investment advice or real-time trading.

Synced the frontend catalog and marketing paths with newly active Geocoding, JustWatch, and TikTok Top Ads endpoints from backend Swagger.

• Added platform discovery for Geocoding and JustWatch with direct links into Docs and Playground.
• Expanded TikTok public positioning to include Top Ads list, detail, analysis, filter, location, recommendation, safety, spotlight, and suggestion endpoints.
• Aligned pricing examples, calculator scenarios, and credit labels with backend billing weights for the new endpoint families.
• Updated homepage, product, feature, platform, footer, getting-started, and recipe surfaces to reflect the active public catalog.
• Kept internal and deprecated endpoint groups out of public marketing promotion.

Synced the public FE catalog with newly active Yahoo Finance endpoints and added finance-focused discovery paths across platform pages, Docs, Playground, pricing, and marketing surfaces.

• Added the Yahoo Finance platform page with direct paths into endpoint docs and Playground examples.
• Published quote, search, historical price, market summary, calendar, screener, financial, options, and ticker module endpoints from the generated catalog.
• Aligned pricing examples and endpoint credit labels with backend billing weights for the new Yahoo Finance family.
• Updated homepage, product, feature, platform, footer, and getting-started surfaces to include Yahoo Finance alongside Google Finance.
• Kept private and absent endpoint families out of public discovery surfaces.

Expanded the public catalog and marketing paths for newly active travel, property, search discovery, podcast, finance, business, marketplace, app-store, and social data APIs.

• Added public platform pages for Airbnb, TripAdvisor, and Zillow with direct paths into endpoint docs and Playground examples.
• Expanded Bing and Brave coverage for web, image, news, video, and suggestion workflows.
• Added Google Jobs, Google Suggest, and Google Trends discovery coverage alongside Google Search, Maps, and Finance surfaces.
• Aligned pricing examples and calculator data with the active backend billing catalog for the new public endpoints.
• Kept deprecated Etsy routes and non-public account APIs out of public platform, footer, and conversion surfaces.

Expanded and refined endpoint coverage across travel lodging, destination research, trend discovery, and property workflows so more common research jobs can use normalized API responses.

• Improved Airbnb review, calendar, and search coverage for travel research flows.
• Restored TripAdvisor discovery for destination, venue, place, and review workflows.
• Added Google Trends discovery for explore, trending, related query, regional interest, category, location, and enum endpoints.
• Expanded Zillow responses with richer typed property details.
• Kept newly available endpoint groups aligned with public Docs, Playground, and availability checks.

Hardened target-specific extraction paths and tightened endpoint lifecycle handling so unavailable or private surfaces fail more predictably.

• Improved browser-backed fallbacks for harder target pages that need rendered HTML before extraction.
• Preserved typed availability errors so clients can distinguish blocked, unavailable, and unsupported cases.
• Cleaned up deprecated and private endpoint surfaces so public catalogs stay focused on supported APIs.

Polished public discovery paths for Docs and Playground by canonicalizing endpoint routes and keeping internal-only groups out of public catalogs.

• Canonicalized endpoint-specific Playground URLs for cleaner public routing.
• Documented the catalog cleanup workflow used when an endpoint should stop appearing publicly.
• Removed deprecated and internal API groups from public discovery surfaces while preserving supported customer flows.

Improved the runtime paths behind browser-assisted APIs so target probes, rendered fetches, and long-running checks complete more consistently.

• Hardened timeout behavior for longer diagnostics and protected target checks.
• Improved browser capacity placement for workloads that need rendered extraction.
• Added local verification helpers and readiness checks that make future runtime changes easier to validate.

Added the customer referral flow, reward tracking, and review controls needed to run referral campaigns from signup through approval.

• Referral links now carry through signup so customers can invite other teams.
• Console and admin surfaces support reviewing referral activity and reward state.
• Billing and usage views have the referral context needed for account-level follow-up.

Improved the execution path for public pages that need browser rendering or extra challenge handling before structured data can be extracted.

• Expanded browser-backed execution coverage for harder upstream pages.
• Improved fallback behavior so API responses fail more transparently when an upstream page cannot be used.
• Documented the reliability lessons that matter for future endpoint work.

Expanded audio and podcast coverage with more Spotify surfaces and synchronized the public catalog so Docs, Playground, and pricing stay aligned.

• Added more Spotify profile, playlist, podcast, chart, and recommendation coverage.
• Synced the endpoint catalog into public Docs, Playground, pricing, and platform pages.
• Updated monitoring coverage for newly documented API surfaces after rollout.

Polished public API discovery pages so endpoint families, platform groups, and filters are easier to scan on desktop and mobile.

• Improved Docs filtering layout and platform group grids.
• Tightened homepage and product page sections around real endpoint examples.
• Added regression coverage for public page layout and catalog synchronization.

Expanded infrastructure health coverage, recovery guidance, and cleanup automation for service-critical platform components.

• Added alerts for resource pressure that can affect public API availability.
• Improved recovery guidance for service health issues after infrastructure churn.
• Added cleanup automation for long-running observability data so health checks stay responsive.

Consolidated marketing navigation and feature pages around the API capabilities customers evaluate before integration.

• Grouped infrastructure features into a clearer product feature hub.
• Made the homepage endpoint example visible by default.
• Connected feature pages to Docs, pricing, Playground, and supported platforms.

Improved public site quality checks and moved mutable site settings into a safer runtime configuration path.

• Added stronger preview quality checks for public pages.
• Separated runtime settings from the built site where possible.
• Normalized public host references across site pages and examples.

Improved how customers can check Crawlora service status from public pages and subscribe to operational updates.

• Added a public status signal to the site footer.
• Improved the public status page feed experience.
• Kept status checks lightweight for faster public page loads.

Added public build metadata, preview restart shortcuts, and hosted preview support for faster site verification.

• Added a public freshness signal for the running site build.
• Improved preview restart ergonomics for public-page review.
• Allowed hosted preview checks to work cleanly during site development.

Refined public Playground routing and indexing behavior so clean endpoint pages are discoverable while private app routes stay out of search results.

• Improved clean public Playground routes for endpoint-specific testing.
• Kept authenticated app routes separate from public discovery.
• Added coverage for public indexing rules and endpoint route generation.

Expanded the public website into a more complete evaluation path for buyers and developers.

• Added feature, comparison, use-case, pricing, and developer guide pages.
• Aligned the web scraping API overview with real docs and samples.
• Improved public metadata and route coverage for organic discovery.

Added and exercised browser backend options used by harder endpoint families that need a rendered page before extraction.

• Expanded browser backend choices for difficult public pages.
• Improved service routing for rendered extraction paths.
• Kept API availability checks aligned with the new browser-backed paths.

Improved sign-in options, account profile presentation, and layout resilience across public platform pages.

• Added social sign-in support and account profile avatars.
• Hardened platform landing pages against layout overflow.
• Improved form behavior around API key fields and browser autofill.

Improved billing actions, active key visibility, usage period defaults, and pricing-to-console handoff.

• Connected pricing plan actions to billing workflows.
• Improved active API key selection and display in Playground and console pages.
• Defaulted usage views to the current billing period for more relevant reporting.

Aligned public domains, frontend routing, API host references, and MCP access points around the Crawlora brand.

• Moved public examples and site links onto Crawlora hostnames.
• Exposed MCP access from the public product surface.
• Linked pricing and billing flows more directly from the public site.

Strengthened session restore, auth-state loading, Playground parameter feedback, and API key workflows.

• Preserved sign-in status across browser refreshes and restarts.
• Clarified request parameter errors and sample response handling in Playground.
• Added focused loading states for overview, API key, and usage sections.

Reworked the homepage, platform pages, footer links, and docs metadata around the live public catalog.

• Made homepage sections catalog-driven with product endpoint examples.
• Added platform landing pages and footer navigation for supported public surfaces.
• Improved API contract metadata for docs, Playground, and agent-facing surfaces.

Added reliability checks, clearer failure classifications, and endpoint deprecation practices for older surfaces.

• Added scraper reliability auditing and guardrails.
• Clarified behavior for deprecated endpoint families.
• Documented lessons for browser-backed and stealth-sensitive endpoint work.

Improved documentation consistency, deprecated endpoint UX, and admin repair paths used during catalog maintenance.

• Improved docs cross-links and catalog consistency.
• Made retired endpoint states easier to understand.
• Added clearer repair paths for maintaining endpoint metadata.

Improved schema views, Playground forms, billing previews, and admin dashboards for day-to-day operations.

• Enhanced Playground discovery, forms, and schema inspection.
• Improved billing preview and admin data flows.
• Added clearer operational filters for console workflows.

Launched the first Crawlora web experience with public marketing pages, dashboard foundations, and console workflows.

• Added the first public site and console experience.
• Introduced dashboard pages for account and API usage workflows.
• Hardened sensitive console flows before broader public use.

Added the public status foundation and expanded route monitoring around the public API surface.

• Added a public status foundation for customer-facing service checks.
• Improved public API route monitoring and hostname checks.
• Expanded startup and availability budgets for smoother service updates.

Expanded service monitoring, public routing checks, and operational dashboards around the Crawlora API surface.

• Added application metrics and alert coverage for API health.
• Improved public route checks for customer-facing endpoints.
• Expanded dashboards for capacity, traffic, and service health.

Expanded professional network and web intelligence endpoints for company research and competitive analysis.

• Added broader LinkedIn company, product, and profile coverage.
• Expanded web intelligence search and detail surfaces.
• Updated public docs around the new business research coverage.

Strengthened the authenticated request path and broadened test coverage around important API contracts.

• Added static API key support for production-style requests.
• Broadened core endpoint test coverage.
• Added focused parsing checks for Google Maps search behavior.

Improved source quality reporting, validation throughput, and traffic accounting for the extraction network.

• Added source quality reporting for network inputs.
• Improved validation scheduling and throughput measurement.
• Persisted traffic and source refresh counters for clearer operational review.

Added mobile app store endpoint families with docs, examples, and integration coverage.

• Added App Store scraper APIs and documentation.
• Added Google Play scraper APIs, examples, and handler coverage.
• Expanded service and integration tests for mobile app data workflows.

Refreshed search endpoint behavior and browser impersonation support for search result extraction.

• Adjusted Google Search extraction for upstream changes.
• Improved browser impersonation support for harder pages.
• Kept search result parsing aligned with current page behavior.

Added Instagram profile, post, and reel coverage with documentation for social data use cases.

• Added Instagram profile extraction.
• Added post and reel coverage.
• Documented social endpoint behavior for developers.

Expanded YouTube transcript and community data alongside Product Hunt search and product discovery endpoints.

• Added YouTube caption, comment, and tag coverage.
• Added Product Hunt product and user search coverage.
• Improved docs for product discovery workflows.

Added Crunchbase organization and person coverage plus YouTube subtitle extraction for research workflows.

• Added company and person research endpoints.
• Added YouTube subtitle extraction.
• Expanded worker-backed enrichment for business data.

Expanded local business enrichment with business type, contact, status, and place data improvements.

• Added business type and place enrichment.
• Expanded website status and contact enrichment workflows.
• Improved Google Maps place and search behavior for local business research.

Expanded TikTok search, creator, hashtag, trend, and video data for social intelligence workflows.

• Added TikTok search user and trending documentation.
• Added creator and hashtag trend coverage.
• Expanded popular trend video and hashtag detail APIs.

Added Google Maps search coverage, documentation, and usage monitoring improvements.

• Added Google Maps search and place extraction.
• Updated documentation for local business workflows.
• Improved API usage monitoring around authenticated requests.

Added API usage monitoring and account identity improvements that later supported console billing and usage pages.

• Added API usage monitoring foundations.
• Added account email and username fields.
• Improved notification handling for account events.

Expanded endpoint families across Google Finance, Amazon, TikTok, Similarweb, and general web parsing.

• Added finance search, ticker, quote, income statement, balance sheet, cash flow, and crypto coverage.
• Added Amazon suggestion, search, and product improvements.
• Added TikTok post, comment, category, explore, and trending coverage.

Added LinkedIn endpoint coverage for people, schools, products, and jobs.

• Added LinkedIn people and school data extraction.
• Added LinkedIn product page coverage.
• Added LinkedIn job data coverage for recruiting and market research workflows.

Added Google Search, News, Image, and Shopping extraction with API documentation.

• Added Google Search result extraction.
• Added Google News and Image Search coverage.
• Added Google Shopping search and detail coverage.

Added sign-in, sign-up, rate limiting, metrics, and the first structured API documentation updates.

• Added account sign-in and sign-up foundations.
• Added rate limiting for fair API usage.
• Added metrics and documentation updates for early API consumers.

Started the platform foundation that later powered endpoint coverage, availability checks, and structured API delivery.

• Established early network validation and peer health workflows.
• Added the first operational building blocks for high-volume extraction.
• Created the base that later supported public API endpoints and monitoring.