WebCrawler API

WebCrawler API is a coding tool. WebCrawler API crawls websites and docs into clean markdown for AI agents, handling JavaScript, CAPTCHAs and proxies automatically. It charges per page with no subscription required, and offers change-tracking feeds and no-code connectors.

WebCrawler API is a developer-focused service that crawls websites, documentation sites and help centers and returns the content as cleaned markdown. A single request can pull data from all pages of a site. The markdown has menus, cookie banners, footers and ads stripped out, so the output can go straight into a prompt, a vector index or storage without another cleanup pass. The service targets teams building AI support bots, knowledge products and similar agents that depend on fresh, readable web content. Instead of running proxies, headless browsers and retry logic themselves, users send a request and the platform handles JavaScript rendering, CAPTCHA solving, residential proxies, rate limits and anti-bot protection, routing each request along the fastest path that works. Frequently requested pages are served from a smart cache, with a max_age=0 parameter to bypass it when live data is needed. In practice, a user signs up, gets an API key and calls the crawl endpoint with a URL and an item limit. Code samples cover cURL, Node.js, Python, PHP and Java, and an optional LLM-cleaned markdown mode produces a more polished result at a higher per-page rate. A Feeds feature watches a site and returns only pages that changed, including diffs, new entries and removed ones, which helps keep a knowledge base current without polling. For people who prefer not to code, guides cover Zapier, Make, n8n and Integrately. Pricing is built around usage: a pay-as-you-go plan with no subscription, plus monthly tiers that lower the per-page rate and raise the number of parallel requests. Compared with self-managed scrapers built on headless browsers, the appeal is operational simplicity. Compared with general scraping platforms, the focus on markdown output for LLM workflows is the distinguishing trait. Teams needing heavy custom extraction logic or structured field-level parsing may find the markdown-first approach narrower than broader scraping suites.