Horseman

Horseman is a configurable desktop web crawler that runs JavaScript snippets across an entire site to extract custom data. It includes over 120 built in snippets, GPT3.5 integration for page analysis, an AI helper that writes snippets from plain language, and an Insights view for drilling into issues.

Horseman is a desktop web crawler for Windows, macOS (including Apple Silicon), and Linux that lets you define exactly what data to pull from every page of a site. At its core are snippets: small pieces of JavaScript that run in the context of each crawled page, manipulate it, and return structured results. If you can run something in Chrome's DevTools console, you can automate it across an entire site with Horseman. More than 120 built in snippets cover common technical audits such as detecting a Largest Contentful Image that loads at low priority, flagging elements that overflow the viewport, extracting readable content with Mozilla's readability.js, and scoring H1 sentiment. Version 0.3 adds GPT integration and an AI helper for people who do not write JavaScript. You can describe the information you want to extract in plain language and let the tool draft the snippet, or send page data to GPT3.5 with a prompt to summarize content, generate meta descriptions, or analyze a page. Results land in a table, and the Insights feature provides a deeper view that groups issues and lets you drill down into the pages that need attention. The tool is built for frontend developers, performance analysts, digital agencies, accessibility experts, SEO specialists, and JavaScript engineers who need repeatable, custom extraction rather than a fixed audit report. In practice you install the desktop app, log in with GitHub, choose or write snippets, start a crawl, and review the output. Compared with hosted crawlers that offer a fixed set of checks, Horseman trades setup time for flexibility: the snippet library covers many standard audits, but the real value appears when you write or generate snippets for site specific questions. The project is distributed directly by its developer through GitHub Sponsors rather than a conventional software subscription. That keeps the pricing low and the development independent, but it also means payment is tied to a GitHub account, device limits apply per tier, and the software is a small, evolving product rather than a large platform with a support organization behind it.