Crawler.sh
Free Local AEO & SEO Spider and a Markdown content extractor
Visit website →About
Crawler.sh is a fast, local-first web crawler and SEO/AEO analysis tool. It crawls entire sites in seconds from the terminal or a native desktop app, runs automated SEO checks, and extracts page content as clean Markdown.
Results can be exported to JSON, CSV, or sitemap XML, and because it runs locally, crawls are not gated by a hosted service's limits or pricing.
It is aimed at SEO practitioners, developers, and content teams who want to audit sites and pull structured content quickly, and who are also thinking about answer-engine optimisation (AEO), how well pages surface in AI-generated answers.
AI-assisted draft, human-reviewed before publishing — see how we choose & review tools.
Why we picked it
Crawler.sh stood out for running entirely local with no hosted-service limits or pricing gate — crawling a whole site from the terminal in seconds is worth naming for someone who's hit a crawl-limit paywall on a hosted SEO tool mid-audit.
Related tools
- 1 Articos Test messaging against simulated buyers at 86% research accuracy Freemium
- 2 Manifold MCP One hosted MCP server for growth data across every channel Freemium
- 3 MePersona AI inbox that drafts fan replies in your voice across 12 platforms Freemium
- 4 ReSO AI Generative engine optimization to get your brand into AI search answers Free
- 5 Zernio One API to publish, schedule and analyze across 17 social platforms Freemium
- 6 OpenSEO Open-source Semrush alternative, works with your agent over MCP Freemium