Crawler.sh: The Developer's "Anti-SaaS" Local SEO Crawler

2026-03-03 | ProductHunt | Official Site

Interface Breakdown: A sleek dark-themed dashboard with a URL input bar and a 'Start Crawl' button at the top. The card-style layout features a Live Feed of URLs and status codes (green for 200, red for 404), an SEO Issues panel highlighting 17 common problems (short titles, missing descriptions, thin content), and a donut chart showing HTTP status distribution. It feels more like a geeky terminal tool than traditional SEO software.

30-Second Quick Judgment

What is it?: A local SEO crawler written in Rust that can scan an entire site in seconds via terminal or desktop app. It checks for 16 types of SEO issues and extracts content into clean Markdown. It also claims to support AEO (Answer Engine Optimization) analysis.

Is it worth your attention?:

YES, if you:

• Frequently audit SEO but hate Screaming Frog's £149/year price tag and 500-URL free limit.
• Need to bulk-convert web pages to Markdown (for LLMs or content libraries).
• Love CLI tools and want to automate SEO checks in your scripts.
• Are curious about the AEO trend and want a tool that supports it.

NO, if you:

• Require Windows or Linux support (currently macOS only).
• Need JavaScript rendering (it can't crawl SPAs).
• Need enterprise features (team collaboration, APIs, cloud sync).
• Already have a mature, paid SEO toolchain that works for you.

How does it stack up?

The Three Big Questions

Is it for me?

Target User Personas:

• Indie Devs / Technical SEOs: Who prefer working in the terminal.
• Content Creators: Who need to bulk-extract web content into Markdown.
• AI App Developers: Who need "LLM-ready" web data.
• Small Teams: With limited budgets needing a solid SEO audit tool.

The Litmus Test:

• If you perform at least one SEO check a week → This saves time.
• If you need to turn competitor blogs into Markdown → This is perfect.
• If you only use Windows → Skip it for now; it won't run.

Is it useful?

ROI Judgment: If you're hitting the 500-URL wall on Screaming Frog's free version or need Markdown extraction, trying the CLI version costs nothing but a few minutes. If you have a full paid suite, there’s no urgent need to switch.

Does it delight?

Delight Points:

1. Rust Speed - "Crawling a whole site in seconds" isn't hyperbole; Rust's async concurrency makes it blazingly fast.
2. Markdown Extraction - Uses readability-rust for content and htmd for Markdown conversion—a feature most SEO crawlers lack.
3. CLI Automation - crawler crawl → crawler seo → crawler export. Three commands, and you're done.

Real User Voice:

"I made a desktop app and a cli tool that you can analyze your sites' SEO issues, broken links etc. Rust based, works faster than screamingfrog." — @mehmetkose (Founder)

"Building crawler.sh week 2. Distributing a desktop app for multiple OS's macos(arm64,x64), linux (*n) and windows hell of a job." — @mehmetkose (Founder, venting about cross-platform distribution)

To be honest, since it launched on March 2, 2026, real user reviews are scarce. The founder is currently the loudest voice.

For the Developers

Tech Stack

Core Implementation Highlights

1. NDJSON Streaming: The .crawl file is line-delimited JSON. The first line is metadata, followed by one page record per line. This allows real-time data processing without waiting for the crawl to finish—smart for large sites.

2. Markdown Pipeline: readability-rust (Rust port of Mozilla's algorithm) extracts the main content, which htmd then converts to Markdown. It includes word counts, authors, and excerpts, making the output perfect for RAG or LLM feeding.

Open Source Status

• Crawler.sh: Closed-source commercial product.
• Open Source Alternatives:
  • LibreCrawl - MIT licensed, Python/Flask, unlimited URLs.
  • SiteOne Crawler - Free and cross-platform.
  • Crawl4AI - Trending on GitHub, optimized for LLMs.
• Build Difficulty: Medium. 3-4 weeks for the Rust engine, 2-3 weeks for SEO logic, 2-3 weeks for the Tauri app. A solo dev could hit MVP in 2-3 months.

For the Product Managers

Pain Point Analysis

The "Why Now": SEO auditing is a consistent need, but "Markdown extraction" is becoming a high-frequency requirement in the AI era. Crawler.sh sits right at that intersection.

Competitive Differentiation

For the Early Adopters

Pricing Breakdown

Hidden Costs: None. No API keys, no cloud fees, no account required.

Quick Start Guide

1. Install CLI: Run the install script in your terminal to download the macOS binary.
2. First Crawl: crawler crawl https://your-site.com
3. View Info: crawler info output.crawl for an overview.
4. SEO Audit: crawler seo output.crawl to export the issue list.
5. Export Data: crawler export output.crawl --format json.

Time to value: 5-10 minutes.

The "Gotchas"

1. macOS Only: Windows/Linux users are out of luck for now.
2. Very Early Stage: Launched March 2026; expect bugs.
3. No JS Rendering: It might miss content on React/Vue-heavy sites.
4. Vague AEO Features: While AEO is in the name, the specific AEO checks aren't clearly defined yet.

Conclusion

The Bottom Line: Crawler.sh is an intriguing mix of "Rust + AEO + Markdown." However, it's extremely new, platform-restricted, and currently lacks community traction. It’s great for curious tech enthusiasts, but not yet ready to be your primary professional tool.

Resource Links

2026-03-03 | Trend-Tracker v7.3