Website Content Crawler Lite

Crawl public website pages and export clean text, Markdown, HTML, metadata, and discovered links.

Use this actor when you need a lightweight CSV, JSON, Excel, or API export of website content for AI/RAG ingestion, SEO audits, content monitoring, lead research, documentation indexing, or automation workflows.

At a glance

• Input: Start URLs, crawl depth, page limit, domain scope, include/exclude globs, output format, robots.txt setting, and timeout.
• Output: Requested URL, loaded URL, title, meta description, H1, text, Markdown or HTML, links, status code, content type, depth, parent URL, and scrape status.
• Output formats: Choose markdown, text, or html.
• Best for: Website-to-Markdown exports, lightweight crawling, AI knowledge ingestion, SEO checks, and content monitoring.
• Robots-aware: Can respect robots.txt before crawling pages.

What can it do?

• Crawl public website pages: Follow links from start URLs within your configured scope.
• Export clean content: Save readable text plus Markdown or HTML output.
• Collect page metadata: Capture title, meta description, H1, status code, content type, depth, and parent URL.
• Discover links: Save absolute links found on each crawled page.
• Use as a website content API: Run from the Apify UI, API, schedules, webhooks, or the official Apify MCP server.

Common workflows

• AI/RAG ingestion: Export page Markdown for embeddings or knowledge bases.
• SEO audits: Collect title, description, H1, status code, and internal link data.
• Content monitoring: Schedule runs to track page text or metadata changes.
• Documentation indexing: Crawl docs pages and save Markdown rows.
• Research pipelines: Feed clean website text into downstream extraction or classification tools.

Input example

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxPages": 25,
  "maxDepth": 1,
  "sameDomainOnly": true,
  "includeGlobs": [],
  "excludeGlobs": ["**/login**", "**/signup**"],
  "outputFormat": "markdown",
  "respectRobotsTxt": true,
  "requestTimeoutSecs": 20
}

Output example

{
  "url": "https://example.com",
  "loadedUrl": "https://example.com/",
  "title": "Example Domain",
  "description": "Example description",
  "h1": "Example Domain",
  "text": "Example Domain This domain is for use in illustrative examples...",
  "markdown": "Example Domain\\n\\nThis domain is for use in illustrative examples...",
  "links": ["https://www.iana.org/domains/example"],
  "statusCode": 200,
  "contentType": "text/html",
  "depth": 0,
  "parentUrl": null,
  "fetchedAt": "2026-07-03T10:00:00.000Z"
}

Tips for best results

• Start with maxDepth: 0 or 1 before allowing a broader crawl.
• Keep sameDomainOnly enabled unless you intentionally want to crawl external links.
• Use includeGlobs for docs, blog, or product sections.
• Use excludeGlobs to skip login, signup, cart, checkout, search, and account pages.
• Choose markdown for RAG and LLM workflows; choose text when you want smaller rows.

Limits and caveats

• This is a lightweight crawler, not a full browser renderer. Heavy JavaScript pages may return limited content.
• The actor crawls public pages only and should not be used for private or logged-in pages.
• Non-HTML content is skipped and recorded with a skip reason.
• Robots.txt rules can skip pages when respectRobotsTxt is enabled.

API usage

curl "https://api.apify.com/v2/acts/fetch_cat~website-content-crawler-lite/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://example.com" }],
    "maxPages": 25,
    "maxDepth": 1,
    "outputFormat": "markdown"
  }'

MCP and AI agents

You can run this actor through the official Apify MCP server at https://mcp.apify.com.

For a focused single-actor tool, use:

https://mcp.apify.com?tools=fetch_cat/website-content-crawler-lite

Agent-friendly inputs are startUrls, maxPages, maxDepth, sameDomainOnly, includeGlobs, excludeGlobs, outputFormat, respectRobotsTxt, and requestTimeoutSecs.

Support

If a run fails, returns no data, or a field looks wrong, open an issue from the Actor page.

Please include the Apify run ID or run URL, input JSON, one example public URL, query, or input item, what you expected, and what the dataset returned. Small reproducible inputs make parsing or site-layout issues much faster to fix.