Website Content Crawler Lite

Crawl public website pages quickly over HTTP and export clean text, Markdown, HTML, metadata, and discovered links with bounded concurrency, retries, robots.txt handling, optional proxies, and a machine-readable run summary.

Use this actor when you need a lightweight CSV, JSON, Excel, or API export of website content for AI/RAG ingestion, SEO audits, content monitoring, lead research, documentation indexing, or automation workflows.

At a glance

• Input: Start URLs, crawl depth, page limit, domain scope, include/exclude globs, output format, robots.txt, retries, concurrency, safe size limits, timeout, and optional proxy.
• Output: Requested URL, loaded URL, title, meta description, H1, text, Markdown or HTML, links, status code, content type, depth, parent URL, and scrape status.
• Output formats: Choose markdown, text, or html.
• Best for: Website-to-Markdown exports, lightweight crawling, AI knowledge ingestion, SEO checks, and content monitoring.
• Robots-aware: Can respect robots.txt before crawling pages.
• Failure-aware: Retries transient network, rate-limit, server, and challenge responses, then records actionable error rows and RUN_SUMMARY.
• Memory-safe defaults: Caps response size, exported content, discovered links, and concurrency to reduce out-of-memory failures.

What can it do?

• Crawl public website pages: Follow links from start URLs within your configured scope.
• Export clean content: Save readable text plus Markdown or HTML output.
• Collect page metadata: Capture title, meta description, H1, status code, content type, depth, and parent URL.
• Discover links: Save absolute links found on each crawled page.
• Use as a website content API: Run from the Apify UI, API, schedules, webhooks, or the official Apify MCP server.

Common workflows

• AI/RAG ingestion: Export page Markdown for embeddings or knowledge bases.
• SEO audits: Collect title, description, H1, status code, and internal link data.
• Content monitoring: Schedule runs to track page text or metadata changes.
• Documentation indexing: Crawl docs pages and save Markdown rows.
• Research pipelines: Feed clean website text into downstream extraction or classification tools.

Input example

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxPages": 25,
  "maxDepth": 1,
  "sameDomainOnly": true,
  "includeGlobs": [],
  "excludeGlobs": ["**/login**", "**/signup**"],
  "outputFormat": "markdown",
  "respectRobotsTxt": true,
  "requestTimeoutSecs": 20,
  "maxConcurrency": 3,
  "maxRequestRetries": 2
}

Output example

{
  "url": "https://example.com",
  "loadedUrl": "https://example.com/",
  "title": "Example Domain",
  "description": "Example description",
  "h1": "Example Domain",
  "text": "Example Domain This domain is for use in illustrative examples...",
  "markdown": "Example Domain\\n\\nThis domain is for use in illustrative examples...",
  "links": ["https://www.iana.org/domains/example"],
  "statusCode": 200,
  "contentType": "text/html",
  "depth": 0,
  "parentUrl": null,
  "fetchedAt": "2026-07-03T10:00:00.000Z"
}

Tips for best results

• Start with maxDepth: 0 or 1 before allowing a broader crawl.
• Keep sameDomainOnly enabled unless you intentionally want to crawl external links.
• Use includeGlobs for docs, blog, or product sections.
• Use excludeGlobs to skip login, signup, cart, checkout, search, and account pages.
• Choose markdown for RAG and LLM workflows; choose text when you want smaller rows.
• Keep the default concurrency and size limits until a representative crawl succeeds; increase them deliberately for larger sites.
• Add a proxy only when the target rate-limits or blocks direct requests.

Limits and caveats

• This is a lightweight crawler, not a full browser renderer. Heavy JavaScript pages may return limited content.
• The actor crawls public pages only and should not be used for private or logged-in pages.
• Non-HTML content is skipped and recorded with a skip reason.
• Robots.txt rules can skip pages when respectRobotsTxt is enabled.
• Redirects outside the configured domain scope and URLs resolving to private/local networks are blocked for safety.
• Responses larger than responseSizeBytes are rejected; exported text/Markdown/HTML is truncated at contentChars.
• A run with zero usable pages fails clearly instead of appearing successful with no results. Partial crawls stop cleanly at runTimeSecs, expose counts in RUN_SUMMARY, and save fingerprinted PENDING_WORK for an idempotent resurrection.

API usage

curl "https://api.apify.com/v2/acts/fetch_cat~website-content-crawler-lite/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://example.com" }],
    "maxPages": 25,
    "maxDepth": 1,
    "outputFormat": "markdown"
  }'

Node.js:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/website-content-crawler-lite').call({
  startUrls: [{ url: 'https://example.com' }],
  maxPages: 25,
  outputFormat: 'markdown',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('fetch_cat/website-content-crawler-lite').call(run_input={
    'startUrls': [{'url': 'https://example.com'}],
    'maxPages': 25,
    'outputFormat': 'markdown',
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

MCP and AI agents

You can run this actor through the official Apify MCP server at https://mcp.apify.com.

For a focused single-actor tool, use:

https://mcp.apify.com?tools=fetch_cat/website-content-crawler-lite

Claude Code CLI:

claude mcp add --transport http apify-website-crawler "https://mcp.apify.com?tools=fetch_cat/website-content-crawler-lite"

Generic MCP client configuration:

{
  "mcpServers": {
    "apify-website-crawler": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=fetch_cat/website-content-crawler-lite"
    }
  }
}

Example prompts:

• “Crawl the public documentation at https://example.com/docs to depth 1 and return Markdown for at most 20 pages.”
• “Extract titles, descriptions, H1s, and internal links from these five public URLs and summarize any failed pages.”

Agent-friendly inputs include startUrls, crawl scope and format controls, bounded concurrency/retries, response/content/link safety limits, and optional proxy configuration. Every run also exposes RUN_SUMMARY in the default key-value store.

Support

If a run fails, returns no data, or a field looks wrong, open an issue from the Actor page.

Please include the Apify run ID or run URL, input JSON, one example public URL, query, or input item, what you expected, and what the dataset returned. Small reproducible inputs make parsing or site-layout issues much faster to fix.

Privacy and data handling

This Actor runs with Apify limited permissions and only processes data needed for the documented run. It uses the inputs you provide and the public records needed to produce the documented dataset to produce the output dataset and sends requests to the website URLs you provide; results are stored in Apify run storage for your account. FetchCat does not use your inputs or outputs for advertising, does not use them for model training, and does not retain them outside the Apify run except for transient support debugging when you explicitly share run details. You are responsible for using the Actor lawfully, respecting the target site's terms, and avoiding unnecessary personal or sensitive data in inputs.