Sitemap URL Extractor

Extract clean URL inventories from XML sitemaps and sitemap indexes.

Use this actor to turn public sitemap files into a structured dataset of URLs with source sitemap metadata, crawl depth, last modified dates, change frequency, priority, HTTP status, and discovery timestamps.

What does Sitemap URL Extractor do?

Sitemap URL Extractor reads website sitemap files and saves every discovered URL as a dataset item.

It supports both direct sitemap URLs and website root URLs.

If you enter a root such as https://apify.com, the actor tries common sitemap locations automatically.

If the sitemap is a sitemap index, the actor follows child sitemaps recursively until it reaches your limits.

You can filter URLs with include and exclude patterns.

You can deduplicate URLs across many sitemap files.

You can keep extraction limited to the same hostname as your original input.

Who is it for?

• 🔎 SEO teams auditing indexable URL inventories.
• 🧪 Technical SEO consultants checking sitemap coverage before crawls.
• 🕷️ Crawler operators seeding downstream crawling jobs with verified URLs.
• 📈 Growth engineers monitoring new pages by section.
• 🧰 Data teams collecting public URL lists for enrichment pipelines.
• 🧾 Content teams exporting article or documentation URLs.
• 🛠️ QA teams building URL regression test sets.

Why use this actor?

Sitemaps are one of the fastest ways to understand a public website structure.

Manual sitemap inspection gets slow when a site has dozens or hundreds of sitemap files.

This actor automates the repetitive parts:

• 🧭 Discover sitemap indexes.
• 🔁 Follow recursive sitemap files.
• 🧹 Normalize and deduplicate URLs.
• 🎯 Keep only sections you care about.
• 📦 Export results in JSON, CSV, Excel, XML, RSS, or HTML from Apify datasets.

Typical use cases

• Build a seed list for a crawler.
• Export all blog post URLs from a site.
• Compare sitemap URL counts over time.
• Find URLs that changed recently by using lastmod.
• Audit whether product, category, help, or documentation pages are present.
• Feed URLs into rank tracking, monitoring, or enrichment workflows.
• Create a clean URL list for link checking.

Input overview

The main input is startUrls.

You can provide one or more sitemap URLs.

You can also provide website roots.

Example sitemap input:

{
  "startUrls": [
    { "url": "https://apify.com/sitemap.xml" }
  ],
  "maxUrls": 100,
  "maxSitemaps": 25
}

Example root input:

{
  "startUrls": [
    { "url": "https://apify.com" }
  ],
  "maxUrls": 100,
  "maxSitemaps": 25
}

Input settings

Output data

Each dataset item is an extracted sitemap URL.

Example input

{
  "startUrls": [
    {
      "url": "https://apify.com/sitemap.xml"
    }
  ],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "includePatterns": [],
  "excludePatterns": [],
  "sameDomainOnly": true,
  "dedupe": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Example output

{
  "url": "https://apify.com/store",
  "sourceSitemapUrl": "https://apify.com/sitemap.xml",
  "sitemapDepth": 0,
  "loc": "https://apify.com/store",
  "lastmod": "2026-07-01",
  "changefreq": null,
  "priority": null,
  "contentType": "application/xml",
  "statusCode": 200,
  "discoveredAt": "2026-07-03T20:30:00.000Z"
}

How to run

1. Open the actor on Apify.
2. Add one or more sitemap URLs or website root URLs.
3. Keep the default maxUrls for a quick test.
4. Add include or exclude patterns if you only need part of a site.
5. Run the actor.
6. Export the dataset in your preferred format.

Filtering examples

Save only blog and docs URLs:

{
  "includePatterns": ["/blog/", "/docs/"]
}

Skip tag, author, and category pages:

{
  "excludePatterns": ["/tag/", "/author/", "/category/"]
}

Keep only product URLs:

{
  "includePatterns": ["/products/"]
}

Tips for best results

• Start with a small maxUrls value.
• Use direct sitemap URLs when you know them.
• Use root URLs when you want the actor to try common sitemap paths.
• Keep sameDomainOnly enabled unless you expect cross-domain sitemap entries.
• Use include patterns to reduce noise before exporting data.
• Use exclude patterns to remove archive, tag, or parameterized pages.
• Disable dedupe only when you need to see duplicate sitemap coverage.

Limits and safeguards

maxUrls stops the run after enough URL records are saved.

maxSitemaps stops recursive sitemap discovery after enough sitemap files are fetched.

These limits protect your budget and make large websites easier to test.

Some websites expose millions of URLs across many sitemap files.

For those sites, run section-specific sitemaps or use include patterns.

Handling sitemap indexes

Many websites use a sitemap index.

A sitemap index is a sitemap file that points to other sitemap files.

This actor follows those child sitemap files automatically.

The sitemapDepth field shows how far a URL was from the original sitemap.

Handling website root URLs

When you enter a root URL, the actor tries common sitemap locations.

For example, https://example.com may expand to common sitemap paths such as /sitemap.xml.

If you already know the exact sitemap file, enter it directly for faster runs.

Integrations

Use the extracted URL dataset with:

• Apify crawlers.
• SEO audit tools.
• Link checkers.
• Content inventory spreadsheets.
• Data enrichment workflows.
• Monitoring jobs that compare datasets over time.
• Webhook automations that trigger when a new run finishes.

API usage

Run Sitemap URL Extractor from your own code with the Apify API.

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const input = {
  "startUrls": [
    {
      "url": "https://apify.com/sitemap.xml"
    }
  ],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "includePatterns": [],
  "excludePatterns": []
};

const run = await client.actor('fetch_cat/sitemap-url-extractor').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient
import os

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("fetch_cat/sitemap-url-extractor").call(run_input={
  "startUrls": [
    {
      "url": "https://apify.com/sitemap.xml"
    }
  ],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "includePatterns": [],
  "excludePatterns": []
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

cURL

curl -X POST "https://api.apify.com/v2/acts/fetch_cat~sitemap-url-extractor/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://apify.com/sitemap.xml"}],"maxUrls":20,"maxSitemaps":10,"includePatterns":[],"excludePatterns":[]}'

Use with AI agents via MCP

Sitemap URL Extractor can be used by AI assistants through the hosted Apify MCP server.

Claude Code setup

claude mcp add --transport http apify "https://mcp.apify.com?tools=fetch_cat/sitemap-url-extractor"

Claude Desktop, Cursor, or VS Code JSON config

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=fetch_cat/sitemap-url-extractor"
    }
  }
}

Example prompts

• "Run Sitemap URL Extractor with this input JSON and summarize the dataset."
• "Export the latest Sitemap URL Extractor results to a table I can review."
• "Schedule this Actor for monitoring and tell me what changed between runs."

Legality and ethics

This actor reads public sitemap files that websites publish for crawlers.

Only extract data you are allowed to use.

Respect website terms, robots policies, and applicable laws.

Do not use extracted URLs for spam, abuse, or unauthorized access.

Troubleshooting

Why did I get fewer URLs than expected?

Check maxUrls, maxSitemaps, include patterns, exclude patterns, and sameDomainOnly.

The website may also have multiple sitemap indexes not linked from the first file.

Why did a root URL return no results?

The site may not use a common sitemap path.

Try entering the exact sitemap URL from the site's robots.txt file or documentation.

Why are some metadata fields empty?

lastmod, changefreq, and priority are optional sitemap fields.

Many websites omit them.

Support

Report bugs, wrong output, blocked runs, or missing fields from the Actor page. Include the Apify run ID or run URL, your input JSON, what you expected, what the Actor returned, and one reproducible public URL so the issue can be tested quickly.