arXiv Paper Search Scraper

Search public arXiv paper metadata by keyword, author, category, sort order, and publication date, then export titles, abstracts, authors, IDs, DOI, journal references, abstract links, and PDF links.

arXiv Paper Search Scraper is a lightweight research-data actor for literature review seed lists, paper monitoring, AI research feeds, newsletter curation, and dataset pipelines.

At a glance

• Input: required query, optional categories, author, date filters, sorting, and maxItems.
• Output: paper title, abstract, authors, arXiv ID, categories, published/updated dates, DOI, journal reference, abstract URL, and PDF URL.
• Controls: filter by arXiv categories, author name, publication date range, sort field, and sort order.
• Exports: download JSON, CSV, Excel, XML, RSS, or HTML, or read results through API/MCP.
• Best for: literature search, recent-paper monitoring, research newsletters, lab alerts, and paper metadata enrichment.

Who is it for?

This actor is for researchers, data analysts, AI teams, newsletter editors, product teams, and academic-monitoring workflows that need structured arXiv paper metadata without manually copying search results. It is especially useful when you need repeatable paper discovery, scheduled topic monitoring, or clean exports for spreadsheets, databases, dashboards, or AI-agent research summaries.

Legality

This actor collects publicly available arXiv metadata returned by arXiv's public search endpoints. You are responsible for using the results lawfully, respecting arXiv policies and applicable copyright/database rules, and avoiding inputs that collect or process personal or sensitive data unnecessarily. The actor does not bypass logins, access private accounts, or download paywalled content.

Input recipes

Use these concrete input patterns when starting a new run:

• Recent LLM Papers: query: "large language models", categories: ["cs.CL", "cs.AI"], sortBy: "submittedDate", sortOrder: "descending", maxItems: 25.
• Graph Neural Network Review: query: "graph neural networks", categories: ["cs.LG"], sortBy: "relevance", maxItems: 30.
• Author Watch: query: "representation learning", author: "Yoshua Bengio", maxItems: 20.
• Quantum Error Correction Scan: query: "quantum error correction", categories: ["quant-ph"], sortBy: "submittedDate", maxItems: 25.
• Date-Bounded AI Safety Search: query: "AI safety", categories: ["cs.AI"], startDate: "2025-01-01", endDate: "2025-12-31", maxItems: 25.

What can it do?

• Search arXiv metadata: query public arXiv records by keyword, phrase, category, author, and publication date.
• Export paper discovery rows: save titles, abstracts, authors, arXiv IDs, categories, dates, DOI, journal references, abstract links, and PDF links.
• Monitor recent research: sort by submitted or updated date and schedule the same query for recurring paper feeds.
• Build literature-review seed lists: collect focused records for research notes, lab alerts, newsletters, or dataset enrichment.
• Use as an arXiv data workflow: export CSV/Excel/JSON, run through the Apify API, or expose the Actor to AI agents through Apify MCP.

Input example

{
  "query": "large language models",
  "categories": ["cs.CL", "cs.AI"],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 25
}

Output example

{
  "atomId": "http://arxiv.org/abs/2401.00001v1",
  "arxivId": "2401.00001v1",
  "paperTitle": "Example paper title",
  "summary": "Paper abstract text...",
  "authors": ["Jane Doe", "John Smith"],
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.AI"],
  "publishedAt": "2024-01-01T00:00:00Z",
  "updatedAt": "2024-01-02T00:00:00Z",
  "comment": "12 pages",
  "journalRef": null,
  "doi": null,
  "absUrl": "http://arxiv.org/abs/2401.00001v1",
  "pdfUrl": "http://arxiv.org/pdf/2401.00001v1",
  "query": "large language models",
  "position": 1
}

Common workflows

• Literature review seed list: search a focused topic and export titles, abstracts, authors, and PDF links to a spreadsheet.
• Recent-paper monitoring: sort by submittedDate descending and schedule the same query weekly.
• Author tracking: combine query and author to follow a researcher or lab topic.
• Category feeds: use arXiv category codes to keep AI, NLP, computer vision, statistics, or quantum searches focused.
• Research enrichment: send abstracts, authors, categories, and URLs to a database, vector index, or newsletter workflow.

Tips for best results

• Use precise terms: retrieval augmented generation is easier to review than AI.
• Add categories: category filters reduce off-topic matches for broad topics.
• Use date filters carefully: startDate and endDate are applied to publication dates after arXiv returns results.
• Respect arXiv pacing: larger runs take longer because the actor paginates through public arXiv API results.
• Check optional fields: DOI, journal reference, and comments are present only when authors provide them.

Limits and caveats

• Public metadata only: the actor exports arXiv metadata, not paywalled paper content.
• No citation graph: it does not collect citation counts or references.
• Date filtering is post-fetch: very broad queries with narrow dates may scan more rows than they save.
• Normal workload design: keep queries focused for repeatable monitoring and predictable run times.

API usage

Node.js example:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/arxiv-paper-search-scraper').call({
  query: 'large language models',
  categories: ['cs.CL', 'cs.AI'],
  sortBy: 'submittedDate',
  sortOrder: 'descending',
  maxItems: 25
});
console.log(run.defaultDatasetId);

Python example:

from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')
run = client.actor('fetch_cat/arxiv-paper-search-scraper').call(run_input={
    'query': 'large language models',
    'categories': ['cs.CL', 'cs.AI'],
    'sortBy': 'submittedDate',
    'sortOrder': 'descending',
    'maxItems': 25,
})
print(run['defaultDatasetId'])

cURL example:

curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~arxiv-paper-search-scraper/runs?token=YOUR_APIFY_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"query":"large language models","categories":["cs.CL","cs.AI"],"sortBy":"submittedDate","sortOrder":"descending","maxItems":25}'

MCP and AI agents

Use the official Apify MCP server for agent workflows. The default server can search and run Actors; the focused URL exposes only this Actor.

Claude CLI add command:

claude mcp add apify-arxiv --transport http "https://mcp.apify.com?tools=fetch_cat/arxiv-paper-search-scraper"

JSON config block:

{
  "mcpServers": {
    "apify-arxiv": {
      "url": "https://mcp.apify.com?tools=fetch_cat/arxiv-paper-search-scraper"
    }
  }
}

Example prompts for MCP usage:

• "Use arXiv Paper Search Scraper to find 25 recent large language model papers in cs.CL and cs.AI, then summarize titles, authors, categories, and PDF links."
• "Run an arXiv search for graph neural networks in cs.LG and return a CSV-ready table with arXiv IDs, authors, abstracts, and PDF URLs."
• "Monitor new AI safety papers in cs.AI from 2025 and list DOI, journal reference, and PDF link when available."

Support

If a run fails, returns no data, or a field looks wrong, open an issue from the Actor page.

Please include the Apify run ID or run URL, input JSON, one example public URL, query, or input item, what you expected, and what the dataset returned. Small reproducible inputs make parsing or site-layout issues much faster to fix.

Privacy and data handling

This Actor runs with Apify limited permissions and only processes data needed for the documented run. It uses the inputs you provide and the public records needed to produce the documented dataset to produce the output dataset and sends requests to arXiv public search/API endpoints; results are stored in Apify run storage for your account. FetchCat does not use your inputs or outputs for advertising, does not use them for model training, and does not retain them outside the Apify run except for transient support debugging when you explicitly share run details. You are responsible for using the Actor lawfully, respecting the target site's terms, and avoiding unnecessary personal or sensitive data in inputs.