arXiv Paper Search Scraper

Search public arXiv paper metadata by keyword, author, category, sort order, and publication date, then export titles, abstracts, authors, IDs, DOI, journal references, abstract links, and PDF links.

arXiv Paper Search Scraper is a lightweight research-data actor for literature review seed lists, paper monitoring, AI research feeds, newsletter curation, and dataset pipelines.

At a glance

• Input: required query, optional categories, author, date filters, sorting, and maxItems.
• Output: paper title, abstract, authors, arXiv ID, categories, published/updated dates, DOI, journal reference, abstract URL, and PDF URL.
• Controls: filter by arXiv categories, author name, publication date range, sort field, and sort order.
• Exports: download JSON, CSV, Excel, XML, RSS, or HTML, or read results through API/MCP.
• Best for: literature search, recent-paper monitoring, research newsletters, lab alerts, and paper metadata enrichment.

Input recipes

Use these concrete input patterns when starting a new run:

• Recent LLM Papers: query: "large language models", categories: ["cs.CL", "cs.AI"], sortBy: "submittedDate", sortOrder: "descending", maxItems: 25.
• Graph Neural Network Review: query: "graph neural networks", categories: ["cs.LG"], sortBy: "relevance", maxItems: 30.
• Author Watch: query: "representation learning", author: "Yoshua Bengio", maxItems: 20.
• Quantum Error Correction Scan: query: "quantum error correction", categories: ["quant-ph"], sortBy: "submittedDate", maxItems: 25.
• Date-Bounded AI Safety Search: query: "AI safety", categories: ["cs.AI"], startDate: "2025-01-01", endDate: "2025-12-31", maxItems: 25.

What can it do?

• Search arXiv metadata: query public arXiv records by keyword, phrase, category, author, and publication date.
• Export paper discovery rows: save titles, abstracts, authors, arXiv IDs, categories, dates, DOI, journal references, abstract links, and PDF links.
• Monitor recent research: sort by submitted or updated date and schedule the same query for recurring paper feeds.
• Build literature-review seed lists: collect focused records for research notes, lab alerts, newsletters, or dataset enrichment.
• Use as an arXiv data workflow: export CSV/Excel/JSON, run through the Apify API, or expose the Actor to AI agents through Apify MCP.

Input example

{
  "query": "large language models",
  "categories": ["cs.CL", "cs.AI"],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 25
}

Output example

{
  "atomId": "http://arxiv.org/abs/2401.00001v1",
  "arxivId": "2401.00001v1",
  "paperTitle": "Example paper title",
  "summary": "Paper abstract text...",
  "authors": ["Jane Doe", "John Smith"],
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.AI"],
  "publishedAt": "2024-01-01T00:00:00Z",
  "updatedAt": "2024-01-02T00:00:00Z",
  "comment": "12 pages",
  "journalRef": null,
  "doi": null,
  "absUrl": "http://arxiv.org/abs/2401.00001v1",
  "pdfUrl": "http://arxiv.org/pdf/2401.00001v1",
  "query": "large language models",
  "position": 1
}

Common workflows

• Literature review seed list: search a focused topic and export titles, abstracts, authors, and PDF links to a spreadsheet.
• Recent-paper monitoring: sort by submittedDate descending and schedule the same query weekly.
• Author tracking: combine query and author to follow a researcher or lab topic.
• Category feeds: use arXiv category codes to keep AI, NLP, computer vision, statistics, or quantum searches focused.
• Research enrichment: send abstracts, authors, categories, and URLs to a database, vector index, or newsletter workflow.

Tips for best results

• Use precise terms: retrieval augmented generation is easier to review than AI.
• Add categories: category filters reduce off-topic matches for broad topics.
• Use date filters carefully: startDate and endDate are applied to publication dates after arXiv returns results.
• Respect arXiv pacing: larger runs take longer because the actor paginates through public arXiv API results.
• Check optional fields: DOI, journal reference, and comments are present only when authors provide them.

Limits and caveats

• Public metadata only: the actor exports arXiv metadata, not paywalled paper content.
• No citation graph: it does not collect citation counts or references.
• Date filtering is post-fetch: very broad queries with narrow dates may scan more rows than they save.
• Normal workload design: keep queries focused for repeatable monitoring and predictable run times.

API usage

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/arxiv-paper-search-scraper').call({
  query: 'large language models',
  categories: ['cs.CL', 'cs.AI'],
  sortBy: 'submittedDate',
  sortOrder: 'descending',
  maxItems: 25
});
console.log(run.defaultDatasetId);

MCP and AI agents

Use the official Apify MCP server for agent workflows. The default server can search and run Actors; the focused URL exposes only this Actor.

Focused Apify MCP server URL:

https://mcp.apify.com?tools=fetch_cat/arxiv-paper-search-scraper

Example agent prompt: "Search arXiv for 25 recent large language model papers in cs.CL and cs.AI, then summarize titles, authors, categories, and PDF links."

Support

If a run fails, returns no data, or a field looks wrong, open an issue from the Actor page.

Please include the Apify run ID or run URL, input JSON, one example public URL, query, or input item, what you expected, and what the dataset returned. Small reproducible inputs make parsing or site-layout issues much faster to fix.