Domain-Scoped Site Crawler for Markdown Datasets: MCP config, n8n workflow and API

{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "maxCrawlDepth": 1,
  "maxCrawlPages": 3,
  "useSitemaps": false,
  "useLlmsTxt": false,
  "respectRobotsTxtFile": true,
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "outputFormat": "markdown",
  "htmlTransformer": "auto",
  "includeImages": false,
  "chunkMarkdown": false,
  "chunkMaxChars": 2000,
  "chunkOverlapChars": 200,
  "parsePdfText": true,
  "takeScreenshots": false,
  "storeSkippedUrls": false
}

---
name: domain-scoped-site-crawler-for-markdown-datasets
description: "One record per crawled page: Markdown/text/HTML content, title, description, author, dates, canonical URL, links, images, OpenGraph, JSON-LD, word count, approximate token count, optional RAG chunks, optional screenshot URL, and content hash. Runs the Apify Actor vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets. Use when the user needs Domain-Scoped Site data with fields such as url, title, depth, wordCount, httpStatusCode and an Apify token is available."
compatibility: "Needs network access, curl, and an APIFY_TOKEN environment variable. Runs are billed to the token owner's Apify account."
metadata:
  actor: "vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets"
  store-url: "https://apify.com/vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets"
---

# Domain-Scoped Site Crawler for Markdown Datasets

Use this skill to fetch Domain-Scoped Site data with the Apify Actor `vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets` and work with the returned rows.

## Before running

1. Check that `APIFY_TOKEN` is set in the environment. Never print it or write it to a file.
2. Build `input.json` from the user's request using the fields below. Keep limits small for a first run.
3. Confirm the search terms or URLs with the user if the request is ambiguous.

## Input fields

| Field | Type | Meaning |
| --- | --- | --- |
| `startUrls` | array, required | One or more URLs to start crawling from. The crawler follows links found on these pages and stays on the domain of the first start URL. |
| `maxCrawlDepth` | integer | How many links deep to follow from the start URLs. 0 means only the start URLs are crawled. |
| `maxCrawlPages` | integer | Maximum number of pages (and files) to store. The crawl stops when this limit is reached. |
| `useSitemaps` | boolean | Look for sitemaps (from robots.txt and /sitemap.xml) and enqueue the URLs they list. Finds pages that are not linked from the start URLs. |
| `useLlmsTxt` | boolean | Look for /llms.txt at the start URL domains and enqueue the files listed there (common on documentation sites). |
| `respectRobotsTxtFile` | boolean | Skip discovered pages that the site's robots.txt disallows. Start URLs are always crawled. |
| `crawlOnlyPagesChangedSince` | string | Optional date (YYYY-MM-DD or full ISO date-time). Crawls only pages that may have changed since then and outputs a delta. Leave empty to crawl everything. |
| `includeUrlGlobs` | array | Only crawl discovered URLs matching these glob patterns (e.g. https://example.com/docs/*). Leave empty to allow everything on the domain. |
| `excludeUrlGlobs` | array | Skip discovered URLs matching any of these glob patterns (e.g. *login*, *.pdf). |
| `includeUrlRegexps` | array | Only crawl discovered URLs matching at least one of these regular expressions. Combined with glob patterns (both must pass when both are set). |
| `excludeUrlRegexps` | array | Skip discovered URLs matching any of these regular expressions. |
| `keepUrlFragments` | boolean | Treat URLs that differ only in #fragment as different pages. Enable for single-page sites that use fragments as routes. |
| `ignoreCanonicalUrl` | boolean | When disabled, pages whose canonical URL points to a page that was already crawled are skipped as duplicates. |
| `outputFormat` | one of `markdown`, `text`, `html` | How page content is returned. Markdown keeps headings, lists, tables, and links — best for LLMs and RAG. |

Example `input.json` (the Actor's published prefill):

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "maxCrawlDepth": 1,
  "maxCrawlPages": 3,
  "useSitemaps": false,
  "useLlmsTxt": false,
  "respectRobotsTxtFile": true,
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "outputFormat": "markdown",
  "htmlTransformer": "auto",
  "includeImages": false,
  "chunkMarkdown": false,
  "chunkMaxChars": 2000,
  "chunkOverlapChars": 200,
  "parsePdfText": true,
  "takeScreenshots": false,
  "storeSkippedUrls": false
}
```

## Run

```bash
curl -sS -X POST "https://api.apify.com/v2/acts/vexaloom-labs~domain-scoped-site-crawler-for-markdown-datasets/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @input.json > results.json
```

The call waits for the run to finish and returns the dataset items as a JSON array. If the run needs more than 300 seconds the API answers with HTTP 408; then start the run with `POST /v2/acts/vexaloom-labs~domain-scoped-site-crawler-for-markdown-datasets/runs`, poll `GET /v2/actor-runs/{runId}` until `SUCCEEDED`, and read `GET /v2/datasets/{defaultDatasetId}/items`.

## Output

Each item is one record. Published fields include: `url`, `title`, `depth`, `wordCount`, `httpStatusCode`.

Summarise what was found, cite the record URLs when present, and say how many items came back. Do not invent values for empty fields.