Website to markdown for RAG: a clean ingestion route
You get a scheduled crawl of a documentation site that files each page as markdown with its URL and title, ready to chunk and embed in a retrieval index. A page is filed again only when its URL is new or its markdown length changes, so the corpus picks up new and edited pages without refiling the rest.
Checked 2026-11-12
Do it now
The workflow from Website to markdown to Notion, ready to import into n8n: Domain-Scoped Site Crawler for Markdown Datasets on Apify, then Notion.
n8n workflowwebsite-to-markdown-to-notion.n8n.json
{
"name": "Website to markdown to Notion",
"nodes": [
{
"id": "7ae13a1e-9c39-494d-aa13-a06c30f44d23",
"name": "Run by hand",
"type": "n8n-nodes-base.manualTrigger",
"typeVersion": 1,
"position": [
0,
0
],
"parameters": {}
},
{
"id": "db9f5cce-ebde-42a7-a959-ea7ee228937b",
"name": "Every day",
"type": "n8n-nodes-base.scheduleTrigger",
"typeVersion": 1.2,
"position": [
0,
200
],
"parameters": {
"rule": {
"interval": [
{
"triggerAtHour": 8
}
]
}
},
"notes": "Activate the workflow to run on this schedule."
},
{
"id": "69301f72-d9d9-4438-a594-a2bd108f0457",
"name": "Run Apify Actor",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
260,
100
],
"parameters": {
"method": "POST",
"url": "https://api.apify.com/v2/actors/vexaloom-labs~domain-scoped-site-crawler-for-markdown-datasets/run-sync-get-dataset-items",
"authentication": "genericCredentialType",
"genericAuthType": "httpHeaderAuth",
"sendBody": true,
"specifyBody": "json",
"jsonBody": "{\n \"startUrls\": [\n {\n \"url\": \"https://docs.apify.com\"\n }\n ],\n \"maxCrawlDepth\": 1,\n \"maxCrawlPages\": 3,\n \"useSitemaps\": false,\n \"useLlmsTxt\": false,\n \"respectRobotsTxtFile\": true,\n \"keepUrlFragments\": false,\n \"ignoreCanonicalUrl\": false,\n \"outputFormat\": \"markdown\",\n \"htmlTransformer\": \"auto\",\n \"includeImages\": false,\n \"chunkMarkdown\": false,\n \"chunkMaxChars\": 2000,\n \"chunkOverlapChars\": 200,\n \"parsePdfText\": true,\n \"takeScreenshots\": false,\n \"storeSkippedUrls\": false\n}",
"options": {
"timeout": 310000
}
},
"notes": "Actor: vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets. Select a Header Auth credential: name Authorization, value Bearer + your Apify token. Edit the JSON body to change the input.",
"notesInFlow": false
},
{
"id": "e0c2145e-33df-4515-af0d-8a64f9b4f913",
"name": "Only records not sent before",
"type": "n8n-nodes-base.removeDuplicates",
"typeVersion": 2,
"position": [
500,
100
],
"parameters": {
"operation": "removeItemsSeenInPreviousExecutions",
"logic": "removeItemsWithAlreadySeenKeyValues",
"dedupeValue": "={{ $json.url }}#{{ String($json.markdown || '').length }}",
"options": {
"scope": "node",
"historySize": 10000
}
},
"notes": "Remembers each record it has passed on (up to 10,000) across runs, so a record is sent once. Needs n8n 1.64 or later."
},
{
"id": "fe097a76-92a9-487f-a7be-de9ec397d62b",
"name": "Keep main fields",
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
740,
100
],
"parameters": {
"jsCode": "// Keeps the Actor's main fields as flat, spreadsheet-safe values.\nconst columns = [\"title\",\"url\",\"depth\",\"wordCount\",\"httpStatusCode\",\"markdown\"];\n// A dotted name such as author.name reads the nested value.\nconst read = (obj, key) => (key in obj ? obj[key] : key.split('.').reduce((v, part) => (v && typeof v === 'object' ? v[part] : undefined), obj));\nreturn $input.all().map((item) => {\n const row = {};\n const present = columns.filter((key) => read(item.json, key) !== undefined);\n // A record shaped differently from the published schema keeps its own fields rather than arriving blank.\n for (const key of present.length >= 2 ? present : Object.keys(item.json)) {\n const value = read(item.json, key);\n row[key] = value === null || value === undefined ? '' : typeof value === 'object' ? JSON.stringify(value) : value;\n }\n return { json: row };\n});"
}
},
{
"id": "25dce6a6-883a-4faf-aef8-79fa0e375ab3",
"name": "Send to Notion",
"type": "n8n-nodes-base.notion",
"typeVersion": 2.2,
"position": [
980,
100
],
"parameters": {
"resource": "databasePage",
"databaseId": {
"__rl": true,
"value": "",
"mode": "list"
},
"title": "={{ [$json.title].filter((v) => v !== undefined && v !== null && v !== '').join(': ') || 'Untitled' }}",
"propertiesUi": {
"propertyValues": []
},
"blockUi": {
"blockValues": [
{
"type": "paragraph",
"textContent": "={{ [[\"url\", $json.url], [\"depth\", $json.depth], [\"wordCount\", $json.wordCount], [\"httpStatusCode\", $json.httpStatusCode]].filter(([, v]) => v !== undefined && v !== null && v !== '').map(([k, v]) => k + ': ' + (typeof v === 'object' ? JSON.stringify(v) : v)).join(' | ').slice(0, 1900) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(0, 1900) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(1900, 3800) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(3800, 5700) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(5700, 7600) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(7600, 9500) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(9500, 11400) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(11400, 13300) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(13300, 15200) || ' ' }}"
}
]
},
"options": {}
},
"notes": "Pick your Notion database. Each page gets the record's main fields and the first 15200 characters of markdown as its content; map extra database properties if you want to filter on them."
},
{
"id": "b679bc4e-e393-4e2b-a94d-347e12fc4383",
"name": "Setup",
"type": "n8n-nodes-base.stickyNote",
"typeVersion": 1,
"position": [
0,
-370
],
"parameters": {
"content": "## Domain-Scoped Site Crawler for Markdown Datasets\n\nSource: [vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets](https://apify.com/vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets) on Apify.\n\n1. In **Run Apify Actor**, create a Header Auth credential: name `Authorization`, value `Bearer YOUR_APIFY_TOKEN` with your token.\n2. Open **Send to Notion** and connect your Notion account.\n3. Edit the JSON body of **Run Apify Actor** to change the input.\n\nGenerated by recipeforagents.com. Validated, not executed: runs are billed to your Apify account.",
"height": 340,
"width": 460,
"color": 6
}
}
],
"connections": {
"Run by hand": {
"main": [
[
{
"node": "Run Apify Actor",
"type": "main",
"index": 0
}
]
]
},
"Every day": {
"main": [
[
{
"node": "Run Apify Actor",
"type": "main",
"index": 0
}
]
]
},
"Run Apify Actor": {
"main": [
[
{
"node": "Only records not sent before",
"type": "main",
"index": 0
}
]
]
},
"Only records not sent before": {
"main": [
[
{
"node": "Keep main fields",
"type": "main",
"index": 0
}
]
]
},
"Keep main fields": {
"main": [
[
{
"node": "Send to Notion",
"type": "main",
"index": 0
}
]
]
}
},
"pinData": {},
"active": false,
"settings": {
"executionOrder": "v1"
},
"meta": {
"templateCredsSetupCompleted": false
},
"tags": []
}Validated, not executed: 11 of 11 checks pass
- It has a nodes array and a connections object
- Every node has a name, type, typeVersion, position and parameters
- Node names are unique
- Every node type is a known n8n core node (or the Apify node)
- Every node uses a typeVersion that n8n ships
- Every connection points at an existing node
- At least one trigger starts the workflow
- Every working node is reachable from a trigger
- Apify calls use POST and a credential for the token
- Actor input matches the Actor's published input schema
- No token or credential value is embedded
A RAG corpus is only as good as its source pages: navigation menus, cookie banners and duplicate URLs pollute the index. A domain-scoped crawl that keeps to the start site, respects robots rules and outputs markdown with the page URL and title gives you clean documents and a citation for every chunk.
What you need
- The start URL of the documentation or help centre you want to index.
- n8n with an Apify Header Auth credential for the Website to markdown to Notion workflow, or any HTTP client for the Actor's API.
- A place for the documents: the workflow files them in Notion, and you can swap that node for a vector store or a database.
- A chunking step in your RAG pipeline; the crawler can also split markdown into chunks itself if you switch chunking on in its input.
Where it breaks
The change signal in the workflow is the markdown length, so an edit that keeps a page the same length is not picked up; clear the Remove Duplicates history once in a while for a full refresh. Notion also caps the text per block, so very long pages arrive cut; send those to a vector store or a file store instead when completeness matters.
Method
Import the Website to markdown to Notion workflow and add the Header Auth credential on Run Apify Actor with Bearer followed by your Apify token.
Set startUrls in the request body to the documentation home page and raise maxCrawlPages and maxCrawlDepth until the crawl covers the sections you need.
Keep outputFormat on markdown and leave respectRobotsTxtFile on, so pages the site asks crawlers to skip stay out of your corpus.
Point Send to Notion at your database, or replace it with a vector store node that takes the markdown and the URL as metadata for each chunk.
Activate the daily schedule; new pages and pages whose markdown length changed come through, and the rest are skipped by Only records not sent before.
Ready-made recipe
Website to markdown to Notion: an n8n workflow you can import, using Domain-Scoped Site Crawler for Markdown Datasets.
Troubleshooting
Why does the workflow or script stop with HTTP 408?
The synchronous endpoint waits at most 300 seconds. After that it answers 408 while the run keeps going in Apify. Lower the result limit, or start the run asynchronously, wait for it to finish and then read its dataset.
Source: Apify API: run Actor synchronously and get dataset items
What do 400, 401, 402 and 403 from the Apify API mean?
Read error.type in the response body. 400 is invalid Actor input. 401 (user-or-token-not-found) means a token reached Apify but it is wrong or revoked. 402 with x402-payment-required means no token reached Apify at all: the n8n credential or the Authorization: Bearer header is missing, and Apify is offering its pay-per-run x402 route for agents instead; any other 402 is an exhausted usage limit on the account that pays for the run. 403 is a token without permission for that Actor or storage.
Source: Apify API: authentication; Apify documentation: agentic payments with x402; Apify API: run Actor synchronously and get dataset items
n8n says Install this node to use it after import. Why?
The workflow names a node or node version your instance does not have, usually a community node. The workflows here use only built-in nodes at versions n8n has shipped since 1.80, so they import on n8n Cloud and older self-hosted instances without community nodes.
Why does the HTTP Request node time out before Apify answers?
Set the node's Timeout option to at least 300000 ms so n8n waits as long as Apify's synchronous endpoint. The workflows here already set 310000. A failure at exactly 30 seconds points to a proxy in front of n8n.
Why does starting an Actor through the API return run information instead of scraped results?
The Run Actor endpoint starts the Actor and immediately returns a Run object without waiting for completion. For results stored in the default dataset, pass the response's defaultDatasetId to Get dataset items; to wait and receive output directly, use the synchronous endpoint for dataset items or a key-value store record.
Source: Apify Run Actor API
Questions
Why markdown and not HTML?
Markdown keeps headings, lists and code blocks while dropping layout markup, which gives cleaner chunks and better retrieval than raw HTML.
How are changed pages handled?
The workflow keys each page on its URL plus the length of its markdown. A new URL or a length change files the page again; nothing is replaced in place.
Can I crawl only part of a site?
Yes. Start from the section's URL and keep the crawl depth low, so the crawler stays inside that part of the site.
How do I cite sources in answers?
Store the page URL and title with every chunk. The crawler returns both, so your RAG answers can link back to the exact page.
Sources
- Apify website crawler, checked 2026-11-09