How to scrape a website with n8n
You get an n8n workflow that crawls a website on a schedule and files each page in Notion with its URL, title, word count and the page text as markdown. It uses six built-in nodes: a manual and a daily trigger, an HTTP Request that runs a hosted crawler Actor, a Remove Duplicates step, a field picker and the Notion node.
Checked 2026-11-12
Do it now
The workflow from Website to markdown to Notion, ready to import into n8n: Domain-Scoped Site Crawler for Markdown Datasets on Apify, then Notion.
n8n workflowwebsite-to-markdown-to-notion.n8n.json
{
"name": "Website to markdown to Notion",
"nodes": [
{
"id": "7ae13a1e-9c39-494d-aa13-a06c30f44d23",
"name": "Run by hand",
"type": "n8n-nodes-base.manualTrigger",
"typeVersion": 1,
"position": [
0,
0
],
"parameters": {}
},
{
"id": "db9f5cce-ebde-42a7-a959-ea7ee228937b",
"name": "Every day",
"type": "n8n-nodes-base.scheduleTrigger",
"typeVersion": 1.2,
"position": [
0,
200
],
"parameters": {
"rule": {
"interval": [
{
"triggerAtHour": 8
}
]
}
},
"notes": "Activate the workflow to run on this schedule."
},
{
"id": "69301f72-d9d9-4438-a594-a2bd108f0457",
"name": "Run Apify Actor",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
260,
100
],
"parameters": {
"method": "POST",
"url": "https://api.apify.com/v2/actors/vexaloom-labs~domain-scoped-site-crawler-for-markdown-datasets/run-sync-get-dataset-items",
"authentication": "genericCredentialType",
"genericAuthType": "httpHeaderAuth",
"sendBody": true,
"specifyBody": "json",
"jsonBody": "{\n \"startUrls\": [\n {\n \"url\": \"https://docs.apify.com\"\n }\n ],\n \"maxCrawlDepth\": 1,\n \"maxCrawlPages\": 3,\n \"useSitemaps\": false,\n \"useLlmsTxt\": false,\n \"respectRobotsTxtFile\": true,\n \"keepUrlFragments\": false,\n \"ignoreCanonicalUrl\": false,\n \"outputFormat\": \"markdown\",\n \"htmlTransformer\": \"auto\",\n \"includeImages\": false,\n \"chunkMarkdown\": false,\n \"chunkMaxChars\": 2000,\n \"chunkOverlapChars\": 200,\n \"parsePdfText\": true,\n \"takeScreenshots\": false,\n \"storeSkippedUrls\": false\n}",
"options": {
"timeout": 310000
}
},
"notes": "Actor: vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets. Select a Header Auth credential: name Authorization, value Bearer + your Apify token. Edit the JSON body to change the input.",
"notesInFlow": false
},
{
"id": "e0c2145e-33df-4515-af0d-8a64f9b4f913",
"name": "Only records not sent before",
"type": "n8n-nodes-base.removeDuplicates",
"typeVersion": 2,
"position": [
500,
100
],
"parameters": {
"operation": "removeItemsSeenInPreviousExecutions",
"logic": "removeItemsWithAlreadySeenKeyValues",
"dedupeValue": "={{ $json.url }}#{{ String($json.markdown || '').length }}",
"options": {
"scope": "node",
"historySize": 10000
}
},
"notes": "Remembers each record it has passed on (up to 10,000) across runs, so a record is sent once. Needs n8n 1.64 or later."
},
{
"id": "fe097a76-92a9-487f-a7be-de9ec397d62b",
"name": "Keep main fields",
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
740,
100
],
"parameters": {
"jsCode": "// Keeps the Actor's main fields as flat, spreadsheet-safe values.\nconst columns = [\"title\",\"url\",\"depth\",\"wordCount\",\"httpStatusCode\",\"markdown\"];\n// A dotted name such as author.name reads the nested value.\nconst read = (obj, key) => (key in obj ? obj[key] : key.split('.').reduce((v, part) => (v && typeof v === 'object' ? v[part] : undefined), obj));\nreturn $input.all().map((item) => {\n const row = {};\n const present = columns.filter((key) => read(item.json, key) !== undefined);\n // A record shaped differently from the published schema keeps its own fields rather than arriving blank.\n for (const key of present.length >= 2 ? present : Object.keys(item.json)) {\n const value = read(item.json, key);\n row[key] = value === null || value === undefined ? '' : typeof value === 'object' ? JSON.stringify(value) : value;\n }\n return { json: row };\n});"
}
},
{
"id": "25dce6a6-883a-4faf-aef8-79fa0e375ab3",
"name": "Send to Notion",
"type": "n8n-nodes-base.notion",
"typeVersion": 2.2,
"position": [
980,
100
],
"parameters": {
"resource": "databasePage",
"databaseId": {
"__rl": true,
"value": "",
"mode": "list"
},
"title": "={{ [$json.title].filter((v) => v !== undefined && v !== null && v !== '').join(': ') || 'Untitled' }}",
"propertiesUi": {
"propertyValues": []
},
"blockUi": {
"blockValues": [
{
"type": "paragraph",
"textContent": "={{ [[\"url\", $json.url], [\"depth\", $json.depth], [\"wordCount\", $json.wordCount], [\"httpStatusCode\", $json.httpStatusCode]].filter(([, v]) => v !== undefined && v !== null && v !== '').map(([k, v]) => k + ': ' + (typeof v === 'object' ? JSON.stringify(v) : v)).join(' | ').slice(0, 1900) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(0, 1900) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(1900, 3800) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(3800, 5700) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(5700, 7600) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(7600, 9500) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(9500, 11400) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(11400, 13300) || ' ' }}"
},
{
"type": "paragraph",
"textContent": "={{ String($json.markdown ?? '').slice(13300, 15200) || ' ' }}"
}
]
},
"options": {}
},
"notes": "Pick your Notion database. Each page gets the record's main fields and the first 15200 characters of markdown as its content; map extra database properties if you want to filter on them."
},
{
"id": "b679bc4e-e393-4e2b-a94d-347e12fc4383",
"name": "Setup",
"type": "n8n-nodes-base.stickyNote",
"typeVersion": 1,
"position": [
0,
-370
],
"parameters": {
"content": "## Domain-Scoped Site Crawler for Markdown Datasets\n\nSource: [vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets](https://apify.com/vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets) on Apify.\n\n1. In **Run Apify Actor**, create a Header Auth credential: name `Authorization`, value `Bearer YOUR_APIFY_TOKEN` with your token.\n2. Open **Send to Notion** and connect your Notion account.\n3. Edit the JSON body of **Run Apify Actor** to change the input.\n\nGenerated by recipeforagents.com. Validated, not executed: runs are billed to your Apify account.",
"height": 340,
"width": 460,
"color": 6
}
}
],
"connections": {
"Run by hand": {
"main": [
[
{
"node": "Run Apify Actor",
"type": "main",
"index": 0
}
]
]
},
"Every day": {
"main": [
[
{
"node": "Run Apify Actor",
"type": "main",
"index": 0
}
]
]
},
"Run Apify Actor": {
"main": [
[
{
"node": "Only records not sent before",
"type": "main",
"index": 0
}
]
]
},
"Only records not sent before": {
"main": [
[
{
"node": "Keep main fields",
"type": "main",
"index": 0
}
]
]
},
"Keep main fields": {
"main": [
[
{
"node": "Send to Notion",
"type": "main",
"index": 0
}
]
]
}
},
"pinData": {},
"active": false,
"settings": {
"executionOrder": "v1"
},
"meta": {
"templateCredsSetupCompleted": false
},
"tags": []
}Validated, not executed: 11 of 11 checks pass
- It has a nodes array and a connections object
- Every node has a name, type, typeVersion, position and parameters
- Node names are unique
- Every node type is a known n8n core node (or the Apify node)
- Every node uses a typeVersion that n8n ships
- Every connection points at an existing node
- At least one trigger starts the workflow
- Every working node is reachable from a trigger
- Apify calls use POST and a credential for the token
- Actor input matches the Actor's published input schema
- No token or credential value is embedded
n8n can fetch a page with its own HTTP Request node, but crawling a site, following links and turning HTML into clean text is where hand-built flows break. Here the crawl runs as a hosted Actor called through Apify's synchronous run endpoint, and n8n only handles the schedule, the token and the destination.
What you need
- n8n Cloud or a self-hosted n8n; the Remove Duplicates step needs n8n 1.64 or later.
- An Apify API token, stored in an n8n Header Auth credential: name Authorization, value Bearer followed by the token.
- A Notion database shared with your n8n Notion integration.
- The start URL of the site you want, for example a documentation home page.
Where it breaks
Most first runs fail on the credential: a token pasted into the JSON body, or a Header Auth value without the word Bearer, gets an unauthorized or payment-required answer from Apify. The second failure is a crawl that takes longer than the synchronous call allows; keep the page limit small at first, and switch to the start-and-poll workflow on the Actor page for large sites.
Method
Download the Website to markdown to Notion workflow below and import it in n8n with Workflows, Import from file.
Open Run Apify Actor and create a Header Auth credential with the name Authorization and the value Bearer followed by your Apify API token.
Edit the JSON body of Run Apify Actor: put your site in startUrls and raise maxCrawlPages and maxCrawlDepth once a small run looks right.
Open Send to Notion, connect your Notion account and pick the database, then run the workflow by hand and check the first pages.
Activate the workflow so the Every day trigger runs it at 08:00; Only records not sent before keeps pages you already filed out of Notion.
Ready-made recipe
Website to markdown to Notion: an n8n workflow you can import, using Domain-Scoped Site Crawler for Markdown Datasets.
Troubleshooting
Why does the workflow or script stop with HTTP 408?
The synchronous endpoint waits at most 300 seconds. After that it answers 408 while the run keeps going in Apify. Lower the result limit, or start the run asynchronously, wait for it to finish and then read its dataset.
Source: Apify API: run Actor synchronously and get dataset items
What do 400, 401, 402 and 403 from the Apify API mean?
Read error.type in the response body. 400 is invalid Actor input. 401 (user-or-token-not-found) means a token reached Apify but it is wrong or revoked. 402 with x402-payment-required means no token reached Apify at all: the n8n credential or the Authorization: Bearer header is missing, and Apify is offering its pay-per-run x402 route for agents instead; any other 402 is an exhausted usage limit on the account that pays for the run. 403 is a token without permission for that Actor or storage.
Source: Apify API: authentication; Apify documentation: agentic payments with x402; Apify API: run Actor synchronously and get dataset items
n8n says Install this node to use it after import. Why?
The workflow names a node or node version your instance does not have, usually a community node. The workflows here use only built-in nodes at versions n8n has shipped since 1.80, so they import on n8n Cloud and older self-hosted instances without community nodes.
Why does the HTTP Request node time out before Apify answers?
Set the node's Timeout option to at least 300000 ms so n8n waits as long as Apify's synchronous endpoint. The workflows here already set 310000. A failure at exactly 30 seconds points to a proxy in front of n8n.
Why does starting an Actor through the API return run information instead of scraped results?
The Run Actor endpoint starts the Actor and immediately returns a Run object without waiting for completion. For results stored in the default dataset, pass the response's defaultDatasetId to Get dataset items; to wait and receive output directly, use the synchronous endpoint for dataset items or a key-value store record.
Source: Apify Run Actor API
Questions
Why not use n8n's own HTTP Request to scrape?
It fetches one page at a time and leaves link following, rendering and HTML cleanup to you. The hosted crawler does those steps and returns clean markdown per page.
Where does the token go?
In a Header Auth credential on the Run Apify Actor node, never in the JSON body, so the workflow file can be shared without a secret.
How do I avoid filing the same page twice?
The Remove Duplicates step remembers what it has passed on. A page comes back only when its URL is new or its markdown length changes.
Can the pages go somewhere other than Notion?
Yes. Swap the Notion node for Google Sheets, a database node or a vector store node; the earlier nodes stay the same.
Sources
- n8n HTTP Request node, checked 2026-11-09
- n8n Schedule Trigger, checked 2026-11-09
- Apify API, checked 2026-11-09