Website to markdown to Qdrant with n8n

Every page of a documentation site is split into markdown chunks, embedded with OpenAI and stored in a Qdrant collection, each chunk carrying its page URL, title and heading path as metadata. A daily run sends a page again only when its URL is new or its markdown length changed, so a retrieval index stays current without embedding the whole site every morning. Validated 2026-11-12.

Put your site in startUrls in the request body. The crawler does the chunking itself: chunkMarkdown is on, with chunks of up to 1500 characters and 150 characters of overlap, and the first runs read up to 50 pages. You need three credentials: the Apify Header Auth credential (Authorization, Bearer and your token), a Qdrant credential with your cluster URL and API key, and an OpenAI credential for the embeddings. The vectors land in the website-docs collection; rename it in Send to Qdrant if you keep several sites apart.

Run on Apify

Built-in nodes only Imports on n8n Cloud and self-hosted n8n 1.80 or later, no community nodes. 11 of 11 checks pass; validated, not executed.

Source: Any public website
Actor: Domain-Scoped Site Crawler for Markdown Datasets
Destination: Qdrant
Store price: $0.99 per 1,000 results (pay per event)
Cost on the Free plan: about $0.42 a run (start $0.40 + about 25 results; the input sets no limit, so more results cost more), about $12.74 a month daily, plus platform usage
Data checked: 2026-11-12

Method

Get an Apify token

Sign in to Apify and copy a personal API token from Console: Settings, API and integrations. The workflow keeps it in an n8n credential, never in the file.

Import the workflow

In n8n choose Workflows, Import from file, or paste the JSON onto an empty canvas.

Run by hand
Every day
Run Apify Actor
Only records not sent before
One item per chunk
Send to Qdrant
Embeddings OpenAI
Chunk text and metadata
Keep chunks whole

n8n workflowwebsite-to-markdown-to-qdrant.n8n.json

Actor on Apify

Show the file (295 lines)

{
  "name": "Website to markdown to Qdrant",
  "nodes": [
    {
      "id": "bca266ba-2adb-46e0-a5f4-859910fe4e1b",
      "name": "Run by hand",
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "position": [
        0,
        0
      ],
      "parameters": {}
    },
    {
      "id": "332c3d6a-4c67-4182-af9d-48f7d2c660d3",
      "name": "Every day",
      "type": "n8n-nodes-base.scheduleTrigger",
      "typeVersion": 1.2,
      "position": [
        0,
        200
      ],
      "parameters": {
        "rule": {
          "interval": [
            {
              "triggerAtHour": 8
            }
          ]
        }
      },
      "notes": "Activate the workflow to run on this schedule."
    },
    {
      "id": "85bdf92e-fb4a-4917-ae77-028ae90d1cab",
      "name": "Run Apify Actor",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        260,
        100
      ],
      "parameters": {
        "method": "POST",
        "url": "https://api.apify.com/v2/actors/vexaloom-labs~domain-scoped-site-crawler-for-markdown-datasets/run-sync-get-dataset-items",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "{\n  \"startUrls\": [\n    {\n      \"url\": \"https://docs.apify.com/\"\n    }\n  ],\n  \"maxCrawlDepth\": 1,\n  \"maxCrawlPages\": 50,\n  \"useSitemaps\": false,\n  \"useLlmsTxt\": false,\n  \"respectRobotsTxtFile\": true,\n  \"keepUrlFragments\": false,\n  \"ignoreCanonicalUrl\": false,\n  \"outputFormat\": \"markdown\",\n  \"htmlTransformer\": \"auto\",\n  \"includeImages\": false,\n  \"chunkMarkdown\": true,\n  \"chunkMaxChars\": 1500,\n  \"chunkOverlapChars\": 150,\n  \"parsePdfText\": true,\n  \"takeScreenshots\": false,\n  \"storeSkippedUrls\": false\n}",
        "options": {
          "timeout": 310000
        }
      },
      "notes": "Actor: vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets. Select a Header Auth credential: name Authorization, value Bearer + your Apify token. Edit the JSON body to change the input.",
      "notesInFlow": false
    },
    {
      "id": "0a8471b2-bec6-4074-a9c2-1b91f6b1fa73",
      "name": "Only records not sent before",
      "type": "n8n-nodes-base.removeDuplicates",
      "typeVersion": 2,
      "position": [
        500,
        100
      ],
      "parameters": {
        "operation": "removeItemsSeenInPreviousExecutions",
        "logic": "removeItemsWithAlreadySeenKeyValues",
        "dedupeValue": "={{ $json.url }}#{{ String($json.markdown || '').length }}",
        "options": {
          "scope": "node",
          "historySize": 10000
        }
      },
      "notes": "Remembers each page version it has passed on (up to 10,000) across runs, so a record is sent once. Needs n8n 1.64 or later."
    },
    {
      "id": "4eb7a962-496b-4f1e-abe7-016f4b32b86f",
      "name": "Setup",
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        0,
        -370
      ],
      "parameters": {
        "content": "## Domain-Scoped Site Crawler for Markdown Datasets\n\nSource: [vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets](https://apify.com/vexaloom-labs/domain-scoped-site-crawler-for-markdown-datasets) on Apify.\n\n1. In **Run Apify Actor**, create a Header Auth credential: name `Authorization`, value `Bearer YOUR_APIFY_TOKEN` with your token.\n2. Click **Execute workflow**. Each record arrives as one n8n item.\n3. Edit the JSON body of **Run Apify Actor** to change the input.\n\nGenerated by recipeforagents.com. Validated, not executed: runs are billed to your Apify account.",
        "height": 340,
        "width": 460,
        "color": 6
      }
    },
    {
      "id": "564eee92-59b6-4ce6-ab99-77a78fef81e3",
      "name": "One item per chunk",
      "type": "n8n-nodes-base.splitOut",
      "typeVersion": 1,
      "position": [
        740,
        100
      ],
      "parameters": {
        "fieldToSplitOut": "chunks",
        "include": "selectedOtherFields",
        "fieldsToInclude": "url, title",
        "options": {}
      },
      "notes": "The crawler returns each page's chunks (chunkMarkdown on); each chunk becomes one item with the page's url and title."
    },
    {
      "id": "014428ca-1c07-47f2-a79a-3a9ff86eb9af",
      "name": "Send to Qdrant",
      "type": "@n8n/n8n-nodes-langchain.vectorStoreQdrant",
      "typeVersion": 1,
      "position": [
        1020,
        100
      ],
      "parameters": {
        "mode": "insert",
        "qdrantCollection": {
          "__rl": true,
          "value": "website-docs",
          "mode": "id"
        },
        "options": {}
      },
      "notes": "Create a Qdrant credential (cluster URL and API key) and name the collection. Each chunk is embedded and stored with its page url and title."
    },
    {
      "id": "41f5b0ea-086d-4da6-a8fd-deff03717cd7",
      "name": "Embeddings OpenAI",
      "type": "@n8n/n8n-nodes-langchain.embeddingsOpenAi",
      "typeVersion": 1.2,
      "position": [
        920,
        320
      ],
      "parameters": {
        "model": "text-embedding-3-small",
        "options": {}
      },
      "notes": "Needs an OpenAI credential. Keep the same model for loading and for querying the collection."
    },
    {
      "id": "c9fb3696-044a-422d-a369-19d096f590db",
      "name": "Chunk text and metadata",
      "type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
      "typeVersion": 1,
      "position": [
        1120,
        320
      ],
      "parameters": {
        "jsonMode": "expressionData",
        "jsonData": "={{ $json.chunks ? $json.chunks.text : $json.text }}",
        "options": {
          "metadata": {
            "metadataValues": [
              {
                "name": "url",
                "value": "={{ $json.url }}"
              },
              {
                "name": "title",
                "value": "={{ $json.title }}"
              },
              {
                "name": "headingPath",
                "value": "={{ [].concat(($json.chunks ? $json.chunks.headingPath : $json.headingPath) ?? []).join(' > ') }}"
              }
            ]
          }
        }
      }
    },
    {
      "id": "ce8b32fe-5e5a-4437-abf5-ac92471f20eb",
      "name": "Keep chunks whole",
      "type": "@n8n/n8n-nodes-langchain.textSplitterRecursiveCharacterTextSplitter",
      "typeVersion": 1,
      "position": [
        1220,
        520
      ],
      "parameters": {
        "chunkSize": 4000,
        "chunkOverlap": 0,
        "options": {}
      },
      "notes": "The crawler already chunked the page at chunkMaxChars; this splitter only breaks a chunk longer than 4,000 characters (a long code block or table)."
    }
  ],
  "connections": {
    "Run by hand": {
      "main": [
        [
          {
            "node": "Run Apify Actor",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Every day": {
      "main": [
        [
          {
            "node": "Run Apify Actor",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Run Apify Actor": {
      "main": [
        [
          {
            "node": "Only records not sent before",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Only records not sent before": {
      "main": [
        [
          {
            "node": "One item per chunk",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "One item per chunk": {
      "main": [
        [
          {
            "node": "Send to Qdrant",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Embeddings OpenAI": {
      "ai_embedding": [
        [
          {
            "node": "Send to Qdrant",
            "type": "ai_embedding",
            "index": 0
          }
        ]
      ]
    },
    "Chunk text and metadata": {
      "ai_document": [
        [
          {
            "node": "Send to Qdrant",
            "type": "ai_document",
            "index": 0
          }
        ]
      ]
    },
    "Keep chunks whole": {
      "ai_textSplitter": [
        [
          {
            "node": "Chunk text and metadata",
            "type": "ai_textSplitter",
            "index": 0
          }
        ]
      ]
    }
  },
  "pinData": {},
  "active": false,
  "settings": {
    "executionOrder": "v1"
  },
  "meta": {
    "templateCredsSetupCompleted": false
  },
  "tags": []
}

Validated, not executed: 11 of 11 checks pass

It has a nodes array and a connections object
Every node has a name, type, typeVersion, position and parameters
Node names are unique
Every node type is a known n8n core node (or the Apify node)
Every node uses a typeVersion that n8n ships
Every connection points at an existing node
At least one trigger starts the workflow
Every working node is reachable from a trigger
Apify calls use POST and a credential for the token
Actor input matches the Actor's published input schema
No token or credential value is embedded

Add the Apify credential

Open Run Apify Actor. Under Credential, create a Header Auth credential with the name Authorization and the value Bearer, a space and your token.

Set the input

The request body holds the Actor input: the Actor's Store prefill with startUrls, maxCrawlPages, outputFormat set for this recipe. Change startUrls and maxCrawlPages to your own.

{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxCrawlDepth": 1,
  "maxCrawlPages": 50,
  "useSitemaps": false,
  "useLlmsTxt": false,
  "respectRobotsTxtFile": true,
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "outputFormat": "markdown",
  "htmlTransformer": "auto",
  "includeImages": false,
  "chunkMarkdown": true,
  "chunkMaxChars": 1500,
  "chunkOverlapChars": 150,
  "parsePdfText": true,
  "takeScreenshots": false,
  "storeSkippedUrls": false
}

Every input field is documented on the Domain-Scoped Site Crawler for Markdown Datasets page.

Only new records go through

The Only records not sent before node (n8n's Remove Duplicates, set to remove items processed in previous executions) remembers each url }}#{{ String(markdown || '').length it has passed on, so a record reaches Qdrant once even when the next run returns it again. It needs n8n 1.64 or later; clear its history from the node to resend everything.
Connect Qdrant

Open Send to Qdrant and create a Qdrant credential with your cluster URL and API key; set the collection name. Open Embeddings OpenAI and select an OpenAI credential. Each chunk is stored with its page url, title and heading path as metadata, so answers can cite the page.
Run once, then turn it on

Click Execute workflow and check the first records in Qdrant. When they look right, activate the workflow so the daily trigger runs it at 08:00 in your instance's timezone.

What lands in Qdrant

The exact fields this workflow sends, in order.

Stored per chunk	From the Actor field
`chunks`
`chunks.text`
`text`
`url`	URL
`title`	Title
`chunks.headingPath`
`headingPath`

Where it breaks

When a page is sent again, its new chunks are added as new vectors and the old vectors for that page stay in the collection. If exact results matter, delete the page's points in Qdrant with a filter on the url metadata before the new version goes in.
The change signal is the length of the page's markdown, so an edit that keeps the length the same is not picked up.
OpenAI bills the embeddings separately from Apify; the cost line on this page covers only the Apify run.
Query the collection with the same embedding model that loaded it, text-embedding-3-small, or the search results will not match.
The workflow uses Apify's synchronous endpoint, which waits up to 300 seconds. Bigger runs answer HTTP 408: lower the limit or switch to an asynchronous run.
Each run is billed to your Apify account at the Store price ($0.99 per 1,000 results (pay per event)).

Bigger runs: the long-run workflow starts the run, waits for it and reads the dataset, so it never hits the 300-second limit.

If the workflow fails

Why does the workflow or script stop with HTTP 408?

The synchronous endpoint waits at most 300 seconds. After that it answers 408 while the run keeps going in Apify. Lower the result limit, or start the run asynchronously, wait for it to finish and then read its dataset.

Source: Apify API: run Actor synchronously and get dataset items. This fix on its own page

What do 400, 401, 402 and 403 from the Apify API mean?

Read error.type in the response body. 400 is invalid Actor input. 401 (user-or-token-not-found) means a token reached Apify but it is wrong or revoked. 402 with x402-payment-required means no token reached Apify at all: the n8n credential or the Authorization: Bearer header is missing, and Apify is offering its pay-per-run x402 route for agents instead; any other 402 is an exhausted usage limit on the account that pays for the run. 403 is a token without permission for that Actor or storage.

Source: Apify API: authentication; Apify documentation: agentic payments with x402; Apify API: run Actor synchronously and get dataset items. This fix on its own page

n8n says Install this node to use it after import. Why?

The workflow names a node or node version your instance does not have, usually a community node. The workflows here use only built-in nodes at versions n8n has shipped since 1.80, so they import on n8n Cloud and older self-hosted instances without community nodes.

Source: n8n issue: unrecognized node type after import. This fix on its own page

Why does the HTTP Request node time out before Apify answers?

Set the node's Timeout option to at least 300000 ms so n8n waits as long as Apify's synchronous endpoint. The workflows here already set 310000. A failure at exactly 30 seconds points to a proxy in front of n8n.

Source: n8n HTTP Request node documentation. This fix on its own page

Every fix in one place

Same data in an agent

Want Claude, Cursor or ChatGPT to call the Actor directly instead of on a schedule? The Domain-Scoped Site Crawler for Markdown Datasets page has MCP configs for eight clients and a SKILL.md.

Questions

Why does the crawler chunk the pages instead of n8n?

The crawler splits along headings, so each chunk keeps its heading path and stays readable. The splitter in n8n only breaks a chunk that is still longer than 4000 characters, such as a long code block or table.

How do I use the collection in an agent?

Add a Qdrant Vector Store node in retrieve mode, or a vector store tool on an n8n AI Agent, pointed at the same collection and the same embedding model. Each hit returns the chunk text with its page URL for citation.

Can I index a bigger site?

Yes. Raise maxCrawlPages and the crawl depth in the request body. For very large sites, use the start-and-poll workflow on the Actor page so the call does not run into the synchronous time limit.

More recipes

Website to markdown to Notion
Crawl a documentation site into markdown on a schedule and file each page in Notion, again whenever a page's…
Any public websiteNotion
Google Trends to Google Sheets
Save Google Trends interest over time and related queries for your terms in a Google Sheet to spot rising…
Google TrendsGoogle Sheets
Bluesky posts to Slack
Forward new Bluesky posts from a profile or search to Slack for brand and topic monitoring.
BlueskySlack
Eventbrite events to Google Sheets
List upcoming Eventbrite events for a city or category in a Google Sheet with date, venue, coordinates and…
EventbriteGoogle Sheets
Google Scholar papers to Notion
File Google Scholar search results in a Notion database with authors, year, citations and link, as a reading…
Google ScholarNotion
Crunchbase funding news to Slack
Post new Crunchbase News funding stories to Slack as a daily deal-flow feed.
Crunchbase NewsSlack

Website to markdown to Qdrant with n8n

Method

Get an Apify token

Import the workflow

n8n workflowwebsite-to-markdown-to-qdrant.n8n.json

Add the Apify credential

Set the input

Only new records go through

Connect Qdrant

Run once, then turn it on

What lands in Qdrant

Where it breaks

If the workflow fails

Same data in an agent

Questions

More recipes

Website to markdown to Notion

Google Trends to Google Sheets

Bluesky posts to Slack

Eventbrite events to Google Sheets

Google Scholar papers to Notion

Crunchbase funding news to Slack