Ctrlk
HomePricingIntegrations
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Web Scraper

Extract text, HTML, Markdown, audio, or PDF content from a webpage and pass it into the rest of your workflow.

Web Scraper

Overview & The "Smart" Mechanism

The Web Scraper pulls content from one or more web pages and turns it into a format your workflow can use. Think of it like a research assistant that visits a page, copies what matters, and hands it back as clean text, HTML, Markdown, audio, or a PDF link.

It is especially useful when you want fresh website content inside Diaflow without copying and pasting by hand. You can feed the result into AI, reports, alerts, summaries, or downstream actions.

Smart features

  • Multiple output formats : extract content as plain text, HTML, Markdown, audio, or PDF depending on what the next step needs.

  • Automatic fallback when a page is hard to read : the node first tries a fast page-reading method, then switches to browser-style rendering if needed.

  • Better handling for dynamic websites : if a page relies on scripts or special rendering, the node can still try to capture the visible page content.

  • Custom headers support : add login or access headers when a page needs them.

  • Multi-page scraping for text modes : collect content from several URLs in one step when working with text-based output.

Common Use Cases

  • Pull article content from industry websites and send it to an AI node for summarization.

  • Capture product pages, help center articles, or policy pages and save the content into reports or knowledge workflows.

  • Extract audio from a public video or download a public PDF for transcription, review, or document processing.

How to Configure

  • URL(s) : Enter one website address or several.

    • Use one URL when you need a single page.

    • Use multiple URLs when you want to combine text content from several pages.

    • You can also pull the URL from an earlier step using @.

  • Content output format : Select the result type you want:

    • Plaintext for summaries, search, AI prompts, and simple analysis.

    • HTML if you need the page structure.

    • Markdown if you want cleaner formatting for notes, docs, or AI inputs.

    • Audio if you want an audio file link from a supported media page.

    • PDF if you want a downloadable PDF link.

  • Custom Headers : Add headers only if the website requires access details, such as authorization or cookies.

  • Enable Cache : Leave this on if the same page is used often and does not change frequently.

  • Caching time : Set how long Diaflow should reuse the saved result before checking the page again.

  • Run the workflow and inspect the returned content or file link before sending it to the next node.

Before & After Example

Before

After

Important Warnings & Best Practices

  • Audio and PDF modes only use the first URL you provide. Extra URLs in the list are ignored.

  • When you scrape multiple URLs in text mode, the node combines everything into one long result. If you need page-by-page separation, scrape them one at a time.

  • Very large pages can slow down the workflow or return more content than you need. Start with focused pages instead of full sites.

  • Some slow or unresponsive websites can delay the workflow for longer than expected.

  • If you try to extract plain text from a PDF link, the result may come back empty. Use PDF mode for PDF files.

  • Use custom headers carefully. Only add them when the site requires access details.

Need help?

Last updated