Crawl a website into markdown for RAG with Firecrawl

What you get

A Python script that crawls a site with Firecrawl and saves each page as a markdown file you can chunk and embed.

Prerequisites

Python 3.9+
A Firecrawl API key (fc-...) from https://www.firecrawl.dev (single scrapes work without a key at low rate limits)

Steps

Install the SDK:

Shell

pip install firecrawl-py

Scrape one page to check the output:

Python

from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key='fc-YOUR_API_KEY')

doc = firecrawl.scrape("https://firecrawl.dev", formats=["markdown", "html"])
print(doc.markdown)  # Access markdown content
print(doc.metadata)  # Access metadata like title, description, statusCode

Crawl the whole site and write one file per page (crawl.py):

Python

from pathlib import Path
from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key='fc-YOUR_API_KEY')

response = firecrawl.crawl('https://example.com',
    limit=100,
    scrape_options={
        'formats': ['markdown'],
    }
)

out = Path("pages")
out.mkdir(exist_ok=True)
for i, page in enumerate(response.data):
    (out / f"page-{i}.md").write_text(page.markdown or "", encoding="utf-8")
print(f"Saved {len(response.data)} pages")

Feed the files in pages/ into your chunker and embedding step (for example the Pinecone recipe).

Test it

Shell

python crawl.py

You should see Saved N pages and a pages/ folder of markdown files. Each page scrape costs 1 credit.