Generic Spider Templates

Most crawls fall into one of two patterns: "follow links matching this regex" or "crawl every URL listed in the site's sitemap". Scrapling ships templates for both so you don't have to hand-write the same parse() boilerplate every time.

Both templates build on LinkExtractor, which pulls URLs out of a Response (or filters a single URL via matches()). SitemapSpider additionally parses sitemap.xml / sitemap_index.xml bodies internally (gzip-compressed or not).

You can use LinkExtractor directly inside any plain Spider.parse(). The templates just save you the wiring.

CrawlSpider

CrawlSpider follows links automatically based on declarative rules.

from scrapling.spiders import CrawlSpider, CrawlRule, LinkExtractor

class BlogCrawler(CrawlSpider):
    name = "blog"
    start_urls = ["https://example.com"]

    def rules(self):
        return [
            CrawlRule(LinkExtractor(allow=r"/posts/"), callback=self.parse_post),
            CrawlRule(LinkExtractor(allow=r"/page/\d+/")),  # follow pagination, no callback
        ]

    async def parse_post(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "url": response.url,
        }

result = BlogCrawler().start()

A CrawlRule pairs a LinkExtractor with an optional callback (a bound method on the spider), an optional priority override for the dispatched Request, and an optional process_request (a bound method that mutates each Request before it's yielded). The default parse() runs every rule against every response and yields a Request per matched URL.

If a rule has no callback, the matched URLs fall through to the spider's default parse() (or stay uncallback'd if you didn't override it). This is convenient for pagination: extract the next-page links to keep the crawl going, but don't need a separate handler.

Combining rules with custom logic

Override parse() and call super().parse(response) to get the rule behavior plus your own yields:

class MySpider(CrawlSpider):
    def rules(self):
        return [CrawlRule(LinkExtractor(allow=r"/posts/"), callback=self.parse_post)]

    async def parse(self, response):
        yield {"page_url": response.url}
        async for req in super().parse(response):
            yield req

Mutating Requests with process_request

def add_priority(self, request, response):
    request.priority = 10
    return request

def rules(self):
    return [CrawlRule(
        LinkExtractor(allow=r"/posts/"),
        callback=self.parse_post,
        process_request=self.add_priority,
    )]

SitemapSpider

SitemapSpider seeds a crawl from sitemap.xml URLs. It uses the same rules() API as CrawlSpider, so the mental model is shared.

from scrapling.spiders import SitemapSpider, CrawlRule, LinkExtractor

class MySitemap(SitemapSpider):
    name = "sm"
    sitemap_urls = ["https://example.com/sitemap.xml"]

    def rules(self):
        return [
            CrawlRule(LinkExtractor(allow=r"/posts/"), callback=self.parse_post),
            CrawlRule(LinkExtractor(allow=r"/products/"), callback=self.parse_product),
        ]

    async def parse_post(self, response):
        yield {"title": response.css("h1::text").get()}

    async def parse_product(self, response):
        yield {"sku": response.css(".sku::text").get()}

result = MySitemap().start()

How URLs are dispatched

For each URL in the sitemap, SitemapSpider checks every rule's LinkExtractor.matches(url) in order. The first matching rule wins, and a Request is yielded with that rule's callback. If no rule matches and rules() is non-empty, the URL is dropped (matches Scrapy's behavior). If rules() returns an empty list, every URL is routed to the spider's parse() method, which raises NotImplementedError by default - override it to handle them.

Sitemap indexes

When SitemapSpider encounters a <sitemapindex> (a sitemap of sitemaps), it descends into each child sitemap automatically. To filter which child sitemaps to descend into, set sitemap_follow to a LinkExtractor:

class MySitemap(SitemapSpider):
    name = "sm"
    sitemap_urls = ["https://example.com/sitemap.xml"]
    sitemap_follow = LinkExtractor(allow=r"/posts-sitemap-\d+\.xml")  # only post sitemaps

Robots.txt support

Put a robots.txt URL directly in sitemap_urls and SitemapSpider will detect it, extract every Sitemap: directive (via protego), and follow each one:

class MySitemap(SitemapSpider):
    name = "sm"
    sitemap_urls = ["https://example.com/robots.txt"]  # Sitemap: directives discovered automatically

Alternate-language URLs

Set sitemap_alternate_links = True to also dispatch <xhtml:link rel="alternate" hreflang="..."> URLs through your rules.

XMLFeedSpider

XMLFeedSpider iterates over the nodes of an XML feed (RSS, Atom, product feeds, etc.). Set itertag to the node name you want (default: "item") and override parse_node(), which is called once per matching node:

from scrapling.spiders import XMLFeedSpider

class RSSSpider(XMLFeedSpider):
    name = "rss"
    start_urls = ["https://example.com/feed.xml"]
    itertag = "item"

    async def parse_node(self, response, node):
        yield {
            "title": node.findtext("title"),
            "link": node.findtext("link"),
            "date": node.findtext("pubDate"),
        }

result = RSSSpider().start()

Like the other callbacks, parse_node() can also yield Request objects (for example, response.follow(node.findtext("link"), callback=self.parse_post)) to crawl into the pages the feed points to.

How nodes are matched and parsed

Each node passed to parse_node() is an lxml element with all namespaces stripped, so node.findtext("title"), node.find("thumbnail").get("url"), and case-sensitive node.xpath(...) work on any feed without namespace maps. A plain itertag like "entry" matches nodes by name regardless of their namespace, which is what you want for Atom and most namespaced feeds. To match a node in one specific namespace, use a prefixed itertag and define the prefix in namespaces:

class ThumbnailSpider(XMLFeedSpider):
    name = "thumbs"
    start_urls = ["https://example.com/feed.xml"]
    itertag = "media:thumbnail"
    namespaces = (("media", "http://search.yahoo.com/mrss/"),)

    async def parse_node(self, response, node):
        yield {"thumbnail": node.get("url")}

Gzipped feeds (.xml.gz or served with a gzip content-type) are decompressed automatically with the same protections the sitemap spider uses, and malformed XML logs a warning instead of crashing the crawl.

CSVFeedSpider

CSVFeedSpider iterates over the rows of a CSV feed. Override parse_row(), which receives each row as a dictionary keyed by the column names:

from scrapling.spiders import CSVFeedSpider

class PriceSpider(CSVFeedSpider):
    name = "prices"
    start_urls = ["https://example.com/products.csv"]

    async def parse_row(self, response, row):
        yield {"product": row["title"], "price": float(row["price"])}

result = PriceSpider().start()

By default, the first row of the feed is used as the header. If the feed has no header row, set headers to the column names yourself, and use delimiter/quotechar for feeds that don't follow the standard comma format:

class PriceSpider(CSVFeedSpider):
    name = "prices"
    start_urls = ["https://example.com/products.csv"]
    headers = ["title", "price", "url"]
    delimiter = ";"

Gzipped feeds are decompressed automatically here as well, as shown above for XMLFeedSpider.

Using LinkExtractor directly

You don't have to use the templates. LinkExtractor works inside any plain Spider:

from scrapling.spiders import Spider, LinkExtractor

class CustomSpider(Spider):
    name = "custom"
    start_urls = ["https://example.com"]

    def __init__(self):
        super().__init__()
        self._links = LinkExtractor(allow=r"/posts/", deny_domains="ads.example.com")

    async def parse(self, response):
        for url in self._links.extract(response):
            yield response.follow(url, callback=self.parse_post)

    async def parse_post(self, response):
        yield {"title": response.css("h1::text").get()}

LinkExtractor reference

LinkExtractor.extract(response) returns a list[str] of absolute, filtered, deduped URLs.

LinkExtractor.matches(url) returns a bool - the URL-only filter (allow/deny/domain/extension), used by SitemapSpider to dispatch URLs without a Response.