docs/topics/link-extractors.rst
.. _topics-link-extractors:
A link extractor is an object that extracts links from responses.
The __init__ method of
:class:~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor takes settings that
determine which links may be extracted. :class:LxmlLinkExtractor.extract_links <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links> returns a
list of matching :class:~scrapy.link.Link objects from a
:class:~scrapy.http.Response object.
Link extractors are used in :class:~scrapy.spiders.CrawlSpider spiders
through a set of :class:~scrapy.spiders.Rule objects.
You can also use link extractors in regular spiders. For example, you can instantiate
:class:LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor> into a class
variable in your spider, and use it from your spider callbacks:
.. code-block:: python
def parse(self, response):
for link in self.link_extractor.extract_links(response):
yield Request(link.url, callback=self.parse)
.. _topics-link-extractors-ref:
.. module:: scrapy.linkextractors :synopsis: Link extractors classes
The link extractor class is
:class:scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor. For convenience it
can also be imported as scrapy.linkextractors.LinkExtractor::
from scrapy.linkextractors import LinkExtractor
.. module:: scrapy.linkextractors.lxmlhtml :synopsis: lxml's HTMLParser-based link extractors
.. autoclass:: LxmlLinkExtractor
.. automethod:: extract_links
.. module:: scrapy.link :synopsis: Link from link extractors
.. autoclass:: Link