docs/topics/download-handlers.rst
.. _topics-download-handlers:
Download handlers are Scrapy :ref:components <topics-components> used to
download :ref:requests <topics-request-response> and produce responses from
them.
The :setting:DOWNLOAD_HANDLERS_BASE and :setting:DOWNLOAD_HANDLERS settings
tell Scrapy which handler is responsible for a given URL scheme. Their values
are merged into a mapping from scheme names to handler classes. When Scrapy
initializes it creates instances of all configured download handlers (except
for :ref:lazy ones <lazy-download-handlers>) and stores them in a similar
mapping. When Scrapy needs to download a request it extracts the scheme from
its URL, finds the handler for this scheme, passes the request to it and gets a
response from it. If there is no handler for the scheme, the request is not
downloaded and a :exc:~scrapy.exceptions.NotSupported exception is raised.
The :setting:DOWNLOAD_HANDLERS_BASE setting contains the default mapping of
handlers. You can use the :setting:DOWNLOAD_HANDLERS setting to add handlers
for additional schemes and to replace or disable default ones:
.. code-block:: python
DOWNLOAD_HANDLERS = {
# disable support for ftp:// requests
"ftp": None,
# replace the default one for http://
"http": "my.download_handlers.HttpHandler",
# http:// and https:// are different schemes,
# even though they may use the same handler
"https": "my.download_handlers.HttpHandler",
# support for any custom scheme can be added
"sftp": "my.download_handlers.SftpHandler",
}
.. seealso:: :ref:security-unencrypted-protocols and
:ref:security-local-resources, for the security implications of the
default http, ftp, file and data handlers.
While Scrapy provides a default handler for http and https schemes,
users may want to use a different handler, provided by Scrapy or by some
3rd-party package. There are several considerations to keep in mind related to
this.
First of all, as http and https are separate schemes, they need
separate entries in the :setting:DOWNLOAD_HANDLERS setting, even though it's
likely that the same handler class will be used for both schemes.
Additionally, some of the Scrapy settings, like :setting:DOWNLOAD_MAXSIZE,
are honored by the default HTTP(S) handler but not necessarily by alternative
ones. The same may apply to other Scrapy features, e.g. the
:signal:bytes_received and :signal:headers_received signals.
.. _lazy-download-handlers:
A download handler can be marked as "lazy" by setting its lazy class
attribute to True. Such handlers are only instantiated when they need to
download their first request. This may be useful when the instantiation is slow
or requires dependencies that are not always available, and the handler is not
needed on every spider run. For example, :class:the built-in S3 handler <.S3DownloadHandler> is lazy.
A download handler is a :ref:component <topics-components> that defines
the following API:
.. class:: SampleDownloadHandler
.. attribute:: lazy
:type: bool
If ``False``, the handler will be instantiated when Scrapy is
initialized.
If ``True``, the handler will only be instantiated when the first
request handled by it needs to be downloaded.
.. method:: download_request(request: Request) -> Response:
:async:
Download the given request and return a response.
.. method:: close() -> None
:async:
Clean up any resources used by the handler.
An optional base class for custom handlers is provided:
.. autoclass:: scrapy.core.downloader.handlers.base.BaseDownloadHandler :members: :undoc-members: :member-order: bysource
.. _download-handlers-exceptions:
.. versionadded:: 2.15.0
The built-in download handlers raise Scrapy-specific exceptions instead of implementation-specific ones, so that code that handles these exceptions can be written in a generic way. We recommend custom download handlers to also use these exceptions.
.. autoexception:: scrapy.exceptions.CannotResolveHostError
.. autoexception:: scrapy.exceptions.DownloadCancelledError
.. autoexception:: scrapy.exceptions.DownloadConnectionRefusedError
.. autoexception:: scrapy.exceptions.DownloadFailedError
.. autoexception:: scrapy.exceptions.DownloadTimeoutError
.. autoexception:: scrapy.exceptions.ResponseDataLossError
.. autoexception:: scrapy.exceptions.UnsupportedURLSchemeError
.. _download-handlers-ref:
Scrapy ships several handlers for HTTP and HTTPS requests. While all of them support basic features, they may differ in support of specific Scrapy features and settings and HTTP protocol features. See the documentation of specific handlers and specific settings for more information. Additionally, as the underlying HTTP client implementations differ between handlers, the behavior of specific websites may be different when doing the same Scrapy requests but using different handlers.
Here is a comparison of some features of the built-in HTTP handlers, see the individual handler docs for more differences:
================== ================= ===================== ====================
Feature H2DownloadHandler HTTP11DownloadHandler HttpxDownloadHandler
================== ================= ===================== ====================
Requires asyncio No No Yes
Requires a reactor Yes Yes No
HTTP/1.1 No Yes Yes
HTTP/2 Yes No Yes
TLS implementation cryptography cryptography Stdlib ssl
HTTP proxies No Yes Yes
SOCKS proxies No No Yes
================== ================= ===================== ====================
You can find additional HTTP download handlers in the scrapy-download-handlers-incubator_ package. This package is made by the Scrapy developers and contains experimental handlers that may be included in some later Scrapy version but can already be used. Please refer to the documentation of this package for more information.
.. _scrapy-download-handlers-incubator: https://github.com/scrapy-plugins/scrapy-download-handlers-incubator
.. _twisted-http2-handler:
.. autoclass:: scrapy.core.downloader.handlers.http2.H2DownloadHandler
| Supported scheme: https.
| :ref:Lazy <lazy-download-handlers>: yes.
| :ref:Requires asyncio support <using-asyncio>: no.
| :ref:Requires a Twisted reactor <asyncio-without-reactor>: yes.
This handler supports https://host/path URLs and uses the HTTP/2 protocol
for them.
It's implemented using :mod:twisted.web.client and the h2 library.
For this handler to work you need to install the Twisted[http2] extra
dependency.
If you want to use this handler you need to replace the default one for the
https scheme:
.. code-block:: python
DOWNLOAD_HANDLERS = {
"https": "scrapy.core.downloader.handlers.http2.H2DownloadHandler",
}
Features and limitations ^^^^^^^^^^^^^^^^^^^^^^^^
.. warning::
This handler is experimental, and not yet recommended for production
environments. Future Scrapy versions may introduce related changes without
a deprecation period or warning.
=========================== ================================================
HTTP proxies No (not implemented)
SOCKS proxies No (not supported by the library)
HTTP/2 Yes
response.certificate :class:twisted.internet.ssl.Certificate object
Per-request bindaddress Yes
TLS implementation pyOpenSSL/cryptography
=========================== ================================================
Other limitations:
No support for HTTP/1.1.
IPv6 support requires setting :setting:TWISTED_DNS_RESOLVER
to scrapy.resolver.CachingHostnameResolver.
No support for the :signal:bytes_received and :signal:headers_received
signals.
Known limitations of the HTTP/2 support:
No support for HTTP/2 Cleartext (h2c), since no major browser supports
HTTP/2 unencrypted (refer http2 faq_).
No setting to specify a maximum frame size_ larger than the default
value, 16384. Connections to servers that send a larger frame will fail.
No support for server pushes_, which are ignored.
.. _frame size: https://datatracker.ietf.org/doc/html/rfc7540#section-4.2 .. _http2 faq: https://http2.github.io/faq/#does-http2-require-encryption .. _server pushes: https://datatracker.ietf.org/doc/html/rfc7540#section-8.2
.. autoclass:: scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler
| Supported schemes: http, https.
| :ref:Lazy <lazy-download-handlers>: no.
| :ref:Requires asyncio support <using-asyncio>: no.
| :ref:Requires a Twisted reactor <asyncio-without-reactor>: yes.
This handler supports http://host/path and https://host/path URLs and
uses the HTTP/1.1 protocol for them.
It's implemented using :mod:twisted.web.client.
Features and limitations ^^^^^^^^^^^^^^^^^^^^^^^^
=========================== ================================================
HTTP proxies Yes
SOCKS proxies No (not supported by the library)
HTTP/2 No (implemented as a separate handler)
response.certificate :class:twisted.internet.ssl.Certificate object
Per-request bindaddress Yes
TLS implementation pyOpenSSL/cryptography
=========================== ================================================
Other limitations:
IPv6 support requires setting :setting:TWISTED_DNS_RESOLVER
to scrapy.resolver.CachingHostnameResolver.
HTTPS proxies to HTTPS destinations are not supported.
.. versionadded:: 2.15.0
.. autoclass:: scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler
| Supported schemes: http, https.
| :ref:Lazy <lazy-download-handlers>: no.
| :ref:Requires asyncio support <using-asyncio>: yes.
| :ref:Requires a Twisted reactor <asyncio-without-reactor>: no.
This handler supports http://host/path and https://host/path URLs and
uses the HTTP/1.1 or HTTP/2 protocol for them.
It's implemented using the httpx library and needs it to be installed.
If you want to use this handler you need to replace the default ones for the
http and https schemes:
.. code-block:: python
DOWNLOAD_HANDLERS = {
"http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
"https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
}
Features and limitations ^^^^^^^^^^^^^^^^^^^^^^^^
.. warning::
This handler is experimental, and not yet recommended for production
environments. Future Scrapy versions may introduce related changes without
a deprecation period or warning or even remove it altogether.
=========================== =======================================
HTTP proxies Yes
SOCKS proxies Yes (SOCKS5; requires httpx[socks])
HTTP/2 Yes (requires httpx[http2])
response.certificate DER bytes
Per-request bindaddress No (not supported by the library)
TLS implementation Standard library ssl
=========================== =======================================
Other limitations:
httpx) which may lead to higher resource usage when
using proxy rotation... setting:: HTTPX_HTTP2_ENABLED
HTTPX_HTTP2_ENABLED ^^^^^^^^^^^^^^^^^^^
.. versionadded:: 2.17.0
Default: False
Whether to enable HTTP/2 support in this handler. The httpx[http2] extra
needs to be installed if you want to enable this setting.
.. autoclass:: scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler
| Supported scheme: data.
| :ref:Lazy <lazy-download-handlers>: no.
| :ref:Requires asyncio support <using-asyncio>: no.
| :ref:Requires a Twisted reactor <asyncio-without-reactor>: no.
This handler supports RFC 2397 data:content/type;base64, data URIs.
.. autoclass:: scrapy.core.downloader.handlers.file.FileDownloadHandler
| Supported scheme: file.
| :ref:Lazy <lazy-download-handlers>: no.
| :ref:Requires asyncio support <using-asyncio>: no.
| :ref:Requires a Twisted reactor <asyncio-without-reactor>: no.
This handler supports file:///path local file URIs. It doesn't
support remote files.
.. autoclass:: scrapy.core.downloader.handlers.ftp.FTPDownloadHandler
| Supported scheme: ftp.
| :ref:Lazy <lazy-download-handlers>: no.
| :ref:Requires asyncio support <using-asyncio>: no.
| :ref:Requires a Twisted reactor <asyncio-without-reactor>: yes.
This handler supports ftp://host/path FTP URIs.
It's implemented using :mod:twisted.protocols.ftp.
.. autoclass:: scrapy.core.downloader.handlers.s3.S3DownloadHandler
| Supported scheme: s3.
| :ref:Lazy <lazy-download-handlers>: yes.
| :ref:Requires asyncio support <using-asyncio>: no.
| :ref:Requires a Twisted reactor <asyncio-without-reactor>: no.
This handler supports s3://bucket/path S3 URIs.
It's implemented using the botocore library and needs it to be installed.