Impit HTTP Client

import CodeBlock from '@theme/CodeBlock';

import BasicUsageSource from '!!raw-loader!./basic-usage.ts'; import CheerioCrawlerSource from '!!raw-loader!./cheerio-crawler.ts'; import HttpCrawlerSource from '!!raw-loader!./http-crawler.ts'; import AdvancedConfigSource from '!!raw-loader!./advanced-config.ts';

Introduction

The ImpitHttpClient is an HTTP client implementation based on the Impit library. It enables browser impersonation for HTTP requests, helping you bypass bot detection systems without running an actual browser.

:::info Successor to got-scraping

Impit is the successor to got-scraping, which is no longer actively maintained. We recommend using ImpitHttpClient for all new projects. Impit provides better anti-bot evasion through TLS fingerprinting and HTTP/3 support, while maintaining a smaller package size.

Impit will become the default HTTP client in the next major version of Crawlee.

:::

Why use Impit?

Websites increasingly use sophisticated bot detection that analyzes:

• HTTP fingerprints: User-Agent strings, header ordering, HTTP/2 pseudo-header sequences
• TLS fingerprints: Cipher suites, TLS extensions, and cryptographic details in the ClientHello message

Standard HTTP clients like fetch or axios are easily detected because their fingerprints don't match real browsers. Unlike got-scraping which only handles HTTP-level fingerprinting, Impit also mimics TLS fingerprints, making requests appear to come from real browsers.

Installation

Install the @crawlee/impit-client package:

npm install @crawlee/impit-client

:::note

The impit package includes native binaries and supports Windows, macOS (including ARM), and Linux out of the box.

:::

Basic usage

Pass the ImpitHttpClient instance to the httpClient option of any Crawlee crawler:

<CodeBlock language="ts">{BasicUsageSource}</CodeBlock>

Usage with different crawlers

CheerioCrawler

<CodeBlock language="ts">{CheerioCrawlerSource}</CodeBlock>

HttpCrawler

<CodeBlock language="ts">{HttpCrawlerSource}</CodeBlock>

Configuration options

The ImpitHttpClient constructor accepts the following options:

Available fingerprints

Impit bundles several realistic browser fingerprints. Using version-specific fingerprints can improve success rates against sophisticated bot detection systems that track browser versions and flag outdated signatures.

When using generic fingerprints (chrome or firefox), Impit automatically selects an appropriate version. For most use cases, the generic options are sufficient. However, if you're targeting a site with particularly strict bot detection, or need to match a specific browser environment, you can specify an exact version:

import { ImpitHttpClient } from '@crawlee/impit-client';

// Use a generic Chrome fingerprint
const chromeClient = new ImpitHttpClient({ browser: 'chrome' });

// Use a specific Chrome version
const chrome131Client = new ImpitHttpClient({ browser: 'chrome131' });

// Or a specific Firefox version
const firefox144Client = new ImpitHttpClient({ browser: 'firefox144' });

Advanced configuration

<CodeBlock language="ts">{AdvancedConfigSource}</CodeBlock>

Proxy support

Proxies are configured per-request through Crawlee's proxy management system, not on the ImpitHttpClient itself. Use ProxyConfiguration as you normally would:

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
import { ImpitHttpClient, Browser } from '@crawlee/impit-client';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ['http://proxy1.example.com:8080', 'http://proxy2.example.com:8080'],
});

const crawler = new CheerioCrawler({
    httpClient: new ImpitHttpClient({ browser: Browser.Chrome }),
    proxyConfiguration,
    async requestHandler({ $, request }) {
        console.log(`Scraped ${request.url}`);
    },
});

How it works

Impit achieves browser impersonation at two levels:

1. HTTP level: Mimics browser-specific header ordering, HTTP/2 settings, and pseudo-header sequences that antibot services analyze.

2. TLS level: Uses a patched version of rustls to replicate the exact TLS ClientHello message that browsers send, including cipher suites and extensions.

This dual-layer approach makes requests appear to come from a real browser, significantly reducing blocks from bot detection systems.

Comparison with other solutions

Related links

• Impit GitHub repository
• Custom HTTP Client guide
• Proxy Management guide
• Avoiding blocking guide