HTTP crawler

import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; import ApiLink from '@site/src/components/ApiLink'; import HttpCrawlerSource from '!!raw-loader!roa-loader!./http_crawler.ts';

This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">HttpCrawler</ApiLink> to build a HTML crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.

<RunnableCodeBlock className="language-js" type="cheerio"> {HttpCrawlerSource} </RunnableCodeBlock>