Index

{{ define "main" }}

Fast and Elegant Scraping Framework for Gophers

GitHub

Colly provides a clean interface to write any kind of crawler/scraper/spider

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

Features

• Clean API
• Fast (>1k request/sec on a single core)
• Manages request delays and maximum concurrency per domain
• Automatic cookie and session handling
• Sync/async/parallel scraping
• Distributed scraping
• Caching
• Automatic encoding of non-unicode responses
• Robots.txt support
• Google App Engine support

{{ highlight func main() { c := colly.NewCollector() // Find and visit all links c.OnHTML("a", func(e *colly.HTMLElement) { e.Request.Visit(e.Attr("href")) }) c.OnRequest(func(r *colly.Request) { fmt.Println("Visiting", r.URL) }) c.Visit("http://go-colly.org/") } "go" "linenos=table" }}

Batteries included

Colly comes with all the tools you need for scraping.

Examples

Business ready

Free for commercial use.

License

Open Source

Development of Colly is community driven and public.

Project

{{ end }}