Web Scraping with Golang — Colly, goquery, and Concurrency

Go is arguably the best mainstream language for the fetching half of web scraping: goroutines make a thousand concurrent requests feel like nothing, and the result compiles to a single binary you can drop on any server. The parsing and anti-bot half is where Python still has the deeper toolbox. This guide builds a Go scraper the honest way — standard library plus goquery first, then Colly for crawling and rate limits, then a worker pool that will not get your IP banned — and tells you exactly where DIY Go scraping stops working and an API takes over.

Why Go for web scraping — and why not

The case for Go is real, so let's make it fairly.

Raw speed and concurrency. Scraping is I/O-bound work, and goroutines are the cheapest concurrency primitive in any mainstream language — a few kilobytes of stack each. A Go scraper saturating 200 concurrent connections is an afternoon project; the equivalent in Python means asyncio, an async HTTP client, and a mental model many teams never quite internalize.

Single-binary deploys. go build gives you one static executable. No virtualenv, no dependency resolution on the server, no "works on my machine." For scrapers that live on cron jobs, containers, or cheap VPSes, this is a genuinely underrated advantage.

Typed output by default. You parse into structs, so schema drift shows up as a compile error or an obvious empty field — not a silent KeyError three days into a run.

Now the honest cons. Go's scraping ecosystem is smaller: Python has BeautifulSoup, Scrapy, Playwright with stealth patches, curl-impersonate bindings, and a decade of Stack Overflow answers for every edge case. Go has goquery, Colly, and chromedp — good tools, but fewer of them, and far fewer anti-bot and stealth libraries. You will also write more boilerplate: error handling on every request, explicit structs for every shape. If your bottleneck is parsing messy HTML or evading sophisticated bot detection rather than throughput, web scraping with Python is the more forgiving starting point — that guide covers the same journey with requests, BeautifulSoup, and Scrapy.

If throughput, deployment, and long-running reliability are what you care about, keep reading.

Level 1: net/http + goquery

The standard library's net/http fetches pages; goquery gives you jQuery-style CSS selectors over the parsed DOM. Install it with:

go get github.com/PuerkitoBio/goquery

Here is a complete scraper for a books.toscrape.com-style catalog page — fetch, check the status code, select elements, extract into structs, emit JSON:

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"strings"

	"github.com/PuerkitoBio/goquery"
)

type Book struct {
	Title   string `json:"title"`
	Price   string `json:"price"`
	InStock bool   `json:"in_stock"`
}

func main() {
	res, err := http.Get("https://books.toscrape.com/")
	if err != nil {
		log.Fatal(err)
	}
	defer res.Body.Close()
	if res.StatusCode != http.StatusOK {
		log.Fatalf("status code error: %d %s", res.StatusCode, res.Status)
	}

	doc, err := goquery.NewDocumentFromReader(res.Body)
	if err != nil {
		log.Fatal(err)
	}

	var books []Book
	doc.Find("article.product_pod").Each(func(i int, s *goquery.Selection) {
		books = append(books, Book{
			Title:   s.Find("h3 a").AttrOr("title", ""),
			Price:   strings.TrimSpace(s.Find(".price_color").Text()),
			InStock: s.Find(".instock.availability").Length() > 0,
		})
	})

	out, err := json.MarshalIndent(books, "", "  ")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(out))
}

The pattern to internalize: NewDocumentFromReader parses the response body once, Find takes any CSS selector, Each iterates matches, and AttrOr / Text pull values with sane fallbacks. Always check res.StatusCode before parsing — goquery will happily parse an error page into an empty selection, and you will chase a phantom bug. Two more standard-library notes for production: use http.Client with an explicit Timeout instead of the zero-value default (which never times out), and remember goquery expects UTF-8 — transcode legacy encodings first.

This level is perfect for one-off jobs against static HTML. The moment you need to follow links across pages, you want a framework.

Level 2: Colly — the Go scraping framework

Colly wraps the fetch-parse-follow loop into a collector with event callbacks. It is the closest thing Go has to Scrapy, and it handles web crawling — traversing links, not just parsing one page — with almost no ceremony. Note the v2 module path:

go get github.com/gocolly/colly/v2

The same bookstore, now crawling every catalog page via the "next" link, with polite delays built in:

package main

import (
	"fmt"
	"log"
	"time"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector(
		colly.AllowedDomains("books.toscrape.com"),
	)

	// Politeness: delay between requests to this domain.
	if err := c.Limit(&colly.LimitRule{
		DomainGlob:  "books.toscrape.com",
		Delay:       500 * time.Millisecond,
		RandomDelay: 500 * time.Millisecond,
	}); err != nil {
		log.Fatal(err)
	}

	c.OnHTML("article.product_pod", func(e *colly.HTMLElement) {
		fmt.Printf("%s — %s\n",
			e.ChildAttr("h3 a", "title"),
			e.ChildText(".price_color"))
	})

	// Pagination: follow the next-page link until it stops existing.
	c.OnHTML("li.next a", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnError(func(r *colly.Response, err error) {
		log.Println("request failed:", r.Request.URL, err)
	})

	if err := c.Visit("https://books.toscrape.com/"); err != nil {
		log.Fatal(err)
	}
}

Three ideas carry the whole framework. OnHTML callbacks fire for every element matching a selector, so extraction and link-following are both just callbacks. Pagination is e.Request.Visit(e.Attr("href")) — Colly resolves relative URLs, deduplicates visited pages, and respects AllowedDomains so you cannot accidentally crawl the wider internet. LimitRule gives you per-domain Delay plus RandomDelay jitter without writing any timing code yourself.

Level 3: concurrency done right

This is the section Go readers came for, and the place most Go scrapers go wrong. The naive move — spawn one goroutine per URL — works in a demo and melts in production: you open thousands of simultaneous connections, the target site rate-limits or bans you, and your own process runs out of file descriptors. Concurrency needs two bounds: how many workers run at once, and how many requests per second leave the process. (For the server-side view of why, see what is rate limiting.)

The idiomatic shape is a bounded worker pool fed by a channel, with a shared time.Ticker acting as a global rate limiter:

func scrapeAll(urls []string, workers, perSecond int) {
	jobs := make(chan string)
	ticker := time.NewTicker(time.Second / time.Duration(perSecond))
	defer ticker.Stop()

	var wg sync.WaitGroup
	for i := 0; i < workers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for u := range jobs {
				<-ticker.C // one tick per request, shared by all workers
				if err := scrapeOne(u); err != nil {
					log.Println(u, err)
				}
			}
		}()
	}

	for _, u := range urls {
		jobs <- u
	}
	close(jobs)
	wg.Wait()
}

Ten workers at five requests per second is a sensible starting point for a site you do not control. The ticker is shared, so adding workers increases overlap tolerance (slow responses do not stall the queue) without increasing request rate — the two knobs stay independent, which is exactly what you want when tuning against a rate limiter.

If you are already inside Colly, you get the same behavior declaratively — create the collector with colly.Async(), cap Parallelism in the LimitRule, and block on Wait:

c := colly.NewCollector(colly.Async())
c.Limit(&colly.LimitRule{
	DomainGlob:  "*",
	Parallelism: 4,
	Delay:       200 * time.Millisecond,
})
// ... register OnHTML callbacks, then:
c.Visit(startURL)
c.Wait() // block until all async requests finish

Either way, the rule is the same: bound the workers, meter the requests, and treat 429 and 403 responses as a signal to back off, not to retry harder.

JS-heavy pages: chromedp exists, but weigh the cost

Everything above assumes the data is in the HTML the server sends. Increasingly it is not — it is rendered client-side by JavaScript, and net/http sees an empty shell. Go's answer is chromedp, which drives a real Chrome instance over the DevTools Protocol (see what is a headless browser):

ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()

var html string
err := chromedp.Run(ctx,
	chromedp.Navigate("https://example.com/js-rendered-listing"),
	chromedp.WaitVisible(".product-grid"),
	chromedp.OuterHTML("html", &html),
)
// feed html into goquery.NewDocumentFromReader as before

It works, and the API is pleasant. But be honest about what you just signed up for: a full Chrome per context (hundreds of MB of RAM), Chrome binaries in your previously tidy single-binary deployment, and roughly 10–50x the latency of a plain HTTP request. Your throughput drops from hundreds of pages per second to a handful. Use chromedp when a specific high-value target truly requires rendering — not as the default path. Often the better move is opening DevTools and finding the JSON endpoint the page itself calls; scraping that with net/http is faster than rendering will ever be.

The production wall — and the API bridge

Scale any of the above against real commercial targets — Amazon, Google, social platforms — and you hit a wall that has nothing to do with your code quality: anti-bot systems. Datacenter IPs get flagged, CAPTCHAs appear, and headers alone will not save you.

Go has a specific problem here worth knowing about. Anti-bot vendors fingerprint the TLS handshake — the cipher suites and extensions a client offers, hashed into a JA3/JA4 signature — and Go's crypto/tls stack produces a handshake that looks nothing like Chrome's. Protected sites can identify a Go client before it sends a single HTTP byte, no matter how carefully you spoof the User-Agent. This is one flavor of browser fingerprinting, and while community forks that impersonate browser TLS exist, they are exactly the kind of niche, fast-decaying dependency the Go ecosystem has fewer of than Python's. For the full landscape of what you are up against, see scraping sites that block bots.

At that point the pragmatic move is to keep Go for what it is great at — orchestration, concurrency, pipelines — and delegate the hostile fetch to a scraping API. Crawlora exposes structured endpoints over plain HTTPS, so calling it is ordinary net/http, and the response is normalized JSON instead of HTML you have to parse and babysit:

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"net/url"
	"os"
)

type searchResponse struct {
	Code int `json:"code"`
	Data []struct {
		ASIN      string  `json:"asin"`
		Link      string  `json:"link"`
		ListPrice float64 `json:"list_price"`
	} `json:"data"`
}

func main() {
	q := url.Values{}
	q.Set("k", "mechanical keyboard")
	q.Set("page", "1")

	req, err := http.NewRequest(http.MethodGet,
		"https://api.crawlora.net/api/v1/amazon/search?"+q.Encode(), nil)
	if err != nil {
		log.Fatal(err)
	}
	req.Header.Set("x-api-key", os.Getenv("CRAWLORA_API_KEY"))

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer res.Body.Close()

	var sr searchResponse
	if err := json.NewDecoder(res.Body).Decode(&sr); err != nil {
		log.Fatal(err)
	}
	for _, item := range sr.Data {
		fmt.Printf("%s  $%.2f  %s\n", item.ASIN, item.ListPrice, item.Link)
	}
}

No proxies to rotate, no TLS impersonation fork to track, no parser to fix when Amazon ships a redesign — typed structs over documented JSON, which is the workflow Go is built for. Proxies, rendering, and retries happen behind the endpoint. There is also an official Go SDK (alongside TypeScript and Python) if you prefer generated clients over raw net/http, and the same worker-pool pattern from Level 3 applies unchanged — point it at API calls instead of raw pages. Billing is pay-on-success: you are charged only for successful 2xx responses, so a failed fetch costs nothing (pricing). Whether to scrape a target yourself or call an API for it is a judgment call — web scraping vs API walks through it.

• Static HTML, one site, modest volume: net/http + goquery is all you need.
• Crawling with pagination: Colly v2 with AllowedDomains and a LimitRule (Delay + RandomDelay).
• High volume: bounded worker pool + shared time.Ticker, or colly.Async with Parallelism — never unbounded goroutines.
• Set a real User-Agent and an http.Client timeout; back off on 429/403.
• JS-rendered pages: check for the underlying JSON endpoint before reaching for chromedp.
• Anti-bot walls and TLS fingerprint blocks: route those targets through a scraping API and keep your Go code for orchestration.

Wrap-up

Go rewards scrapers that look like the systems Go was designed for: concurrent, bounded, typed, deployed as one binary. Start with net/http and goquery, graduate to Colly when you need to crawl, and put a worker pool with a ticker in front of everything. When a target's defenses cost more engineering time than the data is worth, bridge to an API from the same codebase — you can try the endpoints in the playground without writing a line of Go, then browse the docs for the full endpoint catalog.

Keep Go for the pipeline, skip the anti-bot arms race

Structured endpoints, normalized JSON, managed proxies and retries — called with plain net/http or the official Go SDK. Free tier: 2,000 credits/month, no card.

Try the PlaygroundSee the Web Scraping API

Related reading: Web scraping with Python (the same journey in the bigger ecosystem) · Scraping sites that block bots · Web scraping vs API · What is browser fingerprinting