Building a Link Checker in Go Without Overwhelming the Network

Suppose you maintain a blog with years of articles. Some links point to documentation that moved. Others point to projects that no longer exist. Opening every link by hand is possible, but it is work we can give to a small command line tool.

The first implementation is straightforward. Read a list of URLs, make an HTTP request for each, and report the failures. Then one slow website holds up the whole scan. Running every request at once removes that sequential wait, but creates a different problem. Now our tool can open more connections than we intended and send a burst of traffic to someone else's server.

Let's build a link checker in Go with explicit limits. It will accept a file of URLs, check several links concurrently, and write a report that distinguishes a missing page from an uncertain result. We will also make it stop when asked, including when requests are still running.

The example uses only Go's standard library. You do not need to move a PHP application to Go to use it. The checker is a separate executable that can run locally or in a publishing pipeline.

I would start with a reviewed list of absolute HTTP and HTTPS URLs, one per line. That keeps link discovery separate from checking. A content export can produce the list later. Crawling HTML, parsing Markdown, and resolving relative links are separate jobs with their own rules.

For this version, the contract is:

• Ignore blank lines and remove URL fragments before deduplication.
• Check each resulting URL once at the application level, following a bounded redirect chain.
• Limit both concurrent checks and how often we hand a new check to a worker.
• Treat 404 and 410 as broken. Report other failures for review.
• Produce one JSON object per completed check and a nonzero exit status when the scan needs attention.

Removing #installation lets us check the same page once when several articles link to different sections. It also means this tool does not verify that those sections exist. We preserve query strings because changing a query can change the resource.

Use this CLI with a trusted list that you control. It is not a public URL checking service. Rejecting non-HTTP schemes does not stop requests to private networks, localhost, or cloud metadata addresses. Redirects and DNS resolution need restrictions too before accepting URLs from strangers.

A goroutine is a function running concurrently with other goroutines in the same program. We start one with go. A channel lets goroutines send typed values to one another.

It is tempting to start a goroutine for every URL, but that still creates one waiting task per link even if we put a semaphore inside each task to limit HTTP requests. For this tool, I prefer a fixed number of workers that take URLs from a shared channel.

The flow looks like this with four workers:

Reviewed URLs
     |
     v
Dispatcher waits between handoffs
     |
     v
Unbuffered jobs channel
     |
     +------------+------------+------------+
     |            |            |            |
     v            v            v            v
 Worker 1     Worker 2     Worker 3     Worker 4
     |            |            |            |
     +------------+------------+------------+
                  |
                  v
        Unbuffered results channel
                  |
                  v
          One JSON report writer

An unbuffered channel has no queue of stored values. Sending waits for a receiver. When every worker is busy, the dispatcher waits rather than creating more workers. If writing the report slows down, workers wait before taking more jobs. That is backpressure inside our program.

Concurrency and request rate are different controls. Four workers can still make many requests per second if responses arrive promptly. Our dispatcher also waits between job handoffs. Redirects can make additional requests within a check, so this is not a strict per-host HTTP rate limiter.

That distinction matters when a website has a published quota. A larger crawler needs host-specific limits that apply to every request, including redirects. The limits here are a starting policy for a small, supervised scan. They cannot guarantee that every target website will accept the traffic.

Install Go 1.27.3 using the official installation instructions. It is the latest stable release listed on the Go downloads page when this article was checked. The examples target Go 1.27.3 and use no third-party packages.

Create a directory and initialise a module:

mkdir linkcheck
cd linkcheck
go mod init example.com/linkcheck

We will create three files:

linkcheck/
    go.mod
    checker.go
    main.go
    checker_test.go

The next three code blocks belong to checker.go, in the order shown. Keep the package declaration and imports only once, at the top of the file.

Start checker.go with the imports, report type, and input reader:

package main

import (
	"bufio"
	"context"
	"fmt"
	"io"
	"net/http"
	"net/url"
	"strings"
	"sync"
	"time"
)

type Result struct {
	URL      string `json:"url"`
	FinalURL string `json:"final_url,omitempty"`
	Status   int    `json:"status,omitempty"`
	Outcome  string `json:"outcome"`
	Error    string `json:"error,omitempty"`
}

func readURLs(input io.Reader) ([]string, error) {
	scanner := bufio.NewScanner(input)
	scanner.Buffer(make([]byte, 4096), 64*1024)
	seen := make(map[string]bool)
	var urls []string
	line := 0

	for scanner.Scan() {
		line++
		raw := strings.TrimSpace(scanner.Text())
		if raw == "" {
			continue
		}

		parsed, err := url.Parse(raw)
		if err != nil {
			return nil, fmt.Errorf("line %d: invalid URL: %w", line, err)
		}
		if (parsed.Scheme != "http" && parsed.Scheme != "https") ||
			parsed.Hostname() == "" || parsed.User != nil {
			return nil, fmt.Errorf("line %d: expected an absolute HTTP(S) URL without credentials", line)
		}

		parsed.Fragment = ""
		parsed.RawFragment = ""
		normalised := parsed.String()
		if seen[normalised] {
			continue
		}

		seen[normalised] = true
		urls = append(urls, normalised)
	}

	if err := scanner.Err(); err != nil {
		return nil, fmt.Errorf("read URLs: %w", err)
	}
	if len(urls) == 0 {
		return nil, fmt.Errorf("input contains no URLs")
	}

	return urls, nil
}

If you are new to Go, := declares a variable and infers its type. Functions can return multiple values, which is why the caller receives both a URL list and an error. A nil error means the operation succeeded. We check errors where they occur rather than throwing an exception.

No network work starts if the input contains a rejected line, and the error includes a line number so the person maintaining the file can fix the entry without searching the whole list.

bufio.Scanner has a token size limit. We set a 64 KiB ceiling explicitly and check scanner.Err() instead of treating an oversized line as the end of a successful read.

This is deliberately modest URL normalisation. Hostname casing and explicit default ports can still produce distinct entries. More importantly, we keep the unique URL list and deduplication map in memory. The worker count bounds active checks, not the total memory used by the input. For millions of links, I would move deduplication to disk and stream the jobs.

Add these functions below readURLs in checker.go:

func newClient(timeout time.Duration) *http.Client {
	return &http.Client{
		Timeout: timeout,
		CheckRedirect: func(req *http.Request, via []*http.Request) error {
			if len(via) >= 5 {
				return fmt.Errorf("redirect limit reached")
			}
			if req.URL.Scheme != "http" && req.URL.Scheme != "https" {
				return fmt.Errorf("redirect uses an unsupported scheme")
			}
			if req.URL.User != nil {
				return fmt.Errorf("redirect contains credentials")
			}
			return nil
		},
	}
}

func checkURL(ctx context.Context, client *http.Client, target string) Result {
	result := Result{URL: target, Outcome: "review"}
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
	if err != nil {
		result.Error = err.Error()
		return result
	}
	req.Header.Set("User-Agent", "article-linkcheck/1.0")

	response, err := client.Do(req)
	if err != nil {
		result.Error = err.Error()
		return result
	}
	defer response.Body.Close()

	result.Status = response.StatusCode
	result.FinalURL = response.Request.URL.String()

	switch {
	case response.StatusCode >= 200 && response.StatusCode < 300:
		result.Outcome = "ok"
	case response.StatusCode == http.StatusNotFound ||
		response.StatusCode == http.StatusGone:
		result.Outcome = "broken"
	}

	return result
}

Go's HTTP client does not return an error merely because a server answers 404 or 503. We need to inspect the status ourselves. An error from Do means something else prevented the request from completing normally, such as a timeout, a TLS failure, or our redirect policy rejecting a hop.

I would keep those outcomes separate. A 403 may be a site refusing automated clients. A 429 asks us to slow down. A 503 may be temporary. None proves that an editor should delete a link. Even 404 and 410 are observations to investigate, not permission for the tool to edit articles.

The HTTP client documentation recommends reusing clients and states that they are safe for concurrent use. We create one client for the scan. Its timeout covers a check's connection work, redirects, and response reading. It does not include the time a URL spends waiting for dispatch. There is also a separate deadline for the whole scan later.

Our redirect policy sends at most five application-level requests along a chain, including the initial request. A loop becomes a review result rather than an endless scan. This limit is not a security boundary against private network access.

HEAD avoids a response body, but a website can reject it or handle it differently from GET. I would start with GET for a tool meant to check links that readers open. The checker closes the body immediately after receiving headers. It does not download or validate the whole page.

There is a cost to that choice. For HTTP/1.x, reusing a connection can require reading the response body to EOF as well as closing it. Closing early can give up connection reuse. Reading every body to EOF would let a large download consume bandwidth just to inspect a status code.

We choose early closure here. That does not mean zero body bytes cross the network. It means our code does not keep reading them. A later version could drain small bodies within a byte limit, but I would measure connection churn before adding that behaviour.

A successful status also does not prove that a page contains the expected content. A login page, a soft 404, or a JavaScript error screen can return 200. This checker reports HTTP reachability, not browser rendering or content correctness.

Append the final function to checker.go:

func checkAll(
	ctx context.Context,
	client *http.Client,
	urls []string,
	workers int,
	interval time.Duration,
) <-chan Result {
	jobs := make(chan string)
	results := make(chan Result)
	var running sync.WaitGroup
	running.Add(workers + 1)

	go func() {
		defer running.Done()
		defer close(jobs)

		for _, target := range urls {
			timer := time.NewTimer(interval)
			select {
			case <-ctx.Done():
				timer.Stop()
				return
			case <-timer.C:
			}

			select {
			case <-ctx.Done():
				return
			case jobs <- target:
			}
		}
	}()

	for i := 0; i < workers; i++ {
		go func() {
			defer running.Done()

			for target := range jobs {
				if ctx.Err() != nil {
					return
				}
				result := checkURL(ctx, client, target)
				select {
				case <-ctx.Done():
					return
				case results <- result:
				}
			}
		}()
	}

	go func() {
		running.Wait()
		close(results)
	}()

	return results
}

The dispatcher waits for the configured interval before each handoff, including the first. Because it starts a new timer after handing off a job, time spent waiting for a worker does not accumulate a reserve of dispatch permits that could send several checks together when workers become free. Workers share the client but never write to a shared result slice.

Notice who closes each channel. The dispatcher owns jobs and closes it when no more URLs will arrive. A worker cannot close results because another worker might still need to send, so the final goroutine waits for both the dispatcher and every worker before closing that channel.

defer schedules a call for when the surrounding function returns. Here it makes each goroutine call running.Done() even when cancellation causes an early return. The <-chan Result return type lets callers receive results without letting them send into the channel.

Cancellation must also reach blocking operations. The dispatcher can stop while waiting for its timer or a free worker. Requests receive the same context. A worker waiting to send its result can abandon that send when the scan stops. Workers waiting for jobs exit when the dispatcher closes the channel.

The Go article on pipelines and cancellation explains why a producer can leak a goroutine if its consumer leaves without receiving the remaining values. Adding a large channel buffer only postpones that problem. Our caller must drain results or cancel the context.

Cancellation can discard results, including a result that completed just before the cancellation. An interrupted scan is incomplete. We must not report the remaining links as healthy.

Create main.go with the complete entry point:

package main

import (
	"context"
	"encoding/json"
	"flag"
	"fmt"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	os.Exit(run())
}

func run() int {
	workers := flag.Int("workers", 4, "maximum concurrent checks")
	interval := flag.Duration("interval", 250*time.Millisecond, "wait between job handoffs")
	timeout := flag.Duration("timeout", 10*time.Second, "timeout for one check")
	budget := flag.Duration("budget", 5*time.Minute, "time budget for the network scan")
	flag.Parse()

	if flag.NArg() != 1 || *workers < 1 || *interval <= 0 || *timeout <= 0 || *budget <= 0 {
		fmt.Fprintln(os.Stderr, "usage: linkcheck [positive limits] urls.txt")
		return 2
	}

	file, err := os.Open(flag.Arg(0))
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		return 2
	}
	urls, err := readURLs(file)
	file.Close()
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		return 2
	}

	signals, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()
	ctx, cancel := context.WithTimeout(signals, *budget)
	defer cancel()
	client := newClient(*timeout)
	defer client.CloseIdleConnections()

	encoder := json.NewEncoder(os.Stdout)
	results := checkAll(ctx, client, urls, *workers, *interval)
	checked, problems := 0, 0
	for result := range results {
		if err := encoder.Encode(result); err != nil {
			cancel()
			for range results {
			}
			fmt.Fprintln(os.Stderr, err)
			return 2
		}
		checked++
		if result.Outcome != "ok" {
			problems++
		}
	}

	fmt.Fprintf(os.Stderr, "checked=%d total=%d needs_attention=%d\n", checked, len(urls), problems)
	if ctx.Err() != nil {
		fmt.Fprintln(os.Stderr, "scan incomplete:", ctx.Err())
		return 2
	}
	if problems > 0 {
		return 1
	}
	return 0
}

signal.NotifyContext connects Ctrl+C and SIGTERM to the scan context. context.WithTimeout adds a deadline for the network scan. If either fires, the dispatcher and active requests receive the cancellation. The budget starts after input loading, so it does not bound reading the file. The context documentation covers this parent and child relationship.

The report writer is the only goroutine that encodes JSON. Results appear in completion order, not input order. That lets us report a finished link without waiting for an earlier slow link. If output fails, we cancel and drain the channel so the pipeline can finish before returning.

Cancellation does not interrupt an arbitrary blocking write to stdout. This CLI assumes a consumer that keeps reading, such as a normal file. A service with a potentially stalled output destination needs its own output deadline or a different reporting mechanism.

Exit 0 means every check was ok. Exit 1 means at least one result needs attention. Exit 2 means invalid input, an operational error, or an incomplete scan. These are our CLI's conventions. No application retry loop is included. Inspect a failed scan before sending more traffic. Go's transport can still retry certain requests under its documented conditions, so "once" here means one call to Do per URL, not a guaranteed single network attempt.

Build the executable before using its exit status in a pipeline:

go build -o linkcheck .
./linkcheck -workers=4 -interval=500ms -timeout=8s -budget=3m urls.txt > report.jsonl

Unlike running a compiled binary directly, go run does not preserve the program's nonzero exit status unchanged. The flags must also appear before the input filename because the standard flag parser stops at the first positional argument.

For a file containing these illustrative URLs:

https://example.com/docs#install
https://example.com/docs#configuration
https://example.com/removed

The checker makes two logical checks after removing fragments. An illustrative report, assuming those pages respond with 200 and 404, looks like this:

{"url":"https://example.com/docs","final_url":"https://example.com/docs","status":200,"outcome":"ok"}
{"url":"https://example.com/removed","final_url":"https://example.com/removed","status":404,"outcome":"broken"}

Those statuses are examples, not claims about the current contents of example.com. The summary goes to stderr so report.jsonl contains only JSON records.

Tests should not fail because an unrelated website is down. Go's httptest package starts a local HTTP server with a handler that we control. Create checker_test.go:

package main

import (
	"context"
	"net/http"
	"net/http/httptest"
	"reflect"
	"strings"
	"sync/atomic"
	"testing"
	"time"
)

func TestReadURLs(t *testing.T) {
	urls, err := readURLs(strings.NewReader(
		"https://example.com/docs#one\n\nhttps://example.com/docs#two\nhttps://example.com/docs?v=2\n",
	))
	want := []string{"https://example.com/docs", "https://example.com/docs?v=2"}
	if err != nil || !reflect.DeepEqual(urls, want) {
		t.Fatalf("urls=%v err=%v, want %v", urls, err, want)
	}
	if _, err := readURLs(strings.NewReader("file:///etc/passwd\n")); err == nil {
		t.Fatal("expected a rejected URL")
	}
}

func TestCheckURL(t *testing.T) {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		switch r.URL.Path {
		case "/ok":
			w.WriteHeader(http.StatusOK)
		case "/moved":
			http.Redirect(w, r, "/ok", http.StatusMovedPermanently)
		case "/loop":
			http.Redirect(w, r, "/loop", http.StatusFound)
		case "/limited":
			w.WriteHeader(http.StatusTooManyRequests)
		case "/unavailable":
			w.WriteHeader(http.StatusServiceUnavailable)
		default:
			http.NotFound(w, r)
		}
	}))
	defer server.Close()
	client := newClient(time.Second)
	defer client.CloseIdleConnections()

	for _, test := range []struct {
		path    string
		outcome string
		status  int
	}{
		{"/ok", "ok", 200},
		{"/moved", "ok", 200},
		{"/missing", "broken", 404},
		{"/limited", "review", 429},
		{"/unavailable", "review", 503},
		{"/loop", "review", 0},
	} {
		t.Run(test.path, func(t *testing.T) {
			result := checkURL(context.Background(), client, server.URL+test.path)
			if result.Outcome != test.outcome || result.Status != test.status {
				t.Fatalf("unexpected result: %+v", result)
			}
			if test.path == "/moved" && result.FinalURL != server.URL+"/ok" {
				t.Fatalf("unexpected redirect destination: %s", result.FinalURL)
			}
			if test.path == "/loop" && result.Error == "" {
				t.Fatal("expected a redirect error")
			}
		})
	}
}

func TestCheckURLTimeout(t *testing.T) {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		<-r.Context().Done()
	}))
	defer server.Close()
	client := newClient(100 * time.Millisecond)
	defer client.CloseIdleConnections()
	result := checkURL(context.Background(), client, server.URL)
	if result.Outcome != "review" || result.Error == "" {
		t.Fatalf("expected an uncertain timeout result: %+v", result)
	}
}

func TestCheckAllBoundsConcurrency(t *testing.T) {
	var active, peak atomic.Int32
	started := make(chan struct{}, 12)
	release := make(chan struct{})
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		current := active.Add(1)
		defer active.Add(-1)
		for previous := peak.Load(); current > previous; previous = peak.Load() {
			if peak.CompareAndSwap(previous, current) {
				break
			}
		}
		started <- struct{}{}
		select {
		case <-release:
		case <-r.Context().Done():
		}
	}))
	defer server.Close()
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	client := newClient(time.Second)
	defer client.CloseIdleConnections()
	urls := make([]string, 12)
	for i := range urls {
		urls[i] = server.URL
	}
	results := checkAll(ctx, client, urls, 3, time.Millisecond)
	for i := 0; i < 3; i++ {
		select {
		case <-started:
		case <-ctx.Done():
			t.Fatal("workers did not start")
		}
	}
	close(release)
	count := 0
	for result := range results {
		if result.Outcome != "ok" {
			t.Fatalf("unexpected result: %+v", result)
		}
		count++
	}
	if count != len(urls) || peak.Load() != 3 {
		t.Fatalf("results=%d peak=%d", count, peak.Load())
	}
}

func TestCheckAllCancellation(t *testing.T) {
	started := make(chan struct{}, 1)
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		started <- struct{}{}
		<-r.Context().Done()
	}))
	defer server.Close()
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()
	client := newClient(5 * time.Second)
	defer client.CloseIdleConnections()
	results := checkAll(ctx, client, []string{server.URL, server.URL}, 1, time.Millisecond)
	select {
	case <-started:
	case <-time.After(2 * time.Second):
		t.Fatal("request did not start")
	}
	cancel()
	deadline := time.After(2 * time.Second)
	for {
		select {
		case _, open := <-results:
			if !open {
				return
			}
		case <-deadline:
			t.Fatal("pipeline did not stop after cancellation")
		}
	}
}

The concurrency test holds requests open until three workers have reached the server, then releases them and checks both the number of results and the peak number of active handlers. Repeated URLs are intentional in that test. It exercises the worker pool directly, after the separate input deduplication step.

The cancellation test waits until a request has started, then cancels the parent context. It checks that the results channel closes without waiting for the five second client timeout. The test server cooperates with cancellation, but a remote server may continue its own work after our client stops waiting, so this test does not prove that cancellation stops remote execution.

Run the tests with the race detector and check the code:

go test -race -count=10 ./...
go vet ./...

The race detector can find unsynchronised memory access on executed paths. It does not prove that every possible schedule is correct. Channels handle result delivery here, and the test's shared counters use atomic operations.

Before relying on the checker in a publishing pipeline, I would also test output write failures, the whole-scan deadline, and pacing with a controlled transport. None needs a public website.

For a blog, the next useful addition is source locations. A broken URL is easier to fix when the report names the article and line that contains it. Keep a mapping from each normalised URL to its references, then check the URL once and attach the result to those references.

Link extraction deserves a parser appropriate to the content. Markdown has reference links, relative paths, and embedded HTML. A regular expression that catches one inline link format will miss valid links elsewhere. Discovery should produce our reviewed URL list rather than expand this checker into an unrestricted crawler.

I would also keep retries out of the first version. If a host returns 429, more workers are the wrong response. Stop or defer that host and respect its retry instructions when adding a retry policy. The distinction between deadlines, concurrency limits, and retry budgets is covered in When Retries Make Things Worse.

Go gives us enough standard-library tools to build this checker without a web framework or a queue service. The design still depends on choices we have to make ourselves. How much traffic may it send? What does a failed request mean? Who stops the workers when the report is no longer needed?

I would use the checker as a review aid before making it a publishing gate. Collect uncertain results separately, inspect hosts that reject automated requests, and choose limits for the sites you actually check. A tool that marks every temporary failure as a broken link creates more work for the editor.

The complete app, tests, and setup instructions are available in the linkcheck repository on GitHub.

Try it with a small reviewed URL list and a local slow server first. Check that reducing the worker count and cancelling the scan change the behaviour you expect before pointing it at a full archive.

I hope that you liked this article and if you do, don't forget to share this article with your friends!!! See ya!