Project: Concurrent Web Page Fetcher

1. What You'll Build

A command-line tool that fetches any number of URLs at the same time and prints the status code and elapsed time for each. Pass URLs as arguments:

$ go run main.go https://example.com https://golang.org https://httpbin.org/delay/2
200  120ms  https://example.com
200  180ms  https://golang.org
200  2.01s  https://httpbin.org/delay/2
done — 12 URLs in 2.03s (would have taken ~6s serially)

This is the project that makes Go feel magical. You write what looks like sequential code, sprinkle in go, and suddenly everything happens in parallel.

2. First, the Slow Way (Serial)

Before showing the concurrent version, here's how you would write it without goroutines — one URL at a time:

package main

import (
    "fmt"
    "net/http"
    "os"
    "time"
)

func main() {
    urls := os.Args[1:]
    start := time.Now()

    for _, url := range urls {
        t := time.Now()
        resp, err := http.Get(url)
        if err != nil {
            fmt.Printf("ERR  %s  %v\n", url, err)
            continue
        }
        resp.Body.Close()
        fmt.Printf("%d  %s  %s\n", resp.StatusCode, time.Since(t), url)
    }

    fmt.Printf("done — %d URLs in %s\n", len(urls), time.Since(start))
}

This works, but if you give it 12 URLs that each take ~500ms, the whole program takes 6 seconds. We're wasting time waiting on the network when we could be making the other 11 requests.

3. The Fast Way (Concurrent)

The complete concurrent program. Same idea, but each request runs in its own goroutine and reports back through a channel.

package main

import (
    "fmt"
    "net/http"
    "os"
    "time"
)

type result struct {
    url     string
    status  int
    elapsed time.Duration
    err     error
}

func fetch(url string, out chan<- result) {
    t := time.Now()
    resp, err := http.Get(url)
    if err != nil {
        out <- result{url: url, err: err, elapsed: time.Since(t)}
        return
    }
    resp.Body.Close()
    out <- result{url: url, status: resp.StatusCode, elapsed: time.Since(t)}
}

func main() {
    urls := os.Args[1:]
    if len(urls) == 0 {
        fmt.Println("usage: fetcher <url> <url> ...")
        return
    }

    out := make(chan result, len(urls))
    start := time.Now()

    for _, url := range urls {
        go fetch(url, out)
    }

    for i := 0; i < len(urls); i++ {
        r := <-out
        if r.err != nil {
            fmt.Printf("ERR  %-7s %s  (%v)\n", r.elapsed.Round(time.Millisecond), r.url, r.err)
            continue
        }
        fmt.Printf("%d  %-7s %s\n", r.status, r.elapsed.Round(time.Millisecond), r.url)
    }

    fmt.Printf("done — %d URLs in %s\n", len(urls), time.Since(start).Round(time.Millisecond))
}

4. How It Works

Four small ideas working together:

1. A result struct bundles everything we want to know about one request — URL, status code, elapsed time, and any error. We pass these through the channel.
2. fetch is a regular function, but we call it with go fetch(url, out) instead of fetch(url, out). That one keyword launches it as a goroutine and returns immediately. The function runs concurrently with all the others.
3. The channel out is the only way the goroutines talk to main. We made it buffered (make(chan result, len(urls))) so workers can dump their result and exit even if main isn't ready to read yet.
4. The receive loop r := <-out blocks until some goroutine sends a result, then immediately moves on to receive the next. We get results in the order they complete, not the order we sent them — which is exactly what makes this feel snappy.

5. Run It

$ mkdir fetcher && cd fetcher
$ go mod init fetcher
# paste the code into main.go
$ go run main.go \
    https://example.com \
    https://golang.org \
    https://httpbin.org/delay/1 \
    https://httpbin.org/delay/2

Try it with a few URLs first, then crank it up to 20 or 30. Notice that the total time stays roughly the same as your slowest URL — not the sum of all URLs.

6. Extensions

• Add a timeout — use http.Client{Timeout: 5 * time.Second} so a hung server can't stall the whole batch.
• Limit concurrency — firing off 10,000 goroutines at once will exhaust file descriptors. Use a semaphore channel (sem := make(chan struct{}, 50)) to cap parallelism.
• Show response size — read resp.ContentLength and print it next to the status code.
• Read URLs from a file — bufio.Scanner on os.Stdin lets you do cat urls.txt | go run main.go.
• Persist results — write each line to a CSV for later analysis.