© Eric McLean

Prevent deploying broken links to your blog

Having your own blog is fun. Checking internal links or also having an eye on all old URLs you ever linked is not. Fortunately you can automate link checking every time you deploy your website.

I recently read about how links you once posted on your personal page or blog become outdated. The links to external pages are either put in private (403), they vanish completely (404) or they get a proper redirect (302). Whatever the case is, it would be cool the get all your links checked automatically when deploying your site, so you can either start fixing or removing them.

For my Hugo site I wanted to do exactly that, without going to write a scraper to extract links from my site and letting them run through curl. I wanted something to be run against my static HTML files, that I generate - and later deploy to Bunny.net.

I came across a very handy tool written in Rust, called lychee, that does exactly that.

On their website they advertise it with

Catch broken links in seconds
Async, rust-powered simplicity for docs, sites, and codebases

The cool stuff is, lychee works with

• HTML files
• Markdown files
• Text files
• Websites (scraping all links)

which is exactly my use case with static HTML pages.

I implemented it into my deployment workflow, scanning a folder cdn/, where my newly generated files are - and voila, I get a list of URLs with all their HTTP status codes.

1
2
3
4
5
6

- name: "Check links for 404s"
  continue-on-error: true
  run: |
    curl -sLO https://github.com/lycheeverse/lychee/releases/download/nightly/lychee-x86_64-unknown-linux-gnu.tar.gz
    tar -xzf lychee-x86_64-unknown-linux-gnu.tar.gz --strip-components=1
    find ./cdn/ -type f -name '*.html' | ./lychee --files-from -

To just manually check links with lychee on your local machine, you can always run it from command line:

• Check links on a website: lychee https://example.com
• Check links in current folder: lychee .

Of course you can configure (see documentation) which status codes are treated as good or errors. For example, I consider a 403 not an error per se. You can also exclude specific URLs (or via regexp) to not being checked.

This is my lychee.toml config file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Accept 200 (OK) and 403 (Forbidden) responses as valid
accept = ["200", "403"]

max_redirects = 1

# Exclude links
exclude = [
    '\.min\..*\.css$',
    'https://example.com',
    'https://staging.example.com'
]

I run this now after new deployments of my main branch to check if the old links still work and if the new post is linked correctly. So far, this works very well!