Deployment Execution Blueprint

---
title: How to Check Broken Links on a Website Programmatically Using Python
description: A lightweight, multi-threaded Python script blueprint to crawl a website domain, check HTTP status codes, and output all broken 404 links.
category: Python / DevOps
slug: check-broken-links-python
keywords: check broken links python, python broken link checker, web crawler 404, website link validator script, python requests beautifulsoup
---

A broken link (404 Not Found) harms user experience and degrades your website's search engine indexing status. Manually hunting for dead links across hundreds of static flat-files or dynamic routes is highly inefficient, and many cloud-based auditing tools force you into premium monthly tiers.

Building your own bare-metal link crawler in Python provides an automated, free way to audit your technical SEO architecture.

## How the Auditing Engine Works

To efficiently map a site without getting blocked or bottlenecked, a program must execute four specific phases:
1. **Domain Locking:** The script normalizes all crawled URLs and ensures it stays locked to your root domain, avoiding external resource blackholes.
2. **Asynchronous/Multi-threaded State Verification:** It maintains an internal memory matrix of `visited_urls` and `discovered_links` to prevent infinite loops caused by cyclical navigation menus.
3. **DOM Content Scraping:** It parses the HTML structure specifically targeting the standard markup link anchors (`<a href="...">`).
4. **Network Response Extraction:** It fires an HTTP `HEAD` request (which downloads only the server headers rather than the entire page layout) to lightning-fast determine if a status code returns `200 OK` or `404 Broken`.

The Python automation blueprint below utilizes `requests` for fast connection handling and `BeautifulSoup` for clear DOM parsing to build a multi-threaded link validator.

# Python Broken Link Checker Automation Script

import sys
import threading
from queue import Queue
from urllib.parse import urljoin, urlparse
import requests
from bs4 import BeautifulSoup

class LinkCrawler:
    def __init__(self, base_url, max_threads=10):
        self.base_url = base_url
        self.domain = urlparse(base_url).netloc
        self.visited_urls = set()
        self.broken_links = []
        self.queue = Queue()
        self.max_threads = max_threads
        self.lock = threading.Lock()
        
        # Seed the queue with the entry point
        self.queue.put(base_url)

    def is_internal(self, url):
        """Ensure the script stays locked to the target domain."""
        return urlparse(url).netloc == self.domain

    def check_url_status(self, url):
        """Fire a fast HEAD request to evaluate link validity."""
        try:
            # Use HEAD request for speed; fallback to GET if headers are restricted
            response = requests.head(url, timeout=5, allow_redirects=True)
            if response.status_code >= 400:
                # Retry with GET just in case the server blocks HEAD requests
                response = requests.get(url, timeout=5, stream=True)
            return response.status_code
        except requests.RequestException:
            return 999  # Connection error code replacement

    def crawl_worker(self):
        while True:
            try:
                url = self.queue.get(timeout=2)
            except:
                break

            with self.lock:
                if url in self.visited_urls:
                    self.queue.task_done()
                    continue
                self.visited_urls.add(url)

            print(f"[Auditing]: {url}")
            status = self.check_url_status(url)

            if status >= 400:
                with self.lock:
                    self.broken_links.append({"url": url, "status": status})
            else:
                # If internal and valid, extract child links to expand search depth
                if self.is_internal(url) and status == 200:
                    try:
                        response = requests.get(url, timeout=5)
                        soup = BeautifulSoup(response.text, 'html.parser')
                        for anchor in soup.find_all('a', href=True):
                            href = anchor['href']
                            full_url = urljoin(url, href)
                            # Remove trailing fragments or queries for normalization
                            clean_url = full_url.split('#')[0]
                            
                            with self.lock:
                                if clean_url not in self.visited_urls:
                                    self.queue.put(clean_url)
                    except Exception:
                        pass

            self.queue.task_done()

    def run(self):
        threads = []
        for _ in range(self.max_threads):
            t = threading.Thread(target=self.crawl_worker)
            t.daemon = True
            t.start()
            threads.append(t)

        self.queue.join()
        
        print("\n--- AUDIT COMPLETE ---")
        if not self.broken_links:
            print("Success: Zero broken links found across the domain matrix.")
        else:
            print(f"Alert: Found {len(self.broken_links)} broken connections:\n")
            for item in self.broken_links:
                print(f"Status Code [{item['status']}] -> Dead Reference: {item['url']}")

if __name__ == "__main__":
    # Replace with the domain you intend to programmatically inspect
    TARGET_SITE = "https://example.com" 
    
    crawler = LinkCrawler(TARGET_SITE, max_threads=12)
    crawler.run()