🌑 Kage: Shadowing the Web for Permanent Offline Access

Release License

Kage (meaning shadow in Japanese: 影) is a powerful utility designed to clone websites into local directories for offline consumption. Its primary distinction is that it completely strips out all JavaScript, leaving you with a static, secure, and permanent snapshot of the site.

The Core Philosophy

~~"Just use 'Save Page As' in your browser."~~

We've all tried it. You save a page today, but six months later, it's a broken mess: a perpetual loading spinner, a blank white screen, or a file that still tries to ping a dead analytics server.

Kage solves this by:

Driving a legitimate headless Chrome instance.
Waiting for the page to fully settle and render.
Capturing the DOM exactly as a human would see it.
Purging all scripts and rewriting CSS, font, and image paths to be local.

The result is a set of .html files that look like the live site but execute zero code.

⚙️ How It Works

The process follows a linear pipeline to ensure the "shadow" is an accurate reflection of the original:

🛠️ Installation

You have several ways to get Kage running on your system:

Via Go: go install github.com/tamnd/kage/cmd/kage@latest
Prebuilt Binaries: Download a .deb, .rpm, .apk, or archive from the releases page.

Docker (Bundled Chromium):

docker run --rm -v "$PWD/out:/out" ghcr.io/tamnd/kage clone paulgraham.com

Note: Since Kage controls a real browser, it requires Chrome or Chromium. It usually finds the system install automatically, but you can specify a path using the --chrome flag or the KAGE_CHROME environment variable.

Pro tip: Shell completion is supported for bash, zsh, fish, and powershell via kage completion <shell>.

🚀 Quick Start Guide

Want to archive Paul Graham's essays for a flight or for the year 2050? Follow these steps:

1. Clone the content

kage clone paulgraham.com
# Saves to $HOME/data/kage/paulgraham.com/

2. View it offline

kage serve $HOME/data/kage/paulgraham.com
# Now visit http://127.0.0.1:8800

3. (Optional) Package it You can compress the mirror into a single .zim file:

kage pack paulgraham.com
kage open paulgraham.com.zim

4. (Optional) Create a standalone binary Turn the website into a single executable file that serves itself:

kage pack paulgraham.com --format binary -o paulgraham ./paulgraham

📖 Command Reference

Command	Purpose
`kage clone <url>`	Renders a site via headless Chrome and creates a script-free mirror.
`kage serve [dir]`	Hosts a cloned directory via a local HTTP server for previewing.
`kage pack <dir>`	Converts a mirror into a ZIM archive, a binary, or a double-click app.
`kage open <file>`	Launches a packed ZIM file for offline reading.

🔍 Deep Dive: The `clone` Process

Kage performs a polite, breadth-first crawl. It respects robots.txt and utilizes sitemap.xml for discovery. It is designed to be idempotent; whether a page is accessed via http, https, or with/without a trailing slash, it is only fetched once.

If you interrupt the process with Ctrl-C, Kage saves its progress and resumes from the same spot upon restart.

Cloning Examples:

Limited Scope: kage clone paulgraham.com --max-pages 50 --max-depth 2
Specific Section: kage clone go.dev --scope-prefix /doc
Deep Capture: kage clone example.com --subdomains --scroll (The --scroll flag triggers lazy-loaded images).
Update Mirror: kage clone paulgraham.com --refresh

Configuration Flags

The logic for page limits can be expressed as: $\text{Total Pages} \le \text{max-pages} \quad \text{and} \quad \text{Link Distance} \le \text{max-depth}$

Flag	Default	Description
`-o, --out`	`$HOME/data/kage`	The destination directory for the mirror.
`-p, --max-pages`	`0`	Max pages to fetch (0 = unlimited).
`-d, --max-depth`	`0`	Max link depth to follow (0 = unlimited).
`--scope-prefix`	`N/A`	Only crawl paths starting with this string.
`--subdomains`	`false`	Include subdomains of the seed host.
`--exclude`	`N/A`	Path prefixes to ignore (can be used multiple times).
`--scroll`	`false`	Auto-scroll pages to trigger lazy loading.
`--workers`	`4`	Number of concurrent page renders.
`--no-robots`	`false`	Ignore `robots.txt` (use with caution).
`-f, --force`	`false`	Wipe existing mirror before starting.
`--chrome`	`N/A`	Path to the Chrome/Chromium executable.

For a full list of options, run kage clone --help.