software/hoardy-web/./extension/README.md

Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, mirroring, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data.

Files

Raw Source

What is Hoardy-Web Web Extension?

Hoardy-Web is a Web Extension browser add-on that passively captures and collects dumps of HTTP requests and responses (see WRR file format) as you browse the web, and then archives them to your Downloads directory, your own private archiving server (like hoardy-web-sas), or your browser’s local storage.

Practically speaking, you install this and just browse the web normally while Hoardy-Web passively, in background, captures and archives web pages you visit for later offline viewing, mirroring, and/or indexing. Hoardy-Web has a lot of configuration options to help you tweak what should or should not be archived and a very low memory footprint, keeping you browsing experience snappy even on ancient hardware (unless explicitly configured otherwise to, e.g., minimize writes to disk instead).

In other words, this extension implements an in-browser half of your own personal private passive Wayback Machine that archives everything you see, including HTTP POST requests and responses (e.g. answer pages of web search engines), as well as most other HTTP-level data (AJAX/JSON RPC/etc).

By default, this extension will save all captured data into browser’s local storage, so it can be used standalone. To view your archived data, however, you will need to install at the very least the accompanying hoardy-web CLI tool (also there).

If you do not care about archival, you can also use this extension to log and later inspect HTTP traffic generated by various sleazy websites, even when they generate said web traffic on events that can not normally be inspected with browser’s own Network Monitor (e.g. when a page generates HTTP requests when its window closes).

Hoardy-Web is most similar to archiveweb.page and DownloadNet projects, but it works both on Firefox- and Chromium-based browsers, and it follows its own design philosophy, which makes the experience of using it is very different.

Hoardy-Web was previously known as “Personal Private Passive Web Archive” aka pWebArc.

Screenshots

See the gallery.

What does it do exactly? I have questions.

Installation

On Firefox-based browsers (Firefox, Tor Browser, LibreWolf, Fenix aka Firefox for Android, Fennec, Mull, etc)

Install as an unsigned XPI

Install as a temporary add-on

If you are building from source, this is a nice way to do development, since to reload the add-on after making a new build will require a single click.

On Chromium-based browsers (Chromium, Google Chrome, Ungoogled Chromium, Brave, etc)

Hoardy-Web isn’t on Chrome Web Store because Hoardy-Web appears to violate its “Terms of Use”. Specifically, the “enables the unauthorized download of streaming content or media” clause. See higher-level README for more info and discussion.

So, installation on Chromium-based browsers (aka “Chromium forks”) requires a little bit of work. There are several ways you can do it. I recommend you read all of the following subsections first, and then decide what you want to actually try doing. Do not rush.

Install as an unpacked extension

This is the simplest method that will work on all Chromium forks, but Hoardy-Web will not get automatic updates. I.e., you will have to check the Releases page periodically, download, and install new releases manually to update.

Apparently, if you are using Google Chrome, you will get annoying warnings about side-loaded extensions. But you can also whitelist your extensions to prevent it, see the second answer of this stackoverflow question. For a Hoardy-Web downloaded from Releases here, its Chromium extension ID is amklkmnfmgjmofkjifjgngdnajpbpefp.

Install as an unpacked extension built from source

Alternatively, you can build it from source and then follow those same instructions above, except use ./dist/Hoardy-Web-chromium-* directory after pressing Load Unpacked button in the browser’s UI.

Similarly, there will be no automatic updates.

In case you need to whitelist your build, the Chromium extension ID of your build will be written to ./dist/manifest-chromium-id.txt. That ID is derived from a public key, which is derived from a private key, which is generated by the gen-chromium-key.sh script called by build.sh script when you build the extension the very first time. The result then gets stored in ./private/chromium.key.pem and reused between builds.

Install a CRX directly

If your Chromium fork supports installation of third-party CRX files (not fetched from Chrome Web Store), you can do this:

There may or may not be automatic updates for Hoardy-Web, depending of what your Chromium fork comes with. If it supports updates for third-party extensions, you will get updates, if it does not, you will not. The vanilla mainline Chromium comes without any such support. See below for how to fix it.

Install a CRX via drag-and-drop

If your Chromium fork supports installation of third-party CRX files fetched manually:

The updates situation will be exactly the same as above.

How to make Hoardy-Web (and other indie extensions) automatically update on Chromium forks that do not support auto-updates for third-party extensions

Then you can:

E.g., you might want to:

Congratulations, from now on Hoardy-Web — or any other extension that has update_url field set in its manifest.json, regardless of its availability at Chrome Web Store — will get checked for updates periodically.

See chromium-web-store’s README for more info and instructions, especially if you get CRX_REQUIRED_PROOF_MISSING or Apps, extensions and user scripts cannot be added from this website errors.

Development

Build it from source

Debugging

On Firefox-based browsers

On Chromium-based browsers