Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, replay, mirroring, data scraping, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST
requests and responses, as well as most other HTTP
-level data.
Hoardy-Web
Web Extension?Hoardy-Web
is a browser extension (add-on) that passively captures and collects dumps of HTTP
requests and responses as you browse the web, and then archives them using one or more of the following methods:
HTTP
, like
hoardy-web serve
orhoardy-web-sas
;Downloads
directory.Hoardy-Web
produces dumps in a very simple, yet efficient, WRR
file format.
Moreover, Hoardy-Web
implements:
hoardy-web serve
;DOM
snapshots;HTTP
traffic, similar to browser’s own Network Monitor
(except browser-wise, i.e., it can capture HTTP
requests even when a page generates them when its tab/window closes; which a surprising number of websites does to prevent their web traffic from being inspected with Network Monitor
);Work offline
mode).In other words, this extension implements an in-browser half of your own personal private passive Wayback Machine that archives everything you see, including HTTP POST
requests and responses (e.g. answer pages of web search engines), as well as most other HTTP
-level data (AJAX
, JSON RPC
, etc).
For more information see project’s documentation and extension’s Help
page (also distributed with the extension itself, available via the “Help” button from its popup UI), especially the “Frequently Asked Questions” section there.
Also, note that:
Hoardy-Web
DOES NOT send any of your captured web browsing data anywhere, unless you explicitly configure it to do so.Hoardy-Web
DOES NOT send any telemetry anywhere.Hoardy-Web
.Hoardy-Web
was previously known as “Personal Private Passive Web Archive” aka pWebArc
.
See the gallery.
Install the extension from addons.mozilla.org. Then, in a desktop browser, press Extensions
toolbar button and pin Hoardy-Web
. (In mobile browsers there is are no customizable toolbars, unfortunately.)
Alternatively, download the latest Hoardy-Web-firefox-*.xpi
from Releases, and then follow “Install as an unsigned XPI” instructions below.
Alternatively, build it from source and then follow those same instructions.
about:config
, set xpinstall.signatures.required
to false
.about:addons
, click the gear button, select Install Add-on from File
, and select the XPI file in ./extension/dist
directory (or do File > Open File
from the menu and then select the XPI file, or drag-and-drop the XPI file into the browser window).Extensions
toolbar button and pin Hoardy-Web
.If you are building from source, this is a nice way to do development, since to reload the add-on after making a new build will require a single click.
about:debugging#/runtime/this-firefox
, click Load Temporary Add-on
button, and select ./extension/dist/Hoardy-Web-firefox-*/manifest.json
.about:addons
and enable Run in Private Windows
for Hoardy-Web
if your Firefox is running in Private-Windows-only mode.There are several ways you can do this. I recommend you read all of the following subsections first, and then decide what you want to actually try doing. Do not rush.
This is the simplest method that will work on all Chromium forks, but Hoardy-Web
will not get automatic updates. I.e., you will have to check the Releases page periodically, download, and install new releases manually to update.
Hoardy-Web-chromium-*.zip
from Releases.Hoardy-Web-chromium-*
inside for convenience.Extensions > Manage Extensions
in the menu of your browser and enable Developer mode
toggle.Load Unpacked
and select the directory the unpacking of the .zip
file produced. It should have manifest.json
file in it, just navigate to that directory select it and then press the Open
button (or navigate into that directory and then press the Open
button, that will work too).Extensions
toolbar button and pin Hoardy-Web
.Apparently, if you are using Google Chrome, you will get annoying warnings about side-loaded extensions. But you can also whitelist your extensions to prevent it, see the second answer of this stackoverflow question. For a Hoardy-Web
downloaded from Releases here, its Chromium extension ID is amklkmnfmgjmofkjifjgngdnajpbpefp
.
Alternatively, you can build it from source and then follow those same instructions above, except use ./dist/Hoardy-Web-chromium-*
directory after pressing Load Unpacked
button in the browser’s UI.
Similarly, there will be no automatic updates.
In case you need to whitelist your build, the Chromium extension ID of your build will be written to ./dist/manifest-chromium-id.txt
. That ID is derived from a public key, which is derived from a private key, which is generated by the gen-chromium-key.sh
script called by build.sh
script when you build the extension the very first time. The result then gets stored in ./private/chromium.key.pem
and reused between builds.
If your Chromium fork supports installation of third-party CRX files (not fetched from Chrome Web Store), you can do this:
chrome://flags
.#extension-mime-request-handling
flag and set it to Always prompt for install
.Hoardy-Web-chromium-*.crx
from Releases.Hoardy-Web
.There may or may not be automatic updates for Hoardy-Web
, depending of what your Chromium fork comes with. If it supports updates for third-party extensions, you will get updates, if it does not, you will not. The vanilla mainline Chromium comes without any such support. See below for how to fix it.
If your Chromium fork supports installation of third-party CRX files fetched manually:
Hoardy-Web-chromium-*.crx
from Releases.Downloads
folder into your browser’s window.Hoardy-Web
or just install it silently.The updates situation will be exactly the same as above.
Hoardy-Web
(and other indie extensions) automatically update on Chromium forks that do not support auto-updates for third-party extensionsInstall Hoardy-Web
using one of the above methods.
Install chromium-web-store
extension using one of the above methods.
It exists to help you to install extensions from Chrome Web Store and other similar Web Extensions repositories on Chromium forks that do not come with builtin support for Web Extension stores. More importantly, however, it can periodically check all your extensions that have an update_url
field set in their manifest.json
for updates and notify you about them. (Hoardy-Web
, of course, comes with update_url
set.)
So, the simplest way to do this on a most limited Chromium fork is to run
and then simply Load Unpacked
the ./chromium-web-store/src
directory in your Chromium fork.
Then you can:
Extensions > Manage Extensions
in the menu of your browser.Details
button on Chromium Web Store
extension.Extension options
(should be near the bottom) there.E.g., you might want to:
1440
,Enable Chrome Web Store Integration
,Congratulations, from now on Hoardy-Web
— or any other extension that has update_url
field set in its manifest.json
, regardless of its availability at Chrome Web Store — will get checked for updates periodically.
Hoardy-Web
(or another extension which has update_url
field set) via a CRX, then chromium-web-store
can even automatically update it for you.Hoardy-Web
(or another extension) via Load Unpacked
, you will have to manually re-install it from a new release on updates, but you will at least get notified about it updating without you needing to check manually.See chromium-web-store
’s README for more info and instructions, especially if you get CRX_REQUIRED_PROOF_MISSING
or Apps, extensions and user scripts cannot be added from this website
errors.
git clone
this repository.cd extension
.nix-shell ./default.nix
to get the exact build environment I use../build.sh clean firefox-mv2 chromium-mv2
from this directory.dist
directory.about:debugging#/runtime/this-firefox
and press extension’s Inspect
button.Extensions > Manage Extensions
and press Inspect views
link after the extension’s ID.