software/hoardy-web/./extension/page/help.org

Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, replay, mirroring, data scraping, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data.

Files

Raw Source

Don’t read this document outside of Hoardy-Web

If you are reading this on GitHub, it’s highly recommended you switch to the version hosted on the author’s web site instead. This pages is written in org-mode markup language and it uses a lot of its advanced features, which GitHub does not implement, so this page is almost unreadable on GitHub.

Moreover, if you already have Hoardy-Web browser extension installed, then, instead of reading this document on GitHub or from author’s web site, it is highly recommended you refer to this document by clicking the “Help” button in the extension’s own popup UI. Doing that will make this document interactive: the settings popup will be displayed on the right on this page and hovering over or clicking references to configuration options on this page will highlight them in the settings popup. It will also have a nicely formatted “Table of Contents” section there. See screenshots if you want to get an idea of how it will look.

How to read this document

When you open this document by clicking the Help button in Hoardy-Web browser extension’s UI, this page has two parts: this help text, and an iframe with a completely unrolled popup UI in it. The whole page will switch between single- and two-column layouts depending on available viewport width (which depends on device width and zoom level). In single-column layout the popup UI is placed after the end of the help text. In two-column layout they are placed side-by-side.

In both layouts:

In cases when clicking on a link scrolls a page around or navigates to another page, pressing the “Back” button of your browser will get you back to the exact link you clicked and then highlight it, making it easy to get back to reading from the exact place you left off.

Go forth and try it by clicking one or more of the above links.

The above conventions also apply on all other internal pages of Hoardy-Web browser extension and pages that it comes bundled with, e.g., Internal State and Changelog pages.

Introduction

The Hoardy-Web software project has many parts and pieces, all of which are documented separately. This page documents the Hoardy-Web web browser extension (add-on) and only ever mentions the other parts and pieces in passing.

In this document, unless specified otherwise, “Hoardy-Web” refers to the Hoardy-Web extension, not the other parts of the Hoardy-Web software project.

The main purpose of Hoardy-Web is to help you passively, in background, capture and archive HTTP requests and responses your web browser does as you browse the Web. Which is to say, the main function of Hoardy-Web is to attach to your browser’s runtime and track progress of your browser’s HTTP requests and responses, capturing their HTTP headers and bodies at appropriate times in browser’s request and response processing pipeline, dumping the results of those captures to disk as they finish loading from the network.

However, Hoardy-Web does much more than that, implementing a bunch of other features relevant for web archival and replay.

Note that in Hoardy-Web you can check the description of each status line, button, and/or configuration option by hovering over it or by pressing a ? button that comes after it. However, many of those descriptions might make little sense without you first learning the general operation principles of Hoardy-Web and some of its quirks. This document exists to explain all of those.

Glossary

This document’s structure

The rest of this document is organized as follows:

Configs, internal pages, and actions

Hoardy-Web implements hierarchical per-browser/per-window/per-tab/per-source configurations. Which is to say that most capture-related settings of Hoardy-Web can be configured for newly created windows, for new tabs created in each window, for each tab specifically, for new tabs spawned from each specific tab, for each navigation to a certain type of URL (at the moment this is limited to a pre-defined set of URL types, but the generalized machinery will be implemented soon-ish), for background requests, and for requests generated by extensions.

For example, whether Hoardy-Web will track a given HTTP request depends on the per-whatever Track new requests toggles:

The same logic applies to all other toggles that have per-browser (glob), per-window (win), per-tab (tab, chld), and/or per-source (bg, ext) instances.

However, importantly, note that New root windows/tabs (glob) options only get applied to new root windows/tabs. If you drag a tab into a new window, that new window’s per-window settings will be copied from that tab’s old window per-window settings instead of them being inherited from the (glob) settings. Similarly, if you open a link in a new window, e.g. via the context menu, its per-window settings will be copied from the opener window’s per-window settings too.

In general, the elements in the popup are sorted from less specific to more specific, with more specific settings taking precedence, when they are applicable.

For example, Navigations to special URLs options, when applicable, override the corresponding per-tab settings on each navigation to a matching URL.

Finally, Hoardy-Web applies the same idea to most of its internal actions and pages. Which is to say, most of those also have per-browser (glob), per-window (win), and per-tab (tab) instances.

For example, the replay-re-navigation (glob, win, tab) and limbo-manipulation (glob, win, tab) buttons come in such triples. Similarly, Hoardy-Web’s Internal State log pages (glob, win, tab) can be narrowed to data applicable to the whole browser, a specific window, or a specific tab, which also makes Hoardy-Web’s internal actions bound to the buttons there apply to the data displayed there.

Capture and archival

Step 1: Tracking

As noted above, Hoardy-Web attaches to your browser’s runtime and tracks progress of HTTP requests and their responses. Whether Hoardy-Web will track a given HTTP request depends on the Track new requests toggles (glob, win, tab, chld, bg, ext) mentioned above.

Note, however, that disabling any of these toggles does not stop tracking of already initiated requests, it only stops new requests controlled by that toggle from being tracked.

The State Diagram

In Hoardy-Web, reqres change their internal states according to the following state diagram:

(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body received)
   |                           |                              |             |
   |                           v                              v             v
   |                     (no_response)                   (incomplete)   (complete)
   |                           |                              |             |
   |                           \                              |             |
   |\---> (canceled) ----\      \                             |             |
   |                      \      \                            \             |
   |\-> (incomplete_fc) ---\      \                            \            v
   |                        >------>---------------------------->-----> (finished)
   |\--> (complete_fc) ----/                                             /  |
   |                      /                                             /   |
   \----> (snapshot) ----/       /- (collected) <--------- (picked) <--/    |
                                /       ^                      |            |
 (stashToLS?) <----------------/        |                      v            v
      |                                 \- (in_limbo) <- (stashToLS?) <- (dropped)
      v                                           |                         |
  (queued) <---- (unarchived) <- (stashToLS?)     |                         |
      |                               ^           \-----> (discarded) <-----/
   /-\|          /---> {U} -----------/                        ^
   ^  |          |      \                                      |
   |  |          |       \--> (forgotten) ((-> (saved)))       |
   |  |          |                                             |
 {RE} |          v                  {{!saving && !submitting}} |
      |\----> (saveAs) -> (exported) -------------------------/|
      |                       /                                |
      | {U} <-\    /---------/                                 |
      |        v   v                        {{!saving}}        |
      |\----> (srvIO) -> (submitted) -------------------------/|
      |                     /                                  |
      |                    /              {{delete}}           |
      | {U} <-\     /-----/     /-----------------------------/|
      |        \   /           /                               |
      |        v   v          /       {{re-archive}}           |
      |\---> (saveToLS) -> (saved) -------------------> {RE}   |
      |                                                        |
      \--------------------------------------------------------/

The top half of the State Diagram

A new reqres proceeds through the following networking states:

Glossary

The following definitions won’t be referenced below, so you can skip them.

(“No! Let me see!”)

The bottom half of the State Diagram

In principle, when a reqres reaches its finished state, the primary objective of Hoardy-Web with respect to that reqres is complete, so that reqres could be dumped to disk and forgotten about.

Unfortunately for Hoardy-Web, browsers do not allow their extensions to simply write files to user’s file system, and all existing browser APIs that do allow for persistence to disk in some way all have different limitations. Also, it is quite useful to have more states after the finished state to improve the UI and allow for various conditional workflows.

Which is why Hoardy-Web has quite a lot of states after the finished state on the above diagram. To give a simple overview of those, after Hoardy-Web produces a finished reqres, it proceeds as follows:

The following subsections discuss these things in detail.

Step 2: Classification

When a reqres reaches the finished state of the diagram, it gets classified by the computations described below.

Partial, buggy, and incomplete reqres

Sometimes networking issues and/or browser bugs prevent complete collection of (parts of) reqres data or metadata. To provide indication of this, Hoardy-Web keeps track of such things and marks such reqres with flags that indicate that the reqres in question has

Picked and Dropped reqres

Each new finished reqres advances either to the picked or the dropped state. Which of those two states gets selected is decided based on reqres’ final networking state, its partial, buggy, and incomplete flags discussed above, and the Pick reqres for archival when they finish options:

Problematic reqres

A reqres can also be classified as problematic, which shall be discussed in a much later section since the problematic flag is completely orthogonal to the current archiving-related discussion. But, technically speaking, this happens at this point in the Hoardy-Web’s processing pipeline too.

Glossary

Step 3: Collection, Discarding, and Logging

On exit from the finished state of the diagram, after the classification performed there, each reqres gets split into

(Since those tuples can be reconstructed back into the original reqres, the following will continue to refer to them as if nothing has changed when the fact they are now being internally represented by those tuples is not relevant.)

On entering collected or discarded state, loggable metadata of each reqres is copied into the recent reqres log (glob, win, tab) and is kept there until the size of the log reaches this many elements, at which point the older elements of the log start being elided automatically.

You can also ask Hoardy-Web to forget a subset or all of those log elements manually by opening an appropriate log page (glob, win, tab), setting some filters to produce a desired subset, and then pressing the Forget all button.

However, note that reqres still marked as problematic, can’t be automatically elided or manually dropped from the logs until they get unmarked.

Normal mode

Thus, normally:

Limbo mode

However, sometimes you might want to actually look at a web page before deciding if you want to archive it or not. The naive way to do it would be to load a page with capture disabled first, look at it, and then, if you want to save it, enable it, and reload the page again with browser’s cache disabled via Control+F5 (and it has to be Control+F5, not just F5, because otherwise some URLs, on Firefox, might produce reqres in incomplete_fc state, and on Chromium, their re-fetching could be silently skipped).

Obviously, this is both annoying and will force you to fetch everything twice.

Which is why Hoardy-Web implements limbo mode. With one of the limbo mode options (glob, win, tab, chld, bg, ext) enabled, Hoardy-Web will instead capture everything as normal, but then, instead of sending the corresponding newly captured reqres to collected or discarded state, it will put them into in_limbo state where they would linger until

A picked reqres will be put into in_limbo state when its corresponding Pick into limbo setting (glob, win, tab, chld, bg, ext) is enabled. Similarly, a dropped reqres will be put into in_limbo state when its corresponding Drop into limbo setting (glob, win, tab, chld, bg, ext) is enabled. (The latter mainly exist for debugging.)

In combination with Closed tabs options you can use the in_limbo state to implement the following workflows:

If this option is enabled and there are more than this number reqres in_limbo or the total size of all dumps in_limbo is more than this size (in MiB), Hoardy-Web will complain to remind you to collect or discard some of them.

Glossary

(Optional) Step 4: Stashing

In most cases you would probably want to prevent loss of successfully captured but yet unarchived reqres when

before you collected or discarded everything from in_limbo or Hoardy-Web has successfully archived everything from its archiving queue.

This is why Hoardy-Web implements the optional stashToLS? states of the diagram and a special stashed reqres status. The stashed status is, essentially, a flag that says this reqres was temporarily backed up into browser’s local storage. This process can be controlled using the following options:

Moreover, the following section will discuss how Hoardy-Web will try stashing unarchived reqres into browser’s local storage too.

Note however, that even with stashing enabled Hoardy-Web will skip disk IO whenever possible: e.g., if both Archive 'collected' reqres and Submit dumps via 'HTTP' options discussed below are enabled, Hoardy-Web will first try to archive each new collected reqres straight from memory to the archiving server and only if that process fails will it attempt stashing them to local storage instead.

Meaning that

The above also implies that, technically, stashing is not a silver bullet against data loss. To try and make it such would mean unconditional immediate stashing of all captured data, which would waste a lot of disk IO on most Hoardy-Web configurations.

When both Stash reqres into local storage option and Archive 'collected' reqres option are disabled, then, after a new reqres gets queued, Hoardy-Web will generate a new notification complaining about it, unless that option is disabled too.

You can also forcefully stash all currently queued, in_limbo, and unarchived reqres by pressing this button. It stashes everything immediately and unconditionally, ignoring all other stashing settings. When reloading the extension via the Reload button or via Auto-reload on updates option, this action will be run automatically.

If this option is enabled and some reqres fail to stash, a new notification will be generated.

Glossary

Step 5: Archival

After a reqres reaches the queued state of the diagram, Hoardy-Web will attempt to persist it to disk.

When Archive 'collected' reqres toggle is enabled, Hoardy-Web will pop queued reqres from the archival queue one by one and then perform one or more of the following (in order they are listed):

When stashing and/or archiving to local storage, the dump’s get compressed if this option is enabled.

You can enable more than one archival method at the same time. For a given loggable, Hoardy-Web will remember and skip previously successful archival methods if the loggable ever returns to the archival pipeline, which can happen when

Buckets (a.k.a. collections)

Sometimes you might want to semi-automatically split your collected archives into separate disjoint sets. Say, for instance, you want to split out archives generated by a select tab into a separate set you plan to share with somebody else. In Hoardy-Web such sets are called buckets. WARC-based tools sometimes call these “collections” instead.

To implement this, for each reqres in the archival queue, Hoardy-Web takes a bucket value from the corresponding “Bucket” setting (glob, win, tab, chld, bg, ext).

Evaluation of bucket is done just before each archival attempt, so if the queue is not yet empty, and you disable Archive 'collected' reqres, edit some of the “Bucket” settings, and enable it again, Hoardy-Web will start using the new setting immediately.

When exporting via saveAs, bucket value will be used in the file name of the generated fake-Download WRR bundle file (.wrrb) and the dumps will be split into separate fake-Download files by said bucket. I.e., internally, the WRR bundle discussed above is actually a set of per-bucket WRR bundle’s.

When submitting to an HTTP server, Hoardy-Web will specify bucket as a query parameter (named “profile”, for historical reasons) to each HTTP POST request, which will cause the configured archiving server to put those WRR files into a directory with the same name.

When stashing or saving to local storage, Hoardy-Web will record the value of bucket into each loggable before saving data to disk. If you restart your browser, thus starting a new Hoardy-Web session, Hoardy-Web will use the old stashed/saved bucket values for all new attempted archivals of old reqres generated by previous sessions.

So, for example, if you want to share a subset of your captures, you can

Handling of failures

As noted above, if any of the archival methods fail, the reqres in question will be moved into the unarchived state.

Submissions via HTTP of reqres that are unarchived because of networking issues will be retried automatically every 60 seconds. Archivals of reqres rejected by the archiving server or those that failed to be saved to browser’s local storage will not be retried automatically as those usually happen when there is no space left on the device you are archiving to.

Additionally, when a submission via HTTP and/or writes to browser’s local storage fail, Hoardy-Web will assume that the following archivals/writes will fail with the same error too. Usually, this paranoid behaviour is both safer and more efficient, but it can prevent some extra reqres from archiving when only a small subset of all queued reqres are being rejected by the server or fail to be written to local storage. Thus, Hoardy-Web allows you to disable this behaviour by toggling this and/or that options. Though, when no archiving bugs are apparent, you should probably keep both of them enabled.

You can retry all archiving failures by pressing one of this or that buttons. You can also use them to nudge the archiving sub-process awake if something archival-related appears to be stuck.

If this option is enabled and a new reqres recently moved to the unarchived state, a new notification will be generated. If this option is enabled, a new notification will be generated when the archival queue gets empty the very first time or after previous failures.

Glossary

(Optional) Step 6: Re-archival

Hoardy-Web allows reqres saved in browser’s local storage to be re-archived a second, third, fourth, etc time. This feature uses the same archiving pipeline as above with three major differences:

Re-archival can be most useful in following use cases:

Controlling what gets re-archived

The simplest way to re-archive old data is to configure the desired re-archival methods and then press Re-archive all button. This will simply re-archive everything stored in local storage unconditionally.

Since archival and re-archival use the same archiving pipeline, similarly to unarchived reqres, re-archival also remembers if a given reqres was previously (re-)archived using each archival method. Thus, enabling a re-archival method and then pressing Re-archive new button/shortcut (shortcut key of rearchiveAdjunctSaved) will skip re-archival of reqres that were previously archived or re-archived using that same method.

I.e., you can just configure the desired re-archival methods and then press Re-archive new button/shortcut (shortcut key of rearchiveAdjunctSaved) from time to time, with each new press re-archiving a new batch.

If you want more control over what gets re-archived, you can also

Deletion of successfully (re-)archived reqres

In use cases

you can easily delete your previously successfully archived or re-archived reqres from your local storage by

However, the first two options should be used with caution:

In short, re-archival actions with deletion set should be used with lots of care. This is why the rewrite and deletion options and their equivalents on Saved in Local Storage page are ephemeral and will be reset the next time you reopen the popup or that page.

Additionally, note that you can press re-archive and deletion buttons one after the other immediately, their actions will get queued for execution properly and won’t interfere with each other. Thus, if you are re-archiving via submission via HTTP it’s safe to mash them one after the other, you don’t need to sit there and wait for one action to finish before pressing the other button, since submission over HTTP reports errors properly.

However, note that re-archival by submission via HTTP with deletion enabled is safe to use too. On-the-fly deletion is much more efficient compared to re-archival step followed by a separate deletion step because updated reqres’ archival flags need not be written back into local storage. Thus, since submission via HTTP does report its errors properly, successfully re-archived reqres can be immediately and safely deleted from local storage in that case, which would save you a lot of time and disk writes if you are re-archiving a lot of stuff all at once.

Additional features

Tracking of problematic reqres

Conventional web browsers provide no explicit indication when a part of a web page fails to load properly. Apparently, you are expected to actually look at the page with your eyes, notice something looking broken, and reload it manually if so.

Obviously, this can be quite inconvenient when you want to be sure that the whole page with all of its resources was archived. Especially when parts of a dynamically loaded page might simply silently fail to be rendered by associated JavaScript because some of the HTTP requests that JavaScript did in background failed, or, on a static web page, layout and CSS might have made some of the incompletely loaded parts of the page invisible (by design or by accident).

So, to provide an explicit indicator for such situations, Hoardy-Web keeps track of reqres that fail to load properly and, in the classification step, marks them with a problematic flag which influences

In other words, the problematic flag is a UI indicator, it does not influence archival in any way.

What gets marked as problematic is controlled by Mark reqres as 'problematic' when they finish options. By default, HTTP requests that failed to get a response, those that were fetched with buggy metadata or have incomplete response bodies, and those for which the browser reported potentially problematic errors but then Hoardy-Web picked them anyway, will be marked as problematic.

Potentially problematic errors are errors like

but not errors like

In principle, Hoardy-Web could have been designed to never record the errors of the latter category in the first place, thus simplifying the above bit, but Hoardy-Web is designed to follow the philosophy of “collect everything as browser gives it, as raw as possible”. The raw error strings reported by the browser for each reqres can be seen in the logs (glob, win, tab).

When that option is not disabled, new notifications will be produced each time a new problematic reqres appears.

Glossary

Automatic unmarking of problematic reqres

In some situations some of the problematic flags on some of your reqres might become irrelevant as you fetch more data.

For example, say, you load a tab, some of its images fail to load completely and get marked as problematic, so you re-load that tab again, and then all reqres that failed to be fetched the first time get fetched properly the second time. Do you still care that some of those old reqres failed to fetch properly the first time? Probably not.

But to be absolutely pedantic in such a situation, you would actually need to carefully check that all of your old problematic reqres actually got re-fetched properly, just in case the browser decided to skip fetching some of them, which could happen with modern HTML lazy loading features or if the page in question is just awfully dynamic, and only then stop caring about them. Obviously, actually doing this manually each time would be incredibly annoying.

Which is why Hoardy-Web provides a bunch of configuration options that automatically unmark old problematic reqres in various situations. By default, those options are set in a way that produces least annoying and least surprising workflow, but you might want to tweak them if you need that process to be stricter.

In particular, note that the default value of Auto-unmark problematic reqres … regardless of its in_limbo state option produces slightly imprecise results when combined with picking into limbo (glob, win, tab, chld, bg, ext). Thus, if you are feeling pedantic, you might want to disable that option. Though, if you do that, the resulting behaviour of Hoardy-Web will probably feel rather counter-intuitive, at least at first.

Additionally, when that option is set to its right-most state, new notifications will be generated each time an old problematic reqres gets automatically unmarked.

Manual handling of problematic reqres

In most use cases, Hoardy-Web is entirely capable of automatically maintaining a relevant set of still problematic reqres, helping you figure out what still needs to be (re-)fetched to completely archive a given web page properly. However, in some situations, you might also want to manually unmark some of your problematic reqres. To do that you can:

I don’t care about any of this and notifications about problematic reqres annoy me!

If you don’t care about the problematic flag in a select tab and those notifications annoy you, you can disable this per-tab option. If such notification annoy you in general, you can disable the global option instead.

Note, however, that you probably shouldn’t be disabling too many of Mark reqres as 'problematic' when they finish options. This way, even with notifications disabled, you would still be to able see the correct number of problematic reqres in extension’s toolbar button’s badge, popup (glob, win, tab), and the logs (glob, win, tab).

Allowing a tab’s content to settle

Hoardy-Web tracks both the number of in_flight reqres and the timestamp of the last network-fetched finished reqres for each tab and uses them to track when a tab’s content settles.

Which is to say, Hoardy-Web can delay some actions in a tab (currently, replays and snapshots) until all in-flight reqres in that tab become finished, then wait an additional configurable delay (glob, win, tab, chld) to allow that page’s JavaScript to process its fetched data (and, possibly, update its DOM and/or generate new requests), and then run the actual action only if the tab generates no new requests in the meantime.

If the tab does generate new requests, the process will repeat from the beginning up to configured number of times (glob, win, tab, chld) and produce an error notification if that number of retries gets exceeded.

Also, note that the network-fetched finished reqres bit above is important as reqres produced by the DOM snapshotting machinery are ignored by the tab settling machinery. If they were not, a DOM snapshot followed by a replay or another DOM snapshot would have to wait for the page to re-settle again, which would be annoying.

When a Hoardy-Web action affects multiple tabs, each tab settles, or fails to settle, independently from the others.

Replay of previously archived data

If you use hoardy-web serve or something similarly powerful as your archiving server and this option is not disabled, Hoardy-Web will enable its integration with replay over HTTP which enables the following Hoardy-Web features.

Manual replays

With server replays available and enabled, pressing one of the replay buttons (glob, win, tab) will make Hoardy-Web wait for the affected tabs to settle, collect all of their in-limbo reqres, and then re-navigate them to their replay pages. For the former two buttons, a tab can be excluded from the affected set by untoggling this per-tab setting.

Auto-replay mode

Additionally, Hoardy-Web implements per-tab auto-replay mode (glob, win, tab, chld), which automatically redirects all navigations in a tab to their replay URLs.

This mode of operation is most useful when you want to open a replay of a previously bookmarked URL without editing it manually, nor first loading that page from the Internet and then replaying it the normal way (glob, win, tab).

If the archiving server does not support replay or when this option is disabled, auto-replay mode will cancel those requests and generate error notifications instead.

Currently auto-replay mode only works for top-level navigation requests and performs redirects regardless of your archiving server having an archive of the URL in question. (This will be improved in the future.)

Capture of DOM (Document Object Model) snapshots

In principle, Hoardy-Web can capture everything your browser fetches from the network via HTTP as you browse the web. So, web pages that use simple JavaScript code to generate their content from those HTTP requests and/or use simple JavaScript to improve their UI will work fine when you start replaying them via hoardy-web serve over HTTP, via hoardy-web mirror statically, or some such.

However, unfortunately, in the most general case, “from scratch” replay of pages dynamically generated via JavaScript is not guaranteed. For example, consider a web page with a JavaScript code that generates a random number, then queries a remote server with that number, and then renders the result somehow. Obviously, such a web page can not be replayed “from scratch” since it will generate a new random number and your archive probably won’t have the corresponding server’s response for it.

Additionally, more and more dynamic web pages fetch data using WebSockets protocol, which Hoardy-Web can’t capture at the moment.

Moreover, even when a web page can be replayed, sometimes it’s desirable to have a simple HTML dump of its contents, e.g., for full-text indexing purposes.

Which is why, to support the above and related use cases, Hoardy-Web implements DOM snapshots.

Pressing one of DOM snapshot buttons (glob, win, tab) in the popup will capture snapshots of raw HTML’s or XML’s for each frame contained in each affected tab after said tab settles. For the former two buttons, a tab can be excluded from the affected set by untoggling this per-tab setting.

WRR-wise DOM snapshots are 200 OK responses, but with protocol set to SNAPSHOT and method set to DOM.

Work offline mode

Sometimes, you might want to block a select tab from performing new HTTP requests.

Normally, when you replay your archived pages via hoardy-web serve, hoardy-web mirror, or some such, you would probably want the resulting generated pages to remap all their requisites (CSS, images, etc) to point to their local replay URLs and files. (Which is how hoardy-web tool does it by default. Though, it is configurable.)

Unfortunately, HTML5 specification is quite large, complex, and it gets updated all the time. Additionally, hoardy-web can have bugs in its remapping code. Moreover, interactions between remapped pages and some browser extensions can sometimes break things too. In other words, some of those replayed pages could end up referencing Internet resources when they should not.

In a similar vein, imagine a situation where, while looking at your logs (glob, win, tab), you discover that a web page originating from a certain domain is making an awful lot of tracking HTTP requests:

Now, say that, after considering the above, you discover that you dislike the idea of “TVs that watch you back” from Orwell’s “1984” greatly, and you would rather ensure that

In some cases you might even feel paranoid enough to want to prevent your browser from opening non-remapped jump-links (a href) on replay pages, even when you explicitly click them (by accident).

Desktop versions of Firefox-based browsers have a File > Work Offline option that can solve most of this, but it disables all new requests browser-wise, which is quite inconvenient and error-prone if you want to keep some of your tabs offline while not restricting others, and it will break replay over HTTP with hoardy-web serve. Chromium-based browsers do not appear to have such a feature at all.

To solve this issue, Hoardy-Web implements its own granular work offline mode which can be controlled via per-whatever Work offline toggles (glob, win, tab, chld, bg, ext).

Unlike the File > Work Offline option of Firefox, enabling any of these toggles:

Finally, there is also a bunch of options that automatically enable work offline mode in tabs with various classes of URLs. By default, work offline mode is enabled for file: and replay URLs to stop any pages generated by hoardy-web serve and hoardy-web mirror from accessing the Internet.

Switching between and highlighting/selecting tabs

Note that if you spawn a bunch of tabs to fetch stuff in background with middle mouse click or some such and some of those fetches fail, you can easily loose track of where those newly generated problematic reqres you are seeing are coming from. Similarly, if you have picking into limbo enabled in a bunch of tabs you can easily loose track of which tabs have yet un-collected in_limbo reqres.

Thus, naively, if you see Hoardy-Web toolbar button’s badge show you a non-zero number and you don’t know where it’s coming from, you can:

Note, however, that both of the above methods are annoying in different ways.

Which is why Hoardy-Web implements a bunch of buttons and keyboard shortcuts that can switch between and/or highlight/select browser tabs in ways that take problematic, in_limbo, and last finished reqres of each tab into account. See their help strings for detailed descriptions of what those actions do.

Most notably, by using those buttons/shortcuts, the above use cases can be solved by pressing shortcut short of smartSwitchTabsLatest (shortcut key of smartSwitchTabsLatest), looking at the tab in question, pressing shortcut short of unmarkTabProblematic (shortcut key of unmarkTabProblematic), shortcut short of collectTabInLimbo (shortcut key of collectTabInLimbo), shortcut short of discardTabInLimbo (shortcut key of discardTabInLimbo), and/or shortcut short of closeTabThenDiscardInLimbo (shortcut key of closeTabThenDiscardInLimbo) buttons/shortcuts, and repeating until all problematic and/or in_limbo reqres dealt with.

Moreover, for the in_limbo use case, you can hit shortcut short of highlightTabsLatest (shortcut key of highlightTabsLatest), then drag-and-drop all those tabs to a new window, walk through them pressing shortcut short of discardTabInLimbo (shortcut key of discardTabInLimbo) or shortcut short of closeTabThenDiscardInLimbo (shortcut key of closeTabThenDiscardInLimbo) buttons/shortcuts in all the tabs you don’t want, and then press shortcut short of collectWindowInLimbo (shortcut key of collectWindowInLimbo) to collect everything else. Which you can then follow up by shortcut short of replayWindow (shortcut key of replayWindow) to confirm the results.

Keyboard shortcuts

Hoardy-Web implements the following keyboard shortcuts:

Description Currently Default

The above table implements some rudimentary shortcut editing on the browsers that support it. Outside of that, you can view and edit all of your keyboard shortcuts via:

Also, note a very relevant “FAQ” item below.

Context menu actions

Hoardy-Web implements context menu actions to:

Workflows

Using Hoardy-Web with Fenix (Firefox on Android)

Since Export dumps via 'saveAs' option and its re-archival variant do not work under Firefox on Android, you will have to use Submit dumps via 'HTTP' and/or its re-archival variant to extract your archives from Hoardy-Web. There are several ways you can do this:

Using Hoardy-Web with Tor Browser

When running Hoardy-Web under Tor Browser you’ll probably want to configure it in such a way so that all of the machinery of Hoardy-Web is completely invisible to web pages running under your Tor Browser, to prevent fingerprinting.

Thus, you can simply use Hoardy-Web extension configured to Save reqres into local storage, which is the default, and then export your dumps manually at the end of a browsing session via re-archival actions, which would take a couple of clicks, as discussed above.

Or, alternatively, you can switch to using Export dumps via 'saveAs' by default instead, disabling the other archiving methods. I expect this to work fine for 99.99% of the users 99.99% of the time, but, technically speaking, this is unsafe. Also, by default, browser’s UI will be slightly annoying, since Hoardy-Web will be generating new “Downloads” all the time, but that issue can be fixed with a small about:config change.

Using either of the above methods won’t allow you to replay archived captures, though.

Replay on Tor Browser

In theory, running hoardy-web serve listening on a loopback IP address should prevent web pages from accessing it, since the browsers disallow cross-origin requests from non-localhost domains to localhost, thus making the normal Submit dumps via 'HTTP' mode with replay setup quite viable. However, Tor Browser is configured to proxy everything via the TOR network by default, so you need to configure it to exclude the requests to hoardy-web serve from being proxied.

A slightly more paranoid than normal way to do this is as follows:

Why? When using Tor Browser, you probably don’t want to use 127.0.0.1 and 127.0.1.1 as those are normal loopback IP addresses used by most things, and you probably don’t want to allow any JavaScript code running in Tor Browser to (potentially, if there are any bugs) access to those. Yes, if there are any bugs in the cross-domain check code, with this setup a nefarious page’s JavaScript could discover you are using Hoardy-Web (and then, in the worst case, access all of your archived captures and DOS your system by flooding your disk with garbage dumps), but it won’t be able to touch the rest of your stuff listening on your other loopback addresses.

So, while this setup is not super-secure if your Tor Browser allows web pages to run arbitrary JavaScript (in which case, let’s be honest, no setup is secure), with JavaScript always disabled, to me, it looks like a completely reasonable thing to do.

Error messages and codes

Error messages, as seen in generated notifications

Errors recorded in reqres, as seen in the logs

… at (glob, win, tab).

Most error codes are produced by attaching one of the following prefixes to the raw error code given by the browser:

In particular, webRequest::NS_ prefix on Firefox, and webRequest::net:: and debugger::net:: prefixes on Chromium signify various issues produced by the networking stacks of those browsers. For instance:

The exception to the above rule of keeping everything as raw as possible are webRequest::capture:: and debugger::capture:: prefixes which signify various errors produced by Hoardy-Web itself in its webRequest- or debugger-handling code, respectively. In particular:

Quirks and Bugs

If you are reading this page outside of the extension’s UI be sure to read the very top of this page first.

Known Hoardy-Web’s own issues

Known issues that are consequences of issues of all supported browsers

Known issues that are consequences of issues of Firefox-based desktop browsers: Firefox, Tor Browser, LibreWolf, etc

Known issues that are consequences of issues of Firefox-based mobile browsers: Fenix (Firefox for Android), Fennec, Mull, etc

All of the above apply, moreover:

Known issues that are consequences of issues of Chromium-based desktop browsers: Chromium, Chrome, etc

On Chromium-based browsers, there is no way to get HTTP response data without attaching Chromium’s debugger to a tab from which a request originates from. This makes things a bit tricky, for instance:

Moreover, Chromium has the following long-standing issues/bugs making things difficult:

Frequently Asked Questions

If you are reading this page outside of the extension’s UI be sure to read the very top of this page first.

Respecting you

Does Hoardy-Web send any of my captured web browsing data anywhere?

No.

Hoardy-Web only ever sends your data to the configured Server URL when the Submit dumps via 'HTTP' option is enabled.

Nowhere else. Never else.

Does Hoardy-Web collect and send any telemetry anywhere?

No.

For your convenience, Hoardy-Web saves some global stats across restarts (e.g., the Collected, Discarded, Picked, and Dropped lines).

However, none of those are ever sent anywhere and you can reset them at any time.

Will the answers to the above two questions ever change in a future version of Hoardy-Web?

No. I (the author) hate non-consensual data collection.

In fact, as you might have noticed, Hoardy-Web, unlike most other browser extensions, is almost trivial to reproducible-build from source on a POSIX-compliant system with a Nix package manager installed, and it has a privately operated source code mirror.

This is by design, I expect a chunk of Hoardy-Web users to be paranoid enough to only ever build it from source and install the results manually into their LibreWolf or some such, leaving zero telemetry fingerprints anywhere.

Hoardy-Web asks for a lot of permissions, what does it use all those permissions for?

Capture

Can I use Hoardy-Web to capture web pages while my browser runs with JavaScript disabled?

Yes.

Can I use Hoardy-Web to capture web pages that use a lot of JavaScript?

Yes, with the usual caveats of such a use case, which are discussed above.

Can I use Hoardy-Web to capture a web page as it currently is, after all JavaScript was run, not as it was when it was last fetched from the network?

Yes, Hoardy-Web implement capture and archival of DOM snapshots.

How can I make Hoardy-Web capture a web page completely, especially when parts of it are loaded lazily?

In the most general case, you will have to scroll the page around and click random buttons and media elements.

Hoardy-Web has no “autopilot” for doing this, nor will it ever get one, at least as part of Hoardy-Web extension itself, since “autopiloting” is very website-specific. So, at the moment, the most general semi-automated solution is to run a website-specific UserScript via Tampermonkey or some such, then take a DOM snapshot. (Hoardy-Web will implement UserScript-able API for that, eventually.)

On the other hand, if you

then you can simply go to about:config and toggle dom.image-lazy-loading.enabled to false. All images will start being loaded eagerly after that.

Can I use Hoardy-Web to capture a web page without archiving it, look at it, decide if I actually want to save it, and archive it only if I do, all without reloading the page a second time?

Yes, this is why limbo mode exists.

Why do pages under https://addons.mozilla.org/ and https://chromewebstore.google.com/ can not be captured by Hoardy-Web?

Browsers prevent extensions from running on extension store pages to prevent them from manipulating ratings, reviews, and similar things.

However, you can archive https://addons.mozilla.org/ pages by running Hoardy-Web under Chromium and https://chromewebstore.google.com/ pages by running Hoardy-Web under Firefox.

When running Hoardy-Web under Chromium, a lot of my captures fail with debugger::capture::EMIT_FORCED::BY_DETACHED_DEBUGGER, debugger::capture::NO_RESPONSE_BODY::DETACHED_DEBUGGER, webRequest::capture::CANCELED::NO_DEBUGGER, and similar errors. What do I do?

You are either

Also, Chromium will occasionally detach its debugger at random, it just happens.

When running Hoardy-Web under Firefox, my captures get marked as buggy all the time. What do I do?

This happens because of a race condition between webpage’s service/shared worker and browser’s networking code.

Usually, you can ignore these buggy reqres, since loading another related page is likely to fulfill the same URL.

However, if this happens a lot to you, or if it annoys you, you can go to about:config, toggle dom.serviceWorkers.enabled to false, and restart the browser. Alternatively, you can use NoScript or some such extension to disable JavaScript, and thus the offending service/shared workers, on the page in question.

Why does a (specific) URL or some part of it fails to be properly captured by Hoardy-Web?

Did you read the notes on the bugs of the browser you are using?

Most notably:

On Firefox, some websites define/implement keyboard shortcuts that make Hoardy-Web hard to use. E.g., Wikipedia defines a ton of conflicting keyboard shortcuts making Alt+U navigate to their “Upload” page and Alt+O to their “Login” page, among others. Firefox then treats all of those as having higher priority than both Hoardy-Web’s and its own shortcuts, making things really annoying. Can this be worked around somehow?

Kind of.

Go to about:config and set ui.key.generalAccessKey to 0. Doing that will make Firefox ignore all web page shortcuts defined via HTML’s accesskey mechanism. Which is not ideal if you use those sometimes, but this is better than your shortcuts doing completely random things on random websites.

Doing that will, at the very least, fix Hoardy-Web usage on Wikipedia and other MediaWiki websites. However, the above will not fix Hoardy-Web usage on websites that implement shortcuts via JavaScript handlers, those will be used with higher priority anyway.

Personally, I think that Firefox’s implementation of this whole thing is a bit weird. On Firefox, shortcuts defined by web pages have higher priority than browser’s own shortcuts, but add-on shortcuts have lower priority than both web page’s and browser’s own shortcuts. That is, on Firefox, their priority order is pages > browser > extensions. Both pages > extensions > browser and browser > extensions > pages orders I can justify:

Meanwhile, Firefox’s order makes absolutely no sense to me.

In a better world, both the above order and browser’s own shortcuts would be configurable. In such a world, when using pages > extensions > browser order, browsers would also treat web page shortcuts as per-host opt-ins configurable with firewall-like rules, similarly how uBlock Origin configures itself.

In a simpler world, all browsers would just use browser > extensions > pages order. How many websites properly override browser’s own shortcuts in a useful and OS- and locale-aware way, anyway? Are users of those websites even aware that this feature is even supported there? I’m pretty sure the answers to the above questions are “almost none” and “some, most of these are power-users who discover it while trying use those shortcuts for something else”. So, IMHO, that feature introduces a lot of power-user-oriented annoyances while implementing a very fringe supposedly layperson-oriented use case. Please, remind me again, how many laypersons use keyboard shortcuts at all?

Archival

The documentation claims that all Hoardy-Web archival methods except for submission via HTTP are unsafe. Why?

Archival by exporting using saveAs (generation of fake-Downloads) can fail and lose a bit of your collected data at a time if you press a wrong button in you browser’s UI, misconfigure your browser a bit, or your disk gets out of space unexpectedly.

Archival to browser’s local storage (which is what Hoardy-Web is doing by default) can loose all your collected data at the same time if you uninstall the extension by accident.

Meanwhile, archival by submission via HTTP has none of these problems:

Archival to browser’s local storage was added because it was very easy to implement after stashing was added. It is the default because it usually works fine, it properly reports errors, has the most consistent behaviour across all browsers, and does not require the user to install any Python code, which helps with on-boarding.

In the ideal world, browsers would provide a better saveAs API which would have a less annoying UI for the user and would return out-of-disk-space errors to the extension, in which case exporting via saveAs would be the default.

As it is now, the only way to be absolutely sure you data is properly forever-saved to disk when the extension reports it archived is to use submission via HTTP or re-archive carefully.

When running Hoardy-Web under Firefox, enabling export via saveAs makes the browser’s UI quite annoying. Can it be fixed?

Yes, go to about:config and toggle browser.download.alwaysOpenPanel to false.

This page does not answer my question. What do I do?

Did you try searching for stuff with Control+F on this page? This document contains quite a bit of information.

Is your question about the Hoardy-Web browser extension in truth?

If none of those pages mention your problem, open an issue on GitHub or get in touch otherwise.

Usually, the simplest way to search for things is to simply git clone the whole thing and then search there with grep or more advanced tools. This applies not only to this project, but in general.

Was this useful? Do you want me spending more time working on similar things?

Support me via: