pwebarc: ./extension/page/help.org

A suite of tools for mirroring and hoarding web pages you visit for later offline viewing. I.e. your own personal Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data, which also follows “archive everything now, figure out what to do with it later” philosophy

git clone https://github.com/Own-Data-Privateer/pwebarc
git clone https://oxij.org/software/pwebarc

Files

Raw Source

How to read this document

It is highly recommended you view this page by clicking the “Help” button in the extension’s own UI. Doing that will make this page interactive: the settings popup will be displayed on the right on this page and hovering over or clicking on any links pointing to popup.html will highlight those elements in the popup.

See screenshots if you want to see how it will look.

You can still read this page outside of the extension’s UI, but be prepared for all links pointing to popup.html to be useless.

What?

pWebArc (Personal Private Passive Web Archive) is a browser extension that passively captures, collects, and archives dumps of HTTP requests and responses to your own private archiving server (like the dumb archiving server, also there) as you browse the web.

Glossary

General operation

State Diagram

Reqres change their internal states according to the following state diagram (which is explained below):

(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body recived)
   |                           |                              |             |
   |                           v                              v             v
   |                     (no_response)                   (incomplete)   (complete)
   |                           |                              |             |
   |                           \                              |             |
   |\---> (canceled) ----\      \                             |             |
   |                      \      \                            \             |
   |\-> (incomplete_fc) ---\      \                            \            v
   |                        >------>---------------------------->-----> (finished)
   |\--> (complete_fc) ----/                                             /  |
   |                      /                                       /-----/   |
   \----> (snapshot) ----/         /--------------- (picked) <---/          v
                                   |                   |                (dropped)
                                   v                   v                 /  |
       (archived) <- (sIO) <- (collected) <------- (in_limbo) <---------/   |
                       |           ^                   |                    |
                       |           |                   |                    |
                /------/           \-----\             \--> (discarded) <---/
                |                        |
                \-> (failed to archive) -/

Step 1: Tracking

pWebArc attaches to your browser’s runtime and tracks progress of HTTP requests and their responses, capturing both their request and response headers and data at appropriate times in the browser’s request and response processing pipeline.

Whether pWebArc will track a given request depends on the “Track new reqres” toggles (or checkboxes, if you are using a browser without support for the needed CSS) in the settings popup, e.g:

Disabling any of these toggles does not stop tracking of already initiated requests, it only stops new requests controlled by that toggle from being tracked.

The networking states of the State Diagram

As shown on the above diagram, a new reqres proceeds through the following networking states:

In principle, at reaching finished state the reqres can be serialized and saved to disk, but pWebArc provides more states and UI for convenience.

Glossary

Step 2: Classification

On reaching the finished state, pWebArc performs reqres classification controlled by “Pick reqres for archival when they finish” and “Mark reqres as problematic when they finish” settings. The former set decides whether the reqres in question should be picked or dropped, which influences the actions pWebArc will perform in the next step. The latter set decides if the reqres in question should be marked as problematic.

Problematic reqres

The problematic reqres status is a flag (NOT a state) that does not influence archival or any actions discussed in the latter steps. It exists because browsers provide no indication when some parts of the page failed to load properly — they expect you to actually look at the page with your eyes to notice something looking broken (and reload it manually) instead — which is counterproductive when you want to be sure that the whole page with all its resources was archived.

After all, parts of a dynamically loaded page might simply silently fail to be rendered by associated JavaScript because some of the HTTP requests that JavaScript did in background failed, or, on a static web page, layout and CSS might have made some of the incompletely loaded parts of the page invisible (by design or by accident).

So, to provide an indicator for such cases, pWebArc keeps the log of problematic reqres and displays the number of elements in the log in its toolbar button’s badge.

By default, HTTP requests that failed to get a response, those that have incomplete response bodies, and those for which the browser reported potentially problematic errors but then pWebArc picked them anyway, will be marked as problematic.

Problematic errors are errors like

but NOT errors like

(In principle, pWebArc could have been designed to never record the errors of the latter category in the first place, thus simplifying the above bit, but pWebArc is designed to follow the philosophy or “collect everything as browser gives it, as raw as possible, do all the post-processing logic separately, allow for no logic at all, if the user asks for it”.)

The raw error strings reported by the browser for each reqres can be seen in the recent reqres history log.

If this option is enabled pWebArc will generate a desktop notification each time a new problematic reqres get produced. If you don’t care about the problematic flag and it annoys you, you should disable that option, not options under “Mark reqres as problematic when they finish” settings.

Glossary

Displayed on the Picked/Dropped reqres and similar global lines:

On its own lines:

Step 3: Collection, Discarding, and Limbo

Normally, picked reqres proceed to the collected state, which queues them for archival. Similarly, dropped reqres proceed to being discarded from memory.

Limbo mode

However, sometimes you might want to actually look at a web page before deciding if you want to archive it or not. The naive way to do it would be to load a page with capture disabled first, look at it, and then, if you want to save it, enable it, and reload the page with Control+F5 again.

Obviously, this is both annoying and will force you to fetch everything twice.

Which is why “limbo mode” exists. With one of the “limbo mode” options enabled, pWebArc will instead capture everything as normal but then, instead of sending the reqres in question to collected or discarded states immediately, it will put them into in_limbo state where they would linger until you collect it or discard them manually by pressing the appropriate-buttons, or until “Automatic actions for recently closed tabs” options make a decision semi-automatically for you.

A picked reqres will be put into in_limbo when “Pick into limbo” setting is enabled in the currently active tab or when one-of-the-other settings is enabled for other reqres sources.

Similarly, for a dropped reqres will be put into in_limbo when “Drop into limbo” setting is enabled in the currently active tab or when one-of-the-other settings is enabled for other reqres sources. (This latter option mainly exists for debugging.)

If this option is enabled and there are more than this number reqres in_limbo or the total size of all dumps in_limbo is more than this size (in MiB), pWebArc will complain to remind you to collect or discard some of them so that your browser does not waste much memory and so that you won’t loose too much data if something crashes.

Glossary

Displayed on the Collected/Discarded reqres and similar global lines:

On its own lines:

Step 3.5: Logging

On entering collected or discarded state, metadata of each reqres is copied into the recent reqres history-log and is kept there until the size of the log reaches this many elements, at which point the older elements of the log start being elided automatically.

You can also ask pWebArc to forget some history manually by pressing this button to forget history of reqres generated by the currently active tab or that button to forget all history.

Note, however, that problematic reqres will not get automatically elided from the log, nor forgotten by using the above buttons. To forget about them, you will have to unset the problematic flag on the respective reqres via this button, or that button, or use similar buttons in the log.

Step 4: Archival

When “Archive collected reqres” toggle is enabled, pWebArc will pop collected reqres from the archival queue one by one, serialize them into CBOR-formatted dumps, and then push those dumps to the archiving server at “Archive collected reqres to URL” setting by turning each reqres into a POST HTTP request with the dump of the reqres as request body (which is denoted by sIO state on the diagram). It will also specify profile query parameter to the POST request using the appropriate “Profile” setting, e.g.

Evaluation of the profile parameter is done just before the POST request is sent, so if the queue is not yet empty, and you disable “Archive collected reqres”, edit some of the “Profile” settings, and enable it again, pWebArc will start using the new setting immediately.

If this option is enabled and some reqres failed to be archived, a new desktop notification will be generated. If this option is enabled, a new desktop notification will be generated when the archival queue gets empty the very first time or after any failures.

Glossary

On its own line:

Displayed on the Archived/Failed reqres line:

Shortcuts

pWebArc provides a bunch of keyboard and context menu shortcuts to allow using it in more efficient ways.

Keyboard shortcuts

pWebArc provides shortcuts to:

Context menu actions

pWebArc provides context menu actions to:

Quirks and Bugs

Known extension issues

Relevant issues of Firefox, Tor Browser, LibreWolf, etc

Relevant issues of Chromium, Chrome, etc

On Chromium-based browsers, there is no way to get HTTP response data without attaching Chromium’s debugger to a tab from which a request originates from. This makes things a bit tricky, for instance:

Moreover, Chromium has the following long-standing issues/bugs making things difficult:

Error messages and codes

Desktop notifications

Errors recorded in reqres, as seen in the log

Most error codes are produced by attaching one of the following prefixes to the raw error code given by the browser:

In particular, webRequest::NS_ prefix on Firefox, and webRequest::net:: and debugger::net:: prefixes on Chromium signify various issues produced by the networking stacks of those browsers. For instance:

The exception to the above rule of keeping everything as raw as possible are webRequest::pWebArc:: and debugger::pWebArc:: prefixes which signify various errors produced by pWebArc itself in its webRequest- or debugger-handling code, respectively. In particular:

Frequently Asked Questions

Does pWebArc send any of my captured web browsing data to any third-parties?

No. pWebArc only ever sends data to the archiving server URL you specify.

Does pWebArc collect and send any telemetry anywhere?

No. Note, however, that pWebArc does persist some global stat numbers across restarts — e.g., Collected/Discarded reqres — for convenience, but they are never sent anywhere, and you can reset them at any time.

Why do pages under https://addons.mozilla.org/ and https://chromewebstore.google.com/ can not be captured?

Browsers prevent extensions from running on extension store pages to prevent them from manipulating ratings, reviews, and etc such things. However, you can archive https://addons.mozilla.org/ pages by running pWebArc under Chromium and https://chromewebstore.google.com/ pages by running pWebArc under Firefox.

Why does a (specific) URL or some part of it fails to be properly captured?

Did you read the notes on the bugs of the browser you are using above?

Most notably:

On Chromium, a lot of my captures fail with debugger::pWebArc::EMIT_FORCED_BY_DETACHED_DEBUGGER, debugger::pWebArc::NO_RESPONSE_BODY::DETACHED_DEBUGGER, and webRequest::pWebArc::NO_DEBUGGER::CANCELED errors. What do I do?

You are either

Can I capture a web page without archiving it, look at it, decide if I want to save it, and archive it only if I do, all without reloading the page a second time?

Yes. This is why “Pick into limbo” setting exists. See above for more info.

In combination with “Automatic actions for recently closed tabs” options you can implement any of the following workflows:

Can I capture a web page/tab/frame as it currently is, after all JavaScript was run, not as it was when it was last fetched from the network?

Yes, you can capture DOM (Document Object Model) snapshots for the currently active tab by pressing this button in the popup.

Doing that will generate and capture snapshots of raw HTML/XMLs for each frame contained in the currently active tab. (Reqres-wise they will be “200 OK” responses, but with protocol set to "SNAPSHOT" and method set to "DOM".)

You can also do that for all open tabs at once by pressing that button.

How do I properly archive a web page completely, especially when parts of it are loaded lazily?

In the most general case, you will have to scroll the page around and click random buttons and media elements.

pWebArc has no “autopilot” for doing this, nor will it ever get one, at least as part of pWebArc extension, since “autopiloting” is very website-specific. So, at the moment, the most general semi-automated solution is to run a website-specific UserScript via Tampermonkey or some such, wait until everything finishes loading, and then take a snapshot. (pWebArc will get an integration for automating that, eventually.)

On the other hand, if you

then you can simply go to about:config and toggle dom.image-lazy-loading.enabled to false. All images will start being loaded eagerly after that.

This page does not answer my question. What do I do?

If the whole content of this page (not just this section, did you try searching for stuff with Control+F? there’s a lot of info here) does not explain your problem, open an issue on GitHub or get in touch otherwise.