HTTP POST requests and responses, as well as most other HTTP-level data.All notable changes to this project are documented in this file. The format is based on Keep a Changelog. This project adheres to Semantic Versioning.
Also, at the bottom of this file there is a TODO list with planned future changes.
*:
python-v3.13, black-v25.1.0, mypy-v1.20.1, pylint-v4.0.6.python-v3.13 or later is required now.
Changed semantics of path, mq_path, and similar --expressions to never start with a “/” and to end with “/” when a non-trivial (non-empty and not-just-a-“/”) raw_path of the input URL ends in a “/”.
That is, now you have to write %(netloc)s/%(path)s and similar instead of %(netloc)s%(path)s and similar and those expressions might end in a slash now.
All builtin formats were edited accordingly.
Also, the latter could-end-in-a-slash property did produce some observable changes in --output formats, but the default ones stayed the same.
Relatedly, from now on, when raw_path ends in a “/”, the last element of path_parts will be an empty string.
Normally, you would expect the last element to just end in a “/” too in that case, as with Python and most other APIs that work over POSIX, but URLs are a bit crazy in that they allow path components to include quoted slashes, which makes the usual way of doing this inapplicable.
Thus, e.g., http://example.com/abc/%2Fcde/fgh%2F/ would now be parsed as path_parts == ["abc", "/cde", "fgh/", ""].
Switched to kisstdlib-v0.0.13.
The main effect is that <date1>--<date2> date ranges no longer include all of date2 by default.
This feature wasn’t really used in the published versions, so the effects of this should be minimal.
In any case, the old semantics can be recovered by adding “.$” at the end of each such expression.
serve:
Introduced an explicit “find” request that answers to /find/<selector>/<query> requests and made all other request handlers — including those of web, but excluding those of non-interactive unavailable requests — to redirect there when hoardy-web decides it wants to perform a search.
Added support for selectors that specify multiple coma-separated date+times and/or time intervals, as well as optional re and glob options.
Thanks to the new kisstdlib, selectors can now also include advanced dynamic date computations, including in human-readable formats.
You can now ask for all URLs in this_week-1d--now interval (“from last Sunday to now”), for example.
The README.md now has a bunch of examples.
From now on, when performing redirects for selectors of “closest to given timestamp” kind and when generating search pages, produced timestamps always correctly and uniquely identify each reqres in the index, using the smallest possible timestamp precision.
Changed generated search pages to show date+time, HTTP request method, as well as request and response sizes for all matching reqres.
Implemented --fallback, --fallback-output, and related options.
With --fallback enabled and --to set the server will now write everything the clients submit via POST to disk, even when those POSTed dumps can’t be parsed as proper WRRs. Obviously, such submissions won’t be added to the index and thus won’t be replayed. However, when cbor2 library or hoardy-web fail to parse such dumps properly, they will no longer become stuck in the extension, either. Which is why, from now on, the --fallback option is enabled by default.
This is exactly how simple_server (hoardy-web-sas) does it too. If you don’t trust your clients to not submit complete garbage, however, you can get the previous behaviour by running serve with --no-fallback.
Added --bucket-prefix and --fallback-bucket-prefix options, making the use case of multiple serve instances dumping to a single --to destination to no longer need any external post-processing.
scrub (get, mirror, serve, etc):
Reworked scrub to no longer attempt (and fail) to scrub *.ps, *.pdf, and *.epub documents by default.
Internally, replaced dyndoc document kind with psdoc kind and dropped dyndoc kind altogether.
The psdoc kind is now used to classify *.ps and *.pdf documents. The *.epub documents are classified as archive now.
By default, scrub now leaves all of them as-is.
Most notably, this makes it possible to collect, archive, and then replay those documents via mirror and serve subcommands run with their default options.
Reworked scrub with +verbose option set to rename URL-containing HTML tag attributes to censored-<name> when that attribute gets remapped in a “close-ended” way, instead of simply rewriting them into javascript:void(0)s.
Thus, e.g., scrub with /jumps option will now rename all HTML a hrefs pointing to unavailable URLs into censored-hrefs instead of just dropping those attributes.
This makes all those rewrites easier to inspect and debug.
serve:
Improved performance by making requests for known-unavailable page requisites to be processed without touching the index at all.
Improved performance of /(find|web)/<whatever>/* and similar requests by making them iterate over all the available URLs, without performing any URL matching at all.
organize:
organize is working.Documentation:
scrub (get, mirror, serve, etc):
Reworked scrub to censor out unavailable image sources from inside srcset attributes.
This fixed a bug where scrubbing an HTML page with an image that has multiple sources, some of which were archived but others were not, would produce a visually broken page in cases when the browser decides to use a fallback URL as that image’s source.
This fix needs some more testing, since it’s unclear if the current implementation will work in all the possible combinations of img and source tags in all the browsers, but it does work for Firefox and the pages I tested it with.
Fixed scrubbing of source and SVG use tags.
Fixed scrub’s +verbose option to work for stand-alone CSS files properly.
Fixed --remap-closed command line option failing to work properly.
Fixed a couple of obscure URL quoting issues, which fixed:
a bug where serve would sometimes fail to find a previously archived requested URL in its index;
a bug where mirror would sometimes generate broken links.
Fixed scrubs +inline_fallback_icon option failing to work property in some cases.
serve:
serve would sometimes fail to find a previously archived requested URL in its index.DOM-snapshots, incremental improvements, bug fixesYes, extension-v1.29.0 was skipped deliberately.
DOM-snapshotsCore:
From now on, DOM-snapshoting machinery generates incomplete reqres for failed-to-snapshot frames instead of just failing to produce any snapshots for that tab at all.
From now on, DOM-snapshots record and dump anonymous iframes as sub-reqres/WRR of their parent-frame reqres/WRR instead of trying to dump them into separate reqres/WRRs.
This fixed a bug where trying to naively snapshot and then replay a tab with a bunch of anonymous iframes would often produce incorrect results.
Changed documentUrl, originUrl, and Referrer header handling so that the resulting WRRs would match WRRs produced by webRequest/debugger-generated reqres.
In other words, fixed a bug where DOM-snapshots needed special handling when filtering with the hoardy-web tool.
Also, I partially screwed this change up on my first attempt and then captured and archived a couple of gigabytes of useful complex JavaScript-heavy web pages using that version while stress-testing this and other related changes. Thus, extension-v1.29.0 was skipped to make those snapshots easy to filter and fix programmatically later. Which should not matter to you, unless you are planning to build and run between-release extension versions.
Automatic/semi-automatic fixups for those and older WRRs in the hoardy-web are a to-do item ATM.
Fixed a bug where snapshoting and then immediately replaying an already settled tab could sometimes leave newly created SNAPSHOT reqres in_limbo.
Implementing this in a way that actually works property took a surprising amount of debugging and tweaking of tab-settling machinery.
However, from now on you can totally set Settle in <N>s ... to a large-enough value, spawn a bunch of tabs to load a bunch of JavaScript-heavy web pages, then, press Snapshot all and Replay all per-window buttons immediately one after the other, and Hoardy-Web will do its best to snapshot, archive, and replay those tabs as they become ready.
From now on, SNAPSHOT reqres get ignored for the purposes of tab settling and smart tab switches.
Which is to say, performing a DOM-snapshot no longer un-settles its tab, nor does it change that tab’s position in the smart tab switching order.
Core, Internal State pages:
SNAPSHOT reqres would fail to be filtered by windowIds.Core:
Fixed a bug where some per-tab configs and stats/states would frequently fail to be restored properly on extension reload.
Fixed a bug where collecting in_limbo reqres and then immediately spawning a replay in a new tab could replay a partial result.
From now on, replaying a tab collects all in_limbo reqres immediately instead of waiting for the tab to settle. Then, it collects the rest as the tab settles and gets re-navigated to its replay.
In other words, fixed a bug where some stuff could be left in_limbo when you re-navigate a tab to its replay.
Additionally, this way usually allows the archiving server to do most of the required work while the tab is settling, making things more wall-time efficient.
Fixed a bug where some reqres could become stuck in the in_flight state for a very long time when a bunch of stuff is being archived in background, thus preventing other scheduled actions from firing as early as they should.
Fixed some context menu action failing to work.
A bug introduced in extension-v1.28.0. Found by biome lint.
Fixed a potential crash when stashing and/or archiving reqres into local storage.
A bug introduced in extension-v1.27.0. Found by biome lint.
Fixed a bunch of minor code issues found by biome lint and oxlint.
From now on, generated WRRs get dumped into canonical CBORs, with sorted dictionary keys.
Core, Toolbar Button:
Core, Popup UI:
Reworked the core to make updates to Settle in <N>s ... and/or ... retry up to <M> times options apply immediately.
That is, you can set Settle in <N>s ... to a large value, load a new tab, ask for a DOM-snapshot or a replay in it, then set Settle in <N>s ... to 0, press Enter, and the scheduled action will be run immediately instead of the new value of that setting only applying to the next action you schedule.
Improved popup layout a bit.
Made it hide most of the global stuff on the default tab/tag.
Removed all Forget (log) buttons and their RPC calls.
Those buttons were an atavism, they were implemented for a popup UI design that did not work out and I have not used them once since.
Improved and homogenized some scheduled action names.
Separated Log events option out of Debug runtime option.
Fixed a bug where almost_done reqres would fail to be counted and displayed properly.
Found by oxlint.
Fixed a bug where single-column layout would not work.
A bug introduced in extension-v1.28.0.
Found by oxlint.
Core, Notifications, Logging:
Improved re-archiving-related notification generation.
Fixed some typos in notification messages.
Improved some internal logging messages.
Help page, Keyboard shortcuts:
Documented how to disable accesskey-defined shortcuts on Firefox.
Improved “Keyboard Shortcuts” section a bit.
Improved some keyboard shortcut descriptions.
Documented all re-archival errors in the appropriate section.
Updated it to reflect other changes described above.
Internal State pages, Saved in Local Storage page:
*:
prettier+prettier-plugin-curly and/or oxfmt.Core, Shortcuts:
Introduced a new Tab: Close and Discard In Limbo keyboard shortcut that closes the currently active tab and then discards all of its data from limbo, now bound to Alt+Shift+W by default.
This same shortcut was previously used by the action that simply discarded all tab data from limbo, so, from now on, that action is bound to Alt+Shift+D instead.
Edited most of the other shortcuts to make it possible to use the same shortcuts on both Firefox and Chromium.
Re-mapped many default keyboard shortcuts to per-window versions of their actions (see below).
Since the above changes modified most keyboard shortcuts, this release became a good opportunity to rename all related internal commands to use a more consistent naming scheme for the upcoming external extension API, so that was also performed now.
Thus, you’ll probably want to look at the “Keyboard shortcuts” section on the Help page and possibly edit some of them before proceeding. Also, note that that section can now help you quickly reset keyboard shortcuts to their default values on browsers that support it (see below).
Core, Shortcuts, Popup UI:
Tab: Replay in a New Tab keyboard shortcut and popup UI button.Core, Shortcuts, Help page, Popup UI:
Improved shortcut description format, putting the thing they act upon at the very beginning.
Improved many of those descriptions to be clearer, re-aligned some of them with reality.
Reworked the order and grouping of keyboard shortcuts to be nicer, especially on Chromium where its keyboard shortcuts settings UI ignores manifest.commands order, sorting them in lexicographic order instead.
Reorganized the keyboard shortcuts table:
Reworked its format slightly, making it look more like related browser’s settings pages do.
On browsers that support browsers.commands.onChanged event, the above table will now be updated automatically as you are editing your keyboard shortcuts in browser settings, without you needing to reload the whole page.
On browsers that support browser.commands.update call, when the currently set and the default shortcuts differ, the contents of the “Default” column will now become a button that resets the corresponding shortcut to its default.
If you want to change them to something else, however, you’ll still need to go to your browser’s settings.
Some important keyboard shortcuts now also get mentioned directly in the other sections of the Help page, including their actual bindings, which also get updated automatically on browsers that support browsers.commands.onChanged.
From now on, on Chromium, the Firefox variant of manifest.commands gets stored into the extension ZIP/CRX by the build process and the defaults from there are now being used when generating the aforementioned shortcuts table.
Which is to say that, on Chromium, most keyboard shortcuts now come with useful defaults, even though Chromium forbids more than 4 defaults to be set in extension’s manifest.commands.
As a nice side-effect, the ordering and grouping of keyboard shortcuts in the above table is identical on both Firefox and Chromium now.
Core, Popup UI, Shortcuts:
Introduced new per-window configuration and tracking machinery, keyboard shortcuts, and UI:
Implemented .windowId tracking and filtering for reqres.
Made Internal State pages narrowable by windowIds.
Introduced per-window configs and state/stats with all the needed internals and popup UI, including a new tag/tab for the latter.
Implemented save and restore for per-window configs and state/stats for when extension is reloaded.
Implemented the per-window versions of all of Snapshot, Replay, Forget Log, Show State, Show Log, Collect In Limbo, Discard In Limbo, Unmark Problematic, and Stop In Flight actions/buttons/keyboard shortcuts.
Reorganized the popup UI in a major way to work with the above changes.
Most notably, the New root tabs configuration options block became New root windows/tabs and is now completely hidden by default. Instead, there’s now a new New windows block, parts of which are now shown in the default popup UI state, similarly to how it worked before.
Note that the semantics of which per-window config gets used for a newly spawned tab could be a bit tricky, see this newly written piece of the Help page for more info.
Popup UI, Internal State pages, Saved in Local Storage page:
Introduced Sparse UI variants, controlled by a new User Interface > Sparse configuration option.
Having this option enabled makes popup UI buttons and toggles larger and adds more space between different interactive elements, which makes the experience of using the extension on touchscreens much nicer.
From now on, by default, sparse UI variants are enabled on Fenix and disabled everywhere else.
Edited the CSS and popup UI labels to make the popup to be more compact.
Which is most useful on Fenix too.
Internal State pages, Saved in Local Storage page:
buggy reqres flag into buggy_request and buggy_response.Core, Popup UI:
From now on, when re-archival via submission over HTTP is enabled but all other HTTP archival-related options are disabled, the extension won’t try to perform server checks at startup. However, when re-archival via submission over HTTP is enabled, each Re-archive all and Re-archive new button click will now perform the server check before proceeding.
This improves usability on Fenix when the archiving server is only available intermittently.
Core, Toolbar button:
From now on, no internal actions will be scheduled in situations when it’s obvious they would do absolutely nothing (like trying to Collect In Limbo when the limbo is empty).
Since they would do nothing, this change simply eliminates some unneeded UI updates.
Notifications:
problematic reqres add a “…” line at the end of problematic URL list when they want do display more than maximum allowed number of lines.Documented DOM snapshots better.
Documented smart tab switch/highlight machinery.
Reworked the internal machinery to make all links keep their hrefs, making hovering over them informative.
Core, All internal pages:
Introduced some more runtime debugging machinery.
I expect the probability of a broken link or reference appearing on any of the internal pages to be much lower now.
Core:
Fixed a bug where the first tab in a new browser instance would start with the wrong per-tab config values.
A bug introduced in extension-v1.26.0.
Fixed a bug where toolbar’s badge wouldn’t show scheduled replays.
A bug introduced together with that feature in extension-v1.27.0.
Fixed a bug where re-archival would fail to produce its “done” notifications when some things fail to be re-archived.
A bug introduced in extension-v1.27.0.
Fixed a bug where trying to compress uncompressible WRRs would fail to fallback to no compression.
A bug that existed since those functions were introduced.
This bug was probably never actually hit by anyone because almost all WRRs can be compressed, even if a tiny little bit.
On Firefox, fixed a bug where context menu actions that promise to spawn their result in a new window would frequently fail to do so.
A bug that existed since those functions were introduced.
Documentation:
Core, Popup UI, Shortcuts:
Introduced new tab settling machinery:
finished.Settle in <N>s after 'in_flight' and ... retry up to <M> times config settings.In other words, from now on, trying to snapshot or replay a tab that has associated in-flight reqres will wait until all those reqres finish, then wait an additional configurable delay to allow that page’s JavaScript to process its fetched data (and, possibly, update its DOM and/or generate new requests), and then do the actual snapshot or replay only after that page settles.
If the page generates new requests in the meantime, the process will repeat from the beginning up to configured number of times.
Introduced new Smart switch tabs and Highlight tabs actions, popup UI buttons, and shortcuts.
The extension now provides a bunch of actions that can switch and highlight/select tabs in ways that take those last-finished timestamps mentioned above and the numbers of problematic and in_limbo reqres belonging to those tabs into account.
Notably, the extension now provides popup UI buttons and keyboard shortcuts that, in the currently active window:
finished reqres,problematic and/or in_limbo reqres,These features are most useful when you use the limbo machinery extensively, as it allows you to easily switch between the not-yet-archived tabs easily, highlight/select them and them move them to another window, etc.
This is also useful outside of the limbo use-case since it allows you to browse your tabs in order they were loaded/updated.
Added tracking and display of accumulated reqres sizes for picked, dropped, and problematic stats.
Added tracking and display of dumped, written, rewritten, and deleted stats (under Persistence on P&R tab).
Core:
Improved tab replay handling.
From now on, replaying a tab collects all of its in-limbo reqres and then delays navigation to the replay page until everything actually finishes archiving.
Improved extension reloading algorithm.
From now on, the extension won’t attempt to reload while things are still archiving and will stash reqres lazily, only when there’s no other things that can be done to those reqres instead.
The dropped/discarded reqres are now allowed to auto-unmark problematic flags from other dropped/discarded reqres.
From now on, this works similarly to how in_limbo reqres unmark other in_limbo reqres when Auto-unmark 'problematic' reqres ... when a new 'GET' reqres replaces it ... regardless of its 'in_limbo' state is unset.
The picked/collected reqres still unmark everything as before (when auto-unmarking machinery is enabled at all).
This makes the user experience much more pleasant.
From now on, almost_done reqres get processed before all archivals.
This eats slightly more RAM when hundreds of requests are fetching simultaneously, but produces better-compressed bundles more reliably.
Core, Popup UI:
Reworked Work offline, Track new requests, and Track new requests ... including when 'Work offline' is set config toggles to be per-tab/per-source and removed the old global toggles altogether.
The latter ones were unsafe to use in most cases.
Improved task labels used by the Actions ... scheduled line.
From now on, task names of tasks
From now on, all In flight stat lines display the almost_done stat separately from finishing_up.
See the Help page for more info.
Popup UI:
Improved and some many labels.
Improved some help strings.
Internal State pages, Saved in Local Storage page:
Reorganized the UI a bit, edited button labels and help strings.
Switched by-URL filters to use HTML input type="search".
Made by-URL filters trim their inputs to simplify usage while copy-pasting things around.
Added a Reset button to each filter block.
Made all reqres tables display not just reqres sizes, but also their remainders (dumpSize - requestSize - responseSize).
Core, Notifications:
Improved the order in which new notifications get generated, when the extension wants to generate a bunch at once.
From now on, stashing and archival track if the unstashed/unarchived error they are generating is real or was produced by the corresponding Paranoid error handling toggle in the popup. When some errors are of the latter kind, the generated notification will now say so and point the user to those toggles.
Core, Toolbar Button:
From now on, on Chromium, state updates only ever update toolbar icons and badges associated with active tabs, instead of updating them for all tabs.
This is how it works on Firefox too, but on Firefox toolbar buttons are per-window, while on Chromium they are per-tab, so the implementations differed before, now they do not.
This greatly improves extension’s performance on Chromium with lots of open tabs.
Documented the above changes.
Improved documentation.
Core:
Polished internals a bunch more.
Removed some dead code.
Core, Internal State pages:
Core, Popup UI:
Fixed queued task labels used by the Actions ... running line.
Fixed the Stop button on the per-tab “In flight” line to work again.
The relevant bit of code was accidentally left in an unrelated later commit while rebasing the previous release.
Fixed some numerical values failing to be clamped properly in fixConfig.
Popup UI:
Documentation, Popup UI:
Fixed rendering of some tooltips.
Fixed some typos.
From now on, Stop tracking button there respects its reqres filters.
And thus, finally, all buttons on those pages respect their reqres filters now.
Core:
From now on, reloading the extension preserves per-tab config and stats/state.
Similarly, those per-tab things are now also being preserved on tab replacement (when a browser replaces a tab with another tab pre-rendered in background).
Fixed a bug where reqres from closed tabs could forget their proper bucket values and get archived with the default bucket instead.
Fixed a bug where unmarking some problematic reqres and then reloading the extension would make them problematic again.
Fixed a bug where discarding a reqres from limbo would auto-unmark the problematic flag from said reqres.
Fixed a bug where a reqres being discarded could cause other reqres to be auto-unmarked as problematic.
Fixed a bug where a successful archival of a reqres failed to force a retry of some reqres that previously failed to archive to the same archiving server.
Core, Toolbar button, Popup UI:
Fixed a bug where scheduled actions list and/or toolbar button’s badge can become stuck in incorrect states even though global state variables they are derived from are fine.
Fixed a bunch of bugs where toolbar button would fail to be updated properly in a bunch of situations.
Popup UI:
Running actions line now uses a better naming scheme for its actions.Core:
Improved performance of reqres filtering machinery.
Improved debugging log output.
Generalized away some copy-paste.
Improved naming schemes of various thins.
Fixed a bunch of code lint.
Polished the internals.
Core:
*:
Updated for kisstdlib-v0.0.11.
Switched to using kisstdlib’s abbrev and quoter machinery for abbrev* expressions and URL quoting, making things cleaner and improving performance a bit.
mirror:
Switched to using and printing relative paths.
The old behaviour can be produced by giving absolute paths as --to and inputs.
Build:
Improved the build process a bit.
Improved the tests.
Documentation:
--help.serve:
*:
pyproject.toml:
cbor2 module versions to resolve #21. Thanks to @HexagonWin on GitHub for reporting it!pyproject.toml:
cbor2 module versions to resolve #21.Notifications:
Improved some notification messages.
Clicking “Hoardy-Web updated” notification now opens the CHANGELOG.
Popup UI, Shortcuts, Documentation:
Improved a bunch of option doc strings, aligned them with reality.
Re-ordered shortcuts to use “global, then per-tab” action order to improve comprehensibility.
Popup UI, Internal State pages:
Reorganized, edited, and improved that page quite a bit, aligned everything there with reality.
Fixed a bunch of typos.
Documented setup and usage workflows on Fenix.
Reworked the body of “Keyboard shortcuts” section into a table instead of a list.
That table is also completely auto-generated now, so de-synchronization with the actual shortcuts is now impossible.
CHANGELOG:
Core:
Popup UI:
Fixed #options variant, so add-on’s preferences menu should work again, which is most notable on Fenix.
Fix display of re-archive-related warnings.
Core, Replay:
On Firefox, fixed Auto-replay mode sometimes making Firefox fetch to-be-replayed pages from the Internet as their replays were being fetched from the archiving server.
It was a race condition in the code.
From now on, when the archiving server does not support replays, Auto-replay mode will generate error notifications and cancel to-be-replayed requests instead of silently switching to fetching data from the Internet.
Fixed config migrations sometimes resetting some config values to their defaults when migrating through multiple extension versions at once.
Only Generate notificaitons about ... 'problematic' reqres and Track new requests ... including when 'Work offline' is set were ever affected, so you might want to ensure that those are set to the values you actually want.
problematic reqresCore, Capture, Internal State pages, Popup UI:
The machinery behind Auto-unmark 'problematic' reqres option, when that option is enabled, now gets applied regardless of reqres collection status.
In other words, automatic unmarking machinery now gets applied to dropped and/or discarded reqres too.
In other words, from now on, with that option enabled, you will never have more than one problematic GET reqres for each URL.
This makes the extension measurably slower in some use cases, but it greatly reduces the clutter on the Internal State pages, so I think this is an okay trade-off.
Enabled Auto-unmark 'problematic' reqres ... when a new 'GET' reqres replaces it ... regardless of its 'in_limbo' state configuration option by default, for simplicity.
Because the previous default behaviour was confusing even to me, sometimes.
Though, note that its value will not be flipped if you are updating from extension-v1.23.0, so you might want to flip it manually.
*:
Renamed “errored reqres” -> “bugged out reqres” to distinguish them from “reqres errors” and “buggy reqres”.
Names are hard.
Core, Internal State pages, Popup UI:
Bugged out (Errored) status line back to where it was before extension-v1.23.0.Delete action for those from the Popup UI to the state page.Core, Popup UI:
Renamed Re-archive adjunct button -> Re-archive new.
Reorganized the re-archival section a bit.
Added a new Delete (re-)archived button into the re-archival section.
I.e., when re-archiving, you can now press Re-archive new followed by Delete (re-)archived instead of fiddling with on-the-fly deletion options. This is simpler and safer in some use cases, but less efficient in others.
See the Help page for more info.
Core, Notifications:
Re-archival actions now generate better notifications, listing the exact counts of all the actions they take.
From now on, deletion of reqres saved in local storage, via the Delete (re-)archived popup UI button or a similar button on the Saved in Local Storage page generates notifications on completion too.
Core, Internal State pages, Popup UI, Shortcuts:
The state pages now take sessionIds of old stashed reqres into account and Popup UI buttons and shortcuts respect those:
The global state page is still unnarrowed by default and will display all stashed reqres as usual, but it can now be narrowed to any previous session, if desired.
By contrast, per-tab state pages get narrowed to the current session now.
Popup UI, Internal State pages:
Improved styling of tooltips a bit.
Improved styling of tristate toggles, again.
All text and number input fields now generate updates on Enter keypresses.
I.e., you no longer need to move the focus elsewhere to make your changes apply.
Reworked Without errors tristate filter into With errors.
I.e., it’s meaning is now reversed.
Changed formatting of entries to be denser.
Core, Capture, Internal State pages:
All sections of the internal state pages can now be filtered by URL, in a bunch of different ways: by the exact URL, by a sub-string, and via a regular expression.
From now on, most of the action buttons there respect reqres filters. Most notably, this now includes the Retry all button of the “Failed to archive” section too.
I.e., it’s now possible to retry archivals only for a filtered subset of reqres.
The core now tracks and internal state page entries display reqres dump, request, and response sizes.
Though, at the moment, for in-flight reqres those values won’t be updating automatically, only if you manually refresh the page.
Though, obviously, when a reqres stops being in-flight, the displayed values will always be up to date.
Core, Capture:
The webRequest::capture::CANCELED::BY_WORK_OFFLINE reqres error is no longer treated as “important”.
In other words, the extension now treats its own Work offline mode equivalently to ad-blocking extensions like uBlock Origin and similar. This behaviour is both more consistent semantically and more pleasant UI-wise.
This way you can now enable Work offline in a tab, scroll it around, switch from that tab and back to it, and watch that page’s JavaScript try and fail to send tracking requests about those events, without those canceled reqres becoming marked as problematic.
Enabled Track new requests ... including when 'Work offline' is set option, both by default and if you are updating from extension-v1.23.0.
Because after the above change, effectively speaking, its semantic has changed.
Core, Popup UI:
Improved semantics of the actions behind the buttons that run and cancel all currently scheduled actions.
The resulting extension state shouldn’t ever be confusing now.
Core, Internal State pages:
Still problematic reqres no longer get dropped from the logs when you run the Forget all action, as help strings of those buttons claim.
This behavior is slightly annoying, but this way is safer and does not loose problematic reqres when the extension restarts/updates.
Core, Shortcuts:
Show Log shortcuts will now scroll their corresponding Internal State pages in a way that puts tails of the logs there at the bottom of their page frames, thus maximizing the number of log entries shown by default.Core:
Fixed notifications about unarchived reqres not being generated in some cases.
Improved timeout computation logic of the saveGlobals internal action to minimize writes to disk in situations where such minimization is possible.
On Firefox, closing a tab no longer stops its in-flight reqres.
This is how the extension did things in its early versions before Chromium support landed properly.
This way is superior because it allows you to properly capture requests that some websites make from their window.onclose handler. (The browser will abort such requests milliseconds later, but their request parts have a fair chance to get sent anyway. Many nefarious websites use this feature to track you surreptitiously because such requests are impossible to observe via browser’s Network Monitor.)
On the other hand, this way spawned downloads don’t immediately become canceled reqres, turning into proper in_flight reqres which end up being incomplete instead. (Because Firefox does not supply bodies of spawned downloads to the extension.)
Moved handlers of most UI-actions into the main thread while making the archiving process interrupt itself when new handlers get queued.
This appears to have solved all the archiving-related heisenbugs for me.
Popup UI:
Documentation:
manifest.json:
browser_specific_settings.gecko.data_collection_permissions field, which tells AMO that this extension does no third-party data collection.Core, Replay, Popup UI:
Implemented a new Disable 'Auto-Replay' in replayed tabs config option and its popup UI, which is now also enabled by default.
Enabling this option makes Hoardy-Web start untoggling Auto-replay mode automatically when a tab navigates to a replay page.
This makes working with replays much less annoying.
Core, Capture, Popup UI:
Implemented some generic machinery for auto-unmarking of problematic reqres, added a bunch of new config options in the vein of the old Closed tabs > Auto-unmark 'problematic' reqres option, and created a new popup UI section for them.
Most notably, a new Auto-unmark 'problematic' reqres ... when a new 'GET' reqres replaces it option does what is says on the tin.
It is also enabled by default now.
In other words, from now on, when you reload a partially loaded page and some of the URLs that were previously problematic get fetched properly, those old problematic reqres will be unmarked automatically.
However, note that the interaction of these options with the limbo machinery is inherently complex and possibly counter-intuitive, see their help strings for more info.
Core, Capture, Replay, Popup UI:
Notify about 'problematic' reqres is now a tristate of “off”, “notify about new problematic reqres”, “do that and notify about newly unmarked problematic reqres too”, the middle value is the default still.
Renamed Closed tabs > Auto-unmark 'problematic' reqres option to Auto-unmark 'problematic' reqres ... when its tab closes, moved it to the aforementioned popup UI section.
Reworked this option to act immediately instead of using the value of Closed tabs > Auto-collect/discard 'in_limbo' reqres ... after 'N's.
Also reworked it to look at Notify about 'problematic' reqres option for notification generation.
The default value of Auto-collect/discard 'in_limbo' reqres ... after 'N's option is now set to 0.
Replaced Toggle 'Work offline' options impurely config option with a new Track new requests ... including when 'Work offline' is set option, which is now disabled by default.
The behaviour of the latter option is declarative and its implementation requires no weird automatic option flipping anymore.
Core:
Loading of the stashed reqres at startup is now asynchronous.
Thus, most of the time, the extension can start capturing new reqres almost immediately.
Improved some error messages.
Improved logging to browser console.
Simplified and generalized a bunch of internal machinery.
Popup UI, Internal State pages:
Popup UI:
Documentation:
Help page.Core, Capture:
Fixed DOM snapshots failing for pages with top-level HTML/XML comments.
Improved robustness of Auto-Replay mode, making it work for background requests too, hopefully.
Core:
Fixed loading of data saved into storage.local becoming slower and slower with more and more extension usage in some use cases.
The complete fix for this is very computationally expensive, so the current implementation is fixing the issue incrementally, bit by bit, each time you (re-)load the extension, until the most optimal state is reached.
Core, Toolbar button, Popup UI, Internal State pages:
Fixed error messages in the popup UI being unreadable when the extension fails to initialize very early.
Fixed popup UI getting into an inconsistent (and ugly) state when opening it from a new tab while the extension is still initializing.
Fixed extension’s toolbar button’s icon becoming inconsistent in the same situation.
Fixed collecting or discarding from limbo not updating the problematic reqres list properly.
Documentation:
Auto-Replay mode and other annoyance fixesCore, Toolbar button, Icons, Popup UI, Shortcuts:
Implemented Auto-Replay mode, its per-tab options, shortcuts, and toolbar display.
Enabling that mode makes Hoardy-Web start automatically redirecting all navigations in the corresponding tab to their replay URLs.
This is most useful if you want to open a replay of a bookmarked URL skipping the annoying part where you have to add the replay URL prefix to the original URL manually.
I.e., in most cases, you won’t need to bookmark replay URLs anymore. With Auto-Replay enabled, bookmarks of the original URLs will work now just as well.
Core, Popup UI:
Added “Paranoid error handling” options for HTTP and local storage archival methods (on the “P&R” tab, config.submitHTTPParanoid and config.saveLSParanoid settings).
Disabling these disables “… appears to be defunct” archiving failures, making Hoardy-Web attempt archiving each and every reqres separately. This is only useful in situations when something else is buggy, like #20.
Context menu:
Improved ordering of elements.
Added icons for “Replay link …” actions.
Documentation:
Help page, aligning it with reality.Core:
Core, Popup UI:
config.debugConfig (“Allow unsafe config values”) out of config.debugRuntime (“Debug runtime”) since the latter is ok to leave enabled while the former should be disabled most of the time.Core, Popup UI, Saved in Local Storage page, Help page:
Implemented new re-archival machinery. Hoardy-Web now follows the following state diagram:
(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body received)
| | | |
| v v v
| (no_response) (incomplete) (complete)
| | | |
| \ | |
|\---> (canceled) ----\ \ | |
| \ \ \ |
|\-> (incomplete_fc) ---\ \ \ v
| >------>---------------------------->-----> (finished)
|\--> (complete_fc) ----/ / |
| / / |
\----> (snapshot) ----/ /- (collected) <--------- (picked) <--/ |
/ ^ | |
(stashToLS?) <----------------/ | v v
| \- (in_limbo) <- (stashToLS?) <- (dropped)
v | |
(queued) <---- (unarchived) <- (stashToLS?) | |
| ^ \-----> (discarded) <-----/
/-\| /---> {U} -----------/ ^
^ | | \ |
| | | \--> (forgotten) ((-> (saved))) |
| | | |
{RE} | v {{!saving && !submitting}} |
|\----> (saveAs) -> (exported) -------------------------/|
| / |
| {U} <-\ /---------/ |
| v v {{!saving}} |
|\----> (srvIO) -> (submitted) -------------------------/|
| / |
| / {{delete}} |
| {U} <-\ /-----/ /-----------------------------/|
| \ / / |
| v v / {{re-archive}} |
|\---> (saveToLS) -> (saved) -------------------> {RE} |
| |
\--------------------------------------------------------/That is, “re-queueing” mechanism was dropped, instead, reqres saved in local storage can now be sent through the archival pipeline multiple times, without touching the queue. Also, the re-archival machinery now has a separate set of configurable archival methods. And the reqres going through re-archival will be sent through a separate instance of the archival pipeline, in a way that will block Hoardy-Web’s main thread, while recording archival errors separately, reporting them at the end, and then forgetting about them (after all, everything is saved in local storage anyway).
This might not sound that impressive, but it is, because its both more convenient and has much better performance:
as before, collection can still work in parallel with this;
but normal archival now stops completely while re-archival is working, which locks local storage from modification, which allows re-archival machinery to be much more efficient than the re-queueing machinery was, re-archiving everything requested and applying all updates in a single pass over local storage, and using the smallest required number of disk writes;
also, internally, the re-archival machinery is now a simple function, not a collection of distinct schedule-able tasks, which means it can easily be called explicitly, which allows common re-archival workflows to be bound to separate UI buttons and/or keyboard shortcuts;
which Hoardy-Web now does, making things much more convenient.
The latter feature solves a common complaint of the Saved in Local Storage page UI being hard to use. From now on, its use is completely optional. Popup UI now has two simple Re-archive all and Re-archive adjunct buttons that re-archive everything and reqres that were “not previously (re-)archived like this”, respectively.
This can be useful in following use cases:
You want to collect and accumulate lots of reqres before re-archiving them by the means of Export via 'saveAs' to make the UI less annoying and compress the resulting WRR bundles better.
You run Hoardy-Web under a Fenix-based mobile browser or under Tor Browser, where the use of re-archival is quite common and needing to manually re-queue stuff all the time was really annoying.
All of this can be done with a single button-press and using a smallest required number of disk writes now.
Core, Default config:
Set the default config.exportAsMaxSize to 64 MiB again and increased the limit to 512 MiB both on Chromium and on Firefox.
Bigger values are better for WRR bundle compression ratios, and my tests had shown that this actually works fine even on Firefox, even though Mozilla’s docs claim it would not, so.
Core:
Performed a huge internal reworking, splitting everything into a bunch of logically consistent mostly-independent modules, making many functions mostly side-effect pure, etc.
Reworked broadcast machinery quite a bit.
It’s significantly faster now, and internal messages no longer get broadcasted to internal pages that do not request them.
This improved performance when running with many open browser tabs containing internal pages by quite a bit.
Simplified internal command handling quite a bit.
So, the external API is going to be implemented soon-ish.
This also, essentially, auto-generates implementations of many internal actions.
Core, manifest.json:
Added a bunch more useful, but unbound by default, keyboard shortcuts.
Since their actual internal implementations are now free.
Core, Capture:
Renamed fake reqres/WRR flag to request_buggy, because the old name was not at all descriptive.
The webRequest::capture::RESPONSE::BROKEN and the old webRequest::pWebArc::RESPONSE::BROKEN errors were remade into a new response_buggy reqres/WRR flag.
The migration is automatic.
Renamed sent reqres/WRR flag to submitted, since it does not guarantee that the request was actually sent, only that it was submitted to the networking pipeline.
After all, such a request can then be answered from cache, for instance.
The logs will display all of these flags now.
Also, the final networking state in the logs in prepended with a “$” symbol, to make it distinguishable from flags.
Added config options and popup UI for controlling problematic marking and picking of reqres with these flags.
From now on config.markProblematicPartialRequest and config.markProblematicBuggy will only get applied to request that were actually submitted and not from_cache, since !submitted || from_cache implies it was not actually sent.
On Chromium, request_buggy reqres will no longer get origin_url WRR field set from Referer.
This adheres to the “capture as raw as possible” philosophy better.
Documentation:
doc/data-on-disk.md, which now comes bundled with the extension too.Core, Toolbar button, Icons:
Added unsnapshottable and unreplayable icons.
From now on, they are used for displaying the states of the corresponding per-tab config options.
Added vertical bar and dot icons.
They are now used for separating per-tab and per-tab-children toolbar icon animation frames, when they differ.
This makes things much less confusing.
Core, Default config:
config.animateIcon value a bit, to make those icons easier to visually parse.Popup UI:
Renamed Run tab/tag to RT, Class to CLS, moved UI tab/tag to right after CLS and put spacers at better points.
Personally, I misclick less with this layout.
Moved all snapshottable and replayable toggles to the end of their sections.
When viewed from the Help page, per-tab options are now available and settable again.
This makes no technical sense, but this is less UI-surprising.
On Fenix, improved the layout slightly.
Core, Popup UI:
config.debugging into a bunch of more fine-grained configuration options.Documentation:
Improved the Help page.
Improved keyboard shortcut descriptions.
Improved popup UI help strings.
Core:
Fixed a bunch of small persistence-related internal bugs.
As a result, config and persistent stats (aka globals) won’t no longer get written to disk when unchanged.
Fixed globals describing reqres saved in local storage from getting out of sync with reality.
The inconsistent states will fix themselves when you re-archive everything, or if you open the Saved in Local Storage page, disable the view limit, and let it churn through everything once.
Or you can just ignore it, it’s not like this matters much, and it will fix itself as soon as you do something useful with it.
On Fenix, fixed the extension’s config auto-fixing sometimes actually producing broken archival settings.
Which were easy to manually fix, but still.
Core, Popup UI:
Canceled actions will no longer appear in the Running actions list.
They were never actually run, just appeared there for a bit for technical reasons. It no longer works that way.
Running actions will no longer appear in the Scheduled actions list.
Internally, this is how it works still, but the UI will filter them out now, to make it less surprising.
All internal pages:
Core, Popup UI:
Implemented config.limboNotifyInterval option and its popup UI.
It prevents “Too much stuff in limbo” notifications from being generated more frequently than given number of seconds, which makes them much less annoying.
Core:
From now on, retryUnarchived scheduled action can be canceled (until a new reqres gets captured) with the cancelActions (red square) button.
Unavailable and non-archiving servers produce different notification error messages and are handled differently now.
Archivals to broken servers and localStorage produce nicer notification errors now.
Unrecoverable archival failures now get a hint about using the retryFailed popup UI button in their notification messages.
From now on, pressing keyboard shortcuts and popup UI buttons will force immediate updates to displayed notifications.
I.e. no longer relevant notifications will disappear immediately.
(Thanks to @douglasg14b on GitHub for pointing out three out of five of the above.)
Added “Table of Contents” section.
Moved and adapted a bunch move stuff from top-level README.md.
Improved some pieces of it.
Other documentation pages:
Improved rendering of various references, added more of them.
Improved in many random places.
All documentation pages:
Clicking on internal links will now replace history states that point to other links.
This makes the use of “Back” browser button more intuitive.
Core:
On Firefox, fixed an issue when, with a popup UI open, opening the Internal State page would make the popup show the wrong tab stats until a tab switch.
On Firefox, fixed popup UI’s tab stats flickering between inconsistent states when, with a popup UI open, a new tab gets opened, starts loading, and then gets redirected, with all of it happening really fast.
*:
.part files not being properly filtered out from the inputs. (#10 on GitHub. Thanks to @douglasg14b on GitHub for reporting it!)fsyncing things out of proper order sometimes.*:
.part files should be automatically deleted if hoardy-web encounters an FS error.Documentation:
*:
wine..part filtering and archival with serve.*:
black.pylint.kisstdlib.kisstdlib.serve:
stderr. The latter can be disabled by --quiet.Documentation:
*:
.css and .mjs files.WRR files.serve and, likely, organize:
*:
black.pylint.Documentation, Metadata:
*:
--version option works again.serve:
When remapping URLs, hash/fragment parts will be preserved now.
When replaying, HTTP status codes will be preserved now.
From now on, by default, serve will replay archived HTTP headers over HTTP, instead of inlining them into rendered HTML documents (see below).
When both archiving and replaying, newly dumped reqres that fail given input filters will no longer be indexed and made available for replay.
mirror, serve:
Reqres containing redirects (e.g. 302 Found) are handled properly now.
From now on, implicit favicons will be mirrored and replayed properly (see below).
*:
--*grep* filtering of headers with multiple values.scrub, mirror, serve:
Added inline_headers option, making inlining of headers as meta http-equiv tags optional.
Implemented inline_fallback_icon option.
When enabled, this option adds a fallback <link rel=icon href="/favicon.ico"> to the result when the input declares no icons and that URL remaps to something useful.
This option is then enabled by default, thus fixing replay of implicit favicons.
serve:
Implemented --web and --mirror options which control how headers should be replayed.
With --web enabled, serve will evoke scrub with -inline_headers and will replay those headers over HTTP instead.
With --mirror it will continue to use scrub with +inline_headers, like mirror does.
From now on, --web is the default.
Implemented --oldest and --nearest options, similar to those of mirror.
Added more namespaces other than /web/ and made serve use them for different kinds of targets when remapping.
So that, e.g., links pointing to unavailable URLs get remapped to /unavailable/<date>/<url> and links pointing to redirects get remapped to /redirect/<date>/<url>.
This makes links much more informative when hovering other them or when looking at the log output of serve.
For replay, however, all those namespaces are equivalent and can be used interchangeably.
serve, mirror:
Renamed --ignore-bad-inputs -> --ignore-some-inputs.
Changed default input filters to allow reqres containing redirects.
mirror:
Added a default value for the root filters, which is --root-status-re ".[23]00C" to prevent redirects being added as roots.
Added --queue-all-indexed option to make the previous item optional.
Changed (simplified) semantics of the --boring option.
From now on, making a path --boring simply disables queuing of its reqres as roots.
This allows for more interesting uses.
Documentation:
Popup UI:
Reorganized the whole layout by assigning tags to all elements and allowing switching between those tags as if they were tabs.
The original idea was to unroll in steps a-la uBlock Origin, but this is superior.
Improved some help strings.
Core, Popup UI, Shortcuts:
Added Replay from the archiving server configuration option.
It’s a tristate of: disallow, enable if Submit dumps via 'HTTP' option is enabled and the server supports it, enable even if Submit dumps via 'HTTP' option is disabled.
Added Include in global replays per-tab options.
Added popup UI button and keyboard shortcut both of which re-navigate all tabs for which Include in global replays is set to their replays.
Added popup UI button, keyboard shortcut, and context menu item all of which re-navigate currently active tab to its replay.
Added Force 'Work offline' in replayed tabs configuration option which does the same thing the similar options for file: and data: URL does, but for tabs that point to replay URLs. Enabled by default.
Added 🎄 Winter Days mode seasonal theme.
Added Escape notification messages configuration option to help support more notification daemons. Disabled by default.
Merged “Handling of failures” section into “Archival”.
Reworded some awkward places.
Core, Shortcuts, manifest.json:
Improved server checking logic and error messages.
Improved keyboard shortcut descriptions.
Documentation:
Core:
Snapshot buttons and keyboard shortcuts will no longer take DOM snapshots of replay pages, unless Capture snapshots of all URLs option is set.
On Chromium, fixed Hoardy-Web trying to collect and archive replay pages.
This release integrates the extension with tool-v0.20.0, which can now do both archival and replay over HTTP, see below.
Core:
From now on, all requests to all URLs under Server URL will be ignored, allowing you work with tool-v0.20.0-replayed pages without fiddling with any settings.
From now on, the extension will respect archiving server’s settings and features given by its /hoardy-web/server-info endpoint, if such a thing exists.
The default value of Server URL does not specify /pwebarc/dump endpoint anymore, as this is now configurable server-side.
For old configs, you can keep the old value, the archiving server handling code will silently elide that path away.
From now on, before the first archival, the extension will check that a working archiving server is available at the given Server URL and generate errors describing what exactly appears to be broken when not.
Popup UI:
From now on, if you set Server URL setting to an empty string, it will be reset to the default value.
Improved CPU usage when switching tabs really quickly.
HTTP, mirroring of non-GET reqresexport mirror:
export mirror sub-command to just mirror.*:
--no-overwrites options -> --no-overwrite.*, --expr:
source -> agent.raw_path_parts -> path_parts.mq_raw_path -> mq_path.qsl_urlencode atom -> unparse_query.*:
Improved URL normalization:
This changes the file names generated by organize with the default --output format a bit.
serve:
Implemented the serve sub-command, which runs hoardy-web as a web server that can replay archived data over HTTP, a-la heritrix and pywb.
After starting it with something like hoardy-web serve path/to/your/archives, you can then navigate to
This is very reminiscent of the Wayback Machine by design, yes.
Added /hoardy-web/server-info endpoint support for future integration with the extension, similar to that of simple_server (hoardy-web-sas) now does.
Implemented archiving server support when running serve with --archive-to option.
This is similar to simple_server, except newly archived reqres will become available for replay immediately.
Implemented archiving-server-only mode when running serve with --no-replay option.
In this mode, it is essentially equivalent to simple_server, except hoardy-web serve supports arbitrary --output formats.
Implemented --latest option, which only indexes and allows replay for the latest available visit to each available URL.
Archiving new reqres updates the index accordingly, as expected.
mirror:
Implemented rendering of non-GET reqres.
So, e.g., DOM snapshots and web search answer pages done via POST will be included in the outputs now.
If you do not want some of those, you can filter them out with --not-method POST or some such.
scrub, mirror, serve:
From now on, malformed URLs will be kept as-is instead of being voided out.
From now on, more types of IE-pragmas will be censored out by -iepragmas (which is the default).
From now on, scrub will use +verbose and +whitespace as defaults.
This is a much nicer default, and after content-addressed outputs were implemented in tool-v0.19.0, the resulting space savings -verbose,-whitespace produce are mostly inconsequential now.
Simplified semantics of (+|-)pretty, it does not set the verbose option anymore.
*:
--structure and --raw-qbody options.Documentation:
mirror:
Changed semantics of --nearest option a bit. From now on, it will parse its argument as a time interval and then take the middle of it as the target value.
This is much nicer in practice since, from now on, giving --nearest 2024 is much less likely to get you the stuff from 2023. It will try to give you stuff nearest to 2024-07-02 00:00:00 instead.
Improved performance.
*:
--no-remap option -> --raw-sbody, the old name is kept as an alias.Documentation:
Improved documentation and help strings.
Most notably, the input filtering options are shown only once now.
Metadata:
pyproject.toml now explicitly specifies optional mitmproxy file format support.* in theory, but only ever triggered by mirror:
*:
Added -t, --to, and --archive-to aliases for --root.
Added /hoardy-web/server-info endpoint for future integration with the extension.
*:
From now on, “/” and most other non-word symbols (except "_“,”-", and space) in bucket names are forbidden and will be removed.
This will simplify some future things.
From now on, when several buckets are specified via several profile query parameters, the last one will be used.
Renamed --uncompressed -> --no-compress, the old name is kept as an alias.
Slightly improved performance.
Build:
mypy.*:
--expr expressions, sha256 function changed semantics. From now on it returns the raw hash digest instead of the hexadecimal one. To get the old value, use sha256|to_hex.* except organize --move, organize --hardlink, organize --symlink, get, and run:
From now on, all sub-commands except for above can take inputs in all supported file formats.
I.e., you can now do
on mitmproxy dumps without even importing them first.
By default, the above commands now also automatically dispatch between loaders of different file formats based on file extensions. So you can mix and match different file formats on the same command line.
Added a bunch of --load-* options that force a specific loader instead, e.g. --load-wrrb, --load-mitmproxy.
*:
Added a ton of new filtering options.
For example, you can now do:
hoardy-web find --method GET --method DOM --status-re .200C --response-mime text/html \
--response-body-grep-re "\bPotter\b" ~/hoardy-web/rawAs before, these filters can still be used with other commands, like stream, or export mirror, etc.
Also, the overall filtering semantics changed a bit. The top-level logical expression the filters compute is now a large conjunction. I.e. the above example now compiles to, a bit simplified, (response.method == "GET" or response.method == "DOM") and re.match(".200C", status) and (response_mime == "text/html") and re.match("\\bPotter\\b", response.body).
Added a bunch of new --output formats. Mostly, this adds a bunch of output formats that refer to stimes. Mainly, to simplify export mirror --all usage, described below.
export mirror:
Implemented mirroring of different URL visits.
I.e., you can now mirror not just --latest visit to each URL, but an --oldest one, or one --nearest to a given date, or --all of them.
Implemented --latest-hybrid, --oldest-hybrid, and --nearest-hybrid options.
These allow you to mirror each page with resource requisites that are date-vise closest to the stime of the page itself, instead of taking globally --latest, --oldest, or --nearest versions of all requisite URLs.
At the moment, this takes a lot more memory, but makes the results much more consistent for websites that do not use versioned resource requisites.
Implemented --hardlink and --symlink options, which allow mirroring into content-addressed destinations.
I.e. export mirror --hardlink will render and write each mirrored file to <--to>/_content/<hash/based/path>.<ext> and only then hardlink the result to <--to>/<output/format/based/path>.<ext> target destination. And similarly for --symlink.
This saves quite a bit of space when pages refer to the same resource requisites by slightly different URLs, same images and fonts get distributed via different CDN hosts, when you mirror --all visits to some URLs and many of those are absolutely identical, etc.
So, from now on, --hardlink is the default. The old behavior can be archived by running it with --copy instead.
Implemented --relative and --absolute options, which control if URLs should be remapped to relative or absolute file: URLs, respectively.
Documentation:
Build:
test-cli.sh tests.export mirror:
--root-* options now use the same syntax and machinery and support the same filtering options as the normal input filters.
Switched default --output to hupq_n to prevent collisions when using --*-hybrid and --all.
Improved handling of base HTML tags, target attributes are supported now.
Links that reference a page from itself will no longer refer to the page’s filename, even when the link has no fragment.
The results can be a bit confusing, but this makes the new content de-duplication options much more effective.
Made export mirror default filters explicit and changed them from --method "GET" --status-re ".200C" to --method "GET" --method "DOM" --status-re ".200C".
Implemented --ignore-bad-inputs and --index-all-inputs options to allow you to change the above default.
Improved output log format.
*:
Documentation:
*:
Build:
organize, which cover the organize --symlink --latest bug of tool-v0.18.0. Won’t happen again.tool-v0.18.0 introduced a bunch of issues:
organize:
Fixed organize --symlink --latest dereferencing output files, which lead to it overwriting plain WRR source files containing updated URLs with symlinks to their newer versions.
The good news is that this bug was only triggered when organize --symlink --latest was run with some newly archived data and, for each updated URL, it only overwrote the second to last WRR file with a symlink to the latest WRR file. Unfortunately, this error was self-propagating, so those files could then get overwritten again by the next invocation of organize --symlink --latest with some more new data. This could happen up to 7 times, at which point it would start crashing, because of the OS symlink deferencing limit.
You can check if you were affected by running:
The paths it outputs will be the paths of lost WRR files.
A reminder that it is good to do daily backups, I suppose.
The next version will have a test for this, but I’m releasing this hotfix an hour after I discovered this.
Fixed it assert-crashing sometimes when running with --symlink.
Improved memory consumption a bit.
export mirror:
export mirror:
Implemented the --boring option, which allows you to load some input PATHs without adding them as roots, even when no --root-* options are specified.
This make CLI a bit more convenient to use. The README.md has a new example showcasing it.
export mirror, scrub:
Implemented support for @import CSS rules using a string token in place of a URL.
As far as I can see, this syntax is rarely used in practice, but the spec allows this, so.
Implemented interpret_noscript option, which enables inlining of noscript tags when scrub is running with -scripts.
That is, export mirror will now use this feature by default.
This is needed because some websites put link tags with CSS under noscript, thus making such pages look broken when scrubbed with -scripts (which is the default) and then opened in a browser with scripts enabled.
*: Refactored/reworked a large chunk of internals, as a result:
organize can now take WRR bundles as inputs too,export mirror became much faster at indexing inputs that contain archives of the same URLs, repeatedly.In general, these changes are aimed towards making hoardy-web completely input-agnostic. That is, wouldn’t it be nice if you could feed mitmproxy files to export mirror directly, instead of going through import mitmproxy first?
export mirror, scrub:
From now on, it will stop generating link tags with void URLs, it will simply censor them out instead.
scrub with +verbose set will now also show original rel attr values for censored out tags.
Also, in general, the outputs of scrub with +verbose set are much prettier now.
Documentation:
* except organize, get, and run:
All WRR-processing sub-commands except for above can now take WRR bundles as inputs.
That is, you can now directly do
hoardy-web pprint ~/Downloads/Hoardy-Web-export-*.wrrb
hoardy-web export mirror --to ~/web/mirror ~/Downloads/Hoardy-Web-export-*.wrrb
# etcwithout needing to run hoardy-web import bundle first.
Though, at the moment, export mirror will stop respecting --max-memory option for such inputs.
export mirror, scrub:
Implemented support for old-style HTML pages using frameset and frame HTML tags.
Implemented support for stylesheets stored as data: URLs stored in hrefs of link tags.
Yes, this is actually allowed by the specs and the browsers.
*:
MIME types, Content-Type, and Link headers is much more forgiving towards malformed inputs now.export mirror, scrub:
Links pointing to an id on same HTML page will now get emitted as #<id>, not ./<file>#<id>.
Refresh headers with non-HTTP URLs will get censored away now.
Improved error messages in cases when an HTTP header failed to parse.
Improved performance a little bit.
export mirror, scrub:
From now on, scrub will simply drop HTML tag attributes when all URLs in their values get censored away.
Previously, it produced attributes with void URLs instead.
This makes a huge difference for src and srcset attributes of HTML img tags where, before, generated pages plugged void URLs for missing sources, which sometimes confused browsers about which things should be used to display stuff, breaking things.
*:
HTMLs with Byte Order Marks will no longer get mimesniffed as text/plain.
Fixed parsing of quoted MIME parameter values.
Fixed various crashes when processing data generated by the extension running under Chromium.
*:
CSS color scheme.Documentation:
*:
Popup UI:
Reverted most of the block reordering bit of popup UI rework of extension-v1.17.0.
The “Globally” block is near the top again.
Edited the “Persistence” block a bit more.
Mainly, to stop graying out always-useful stat lines, even when the associated features are disabled.
This prevents possible confusion of what buttons can be used when.
Renamed some options and stat lines, mostly to make their names shorter to make popup UI on Fenix more readable.
Toolbar button:
Edited its title format to be much shorter, especially on Fenix.
Reverted the ordering of parts there to how it was before extension-v1.17.0.
The (much shorter now) “globally” part is at the front again because otherwise the badge being at the front there too without an explanation of its format is kind of confusing.
Core, All internal pages:
Improved internal async message handling infrastructure, making things slightly more efficient.
Improved initialization functions of all internal pages, making them more efficient and making the resulting UI much less jiggly when changing zoom level and/or jumping around between pages.
*:
build.sh firefox target to firefox-mv2, for consistency.UI:
Fixed flaky rendering of Help and Changelog pages on Fenix.
They render properly now the very first time you load them, no reloads needed.
Fixed duplication of history entries when navigating internal links.
Fixed source links sometimes failing to being highlighted when pressing the browser’s “Back” button.
Fixed some small CSS nitpicks.
Popup UI, Documentation:
*:
webNavigation permission is used for, improved the rest a bit.Work offline mode, add-on reloading with its state preserved, new options, etcIn related news, I have 💸☕ a Patreon account now.
Core:
Fixed a bug in upgradeConfig that was resetting bucket settings to their default values or upgrade to extension-v1.13.0. So, this is no longer relevant, but still. Also, refactored code there to prevent such errors in the future.
However, just in case, if you previously set bucket settings to something other than their default values and those settings are important to you, you should probably check your settings to ensure everything there is set as you expect it to be.
Core, Popup UI, Documentation:
Renamed failed state and related failed* stats to unarchived state and unarchived* stats. Introduced a new failed stat that is now a sum of unstashed and unarchived stats. Edited the popup UI and the other pages appropriately.
This makes documentation’s terminology more consistent, and simplifies UI a bit.
In particular, the Retry button of Queued/Failed stat line will both retry stashing unstashed and archiving unarchived reqres now.
Popup UI:
Reworked the whole thing quite a bit:
From now on, setting Bucket for the current tab will set Bucket for its new children too, similar to how the rest of those settings work.
From now on, setting any of the Bucket settings to nothing will reset it to the parent/default value. I.e.:
Bucket of This tab's new children to nothing will reset it to Bucket value of This tab.Bucket of This tab to nothing will reset it to Bucket value of New root tabs.Bucket of New root tabs to nothing will reset it to default.The previous “Desktop” JavaScript-generated layout became columns CSS layout and JS-operation mode, while the “Mobile” JavaScript-generated layout became linear CSS layout and JS-operation mode. The page will now automatically switch between these two layouts and modes synchronously, depending on viewport width.
(As before, in linear mode hovering over a link does nothing, but in columns mode, hovering over a link referring to a target in popup UI scrolls the popup UI column to that target and highlights it.)
I.e., this means that on a Desktop browser, you can now zoom the Help page to arbitrary zoom levels and it will just switch between layouts and link-hover behaviors depending on available viewport width.
Greatly improved the styling of all links and documented it on the Help page.
All internal pages:
All internal pages now color-code links depending on where they point to, using exactly the same CSS as the Help page.
All pages now use the same history state handling behaviour.
I.e., using the “Back” button of your browser will now not only go back, but also highlight the last link you clicked.
All documentation pages now set viewport width to device-width, set content’s max-width to 900px and width to 100% - padding, preventing horizontal scroll, when possible.
Improved the CSS styling in general.
Core, Popup UI, General UI:
Implemented a new popup UI tristate toggle named Color scheme which allows Hoardy-Web’s color-scheme to be different from the browser’s default.
Implemented a mechanism and popup UI settings for applying additional themes and experimental features.
And then I looked at the date. Which is why ◥▅◤◢▅◣◥▅◤ Hoardy-Web now has 🦇 Halloween mode. ◥▅◤◢▅◣◥▅◤.
Also, from now on, the neutral states of tristate toggles are displayed with toggle knobs being in the middle of the things, not on their left. This is not a political statement. This mans that all tristate toggles, from left to right, now go false -> null -> true both internally (exactly as they did before) and externally (which is new).
Core, Toolbar button, Icons:
Replaced toolbar button’s icons representing Cartesian products of other icons with animations.
In other words, the previous “this tab has limbo mode enabled while this tab’s children do not” icon will now instead be represented with an animation that switches between “this tab has limbo mode enabled” and “this tab is idle” icons instead.
This both takes less space in the XPI/CRX, makes for a cuter UI, and is the only reasonable solution when the core wants to display more than two icons at the same time.
Improved toolbar button’s badge and title format a bit.
“This tab” part goes first now, then “its new children”, then “globally”.
Also, the order of sub-parts of those strings is more consistent now.
From now on, internal UI updater will generate icon animation frames for all important statuses and setting states.
main icon will be inserted at the end to make it obvious when the animation loop restarts (otherwise, it’s easy to interpret such animation loops incorrectly).The update frequency of toolbar button’s icon, badge, and title now depends on the amount of not yet done stuff still queued in the core.
I.e., from now on, when the core has a lot of stuff to do (like when re-archiving thousands of reqres at the same time), it will start updating toolbar button’s properties less to trade update latency for improved performance, and vice versa.
Greatly improved performance of state display updates. It’s uses 2-1000x less CPU now, depending on what the core is doing.
Icons:
Renamed the error icon to failed and added a new error icon.
From now on, the failed icon will only be used for archival/stashing errors, while the error icon will only be used for internal errors (i.e. bugs).
Improved all icons to make them more visually distinct when they are being rendered at 48x48 or less, both in light and dark mode.
On Chromium, all icons are now rendered with transparent backgrounds, so now they will look nice in the dark mode too.
Core, Popup UI, Toolbar button:
From now on, popup UI and toolbar button’s badge and title will display information about currently running internal actions.
(Implementing this took a surprising amount of effort in improvements to infrastructure code.)
Core, Toolbar button, Icons:
in_limbo icon for “this tab has data in limbo” status. Unlike most other icons, this icon will never be used alone, it will always be an animation frame of something longer.Core, Popup UI, Toolbar button:
Animate toolbar icon every setting for controlling toolbar icon animation speed.Core, Toolbar button:
Fixed a bunch of bugs that prevented updates to toolbar button’s icon and badge in some cases.
The icon and the badge will no longer get stuck when the core is very busy, like when re-archiving a lot of stuff all at once.
Work offline modeCore, Popup UI, Toolbar button, Icons, Documentation:
Implemented Work offline mode, options, their popup UI, shortcuts, and icons.
This mode does the same thing as File > Work Offline checkbox of Firefox, except it supports per-tab/per-source operation, not just the whole-browser one. Also, enabling any of these options will not break requests that are still in flight, and the requests they do cancel can be logged.
That is, enabling Work offline in a tab will start canceling all new requests that tab generates, and the resulting canceled reqres will get logged if Track new requests option is enabled in the same tab. Similarly for background tabs and other sources.
This can be generally useful for debugging your own websites with dynamic responsive CSS, or if you just want to prevent a tab from accessing the network for some reason.
However, the main reason this exists is that the files generated by hoardy-web export mirror do not get scrubbed absolutely correctly at the moment, and the resulting pages can end up with some references to remote resources (in cases when an mirrored page uses some rare HTML and CSS tag combinations, or lazy-load images via JavaScript, but still). With Work offline options enabled in a tab, you can now be sure that opening pages generated by hoardy-web export mirror won’t send any requests to the network.
In fact, from now on, by default, Hoardy-Web will enable Work offline in all tabs pointing to file: URLs. This can be disabled in the settings.
Added a new offline toolbar icon to display the above state.
Documentation:
Help page.Core, Popup UI, Documentation:
Implemented reloadSelf action that reloads the add-on while preserving its state.
This action is different from similar Reload buttons in browser’s own UI in that triggering this action will reload the add-on while preserving its state. Meanwhile, using the browser’s buttons will reset everything and loose all reqres that are both unarchived and unstashed.
Added a popup UI button for triggering this action. (The button is only shown when a new version is available, unless debugging is enabled.)
Implemented Auto-reload on updates setting to automate away clicking of that button on updates. Though, it is currently disabled by default, because this feature is a bit experimental at the moment.
That is, from now on, after the browser notifies the add-on that it ready to be updated, the popup UI will display a button allowing you to reload it, so that the browser could load the new version instead. Alternatively, you can now enable Auto-reload on updates and it would do that automatically.
Documentation:
Help page.Core, Popup UI:
Implemented Export via 'saveAs' > Bundle dumps option as separate toggle instead of forcing you to set the maximum size to 0 to get the same effect.
Implemented Include in global snapshots per-tab/per-source setting.
I.e., you can now exclude specific tabs from being included in all-tab DOM-snapshots even when Track new requests option is enabled.
Notifications, Popup UI:
Implemented Notify about 'problematic' reqres per-tab/per-source setting.
I.e., you can now exclude specific tabs from generating notifications about problematic reqres even when Generate notifications about > ... new 'problematic' reqres option is enabled.
Notifications, Documentation:
From now on, clicking an error notification will open the relevant section of the Help page while doing nothing for other notifications.
Also, improved that section a little bit.
Documentation:
Core:
On Chromium, there will no longer be duplicates in reqres errors lists.
From now on, when you close a tab, all in-flight reqres in it will be emitted with *::capture::EMIT_FORCED::BY_CLOSED_TAB error set. Before, some of them sometimes finished with webRequest::*_ABORT errors instead.
Refactored a lot of internal stuff, simplifying the ways in which many internal things are done.
Icons:
Core, Documentation:
Fixed handling of interactions between page scrolling, node hilighting, and help tooltips.
I.e., on the Help page, highlighting an option in the popup UI by hovering over a link there, and then clicking on the help tooltip of the highlighted option will no longer make the UI look weird.
Core, Notifications:
unarchived reqres from being closed automatically.scrub, export mirror:
Implemented inlining of Link, Refresh, Content-Security-Policy, and some other HTTP headers into the mirrored HTML files as meta http-equiv tags.
scrub now has (+|-)navigations option which controls whether the resulting meta http-equiv=refresh headers should be kept or censored out, -navigations is the default.
Also, CSP headers are not supported yet and, thus, the generated meta http-equiv=content-security-policy tags will get immediately censored out, which is usually invisible, but can be seen with +verbose set.
export mirror:
--max-memory option, allowing you to sacrifice arbitrary amounts of RAM to improve performance.Build:
Added unit tests for all internal parsers.
Added a lot of new integration tests.
export mirror:
Improved the mirroring algorithm, switched to a completely recursive implementation, in preparation for future extensions with cool features.
From now on, writes to all files that are being mirrored (not just the top-level ones) will be atomic with respect to their dependencies.
*:
Refactored internals a lot.
Improved performance a bit.
Core, Capture:
On Chromium, fixed request tracking being frequently broken since extension-v1.15.0.
Fixed reqres without responses but with networking errors having “Responded at” field set in the logs.
get, export mirror, etc:
Restricted the idna workaround of tool-v0.15.4 to hostnames with “–” in [2:4] character positions.
The previous iteration made parse_url start accepting many malformed URLs.
URL parsing will now strip hostnames of leading and following whitespace, like browsers do.
Mainly, this improves export mirror outputs.
Fixed output formatting when redirecting output to a non-tty destination.
*:
File path parts starting with “.” in profiles (i.e. buckets) given by clients are ignored now.
This prevents escapes from the given --root.
The previous behaviour was not really a security issue, given that the server is not designed to be run with untrusted clients, and filenames are generated by it, not the clients. But still.
Renamed command-line options:
--no-print-cbors -> --no-print--default-profile -> --default-bucket--ignore-profiles -> --ignore-bucketsThis is makes them use the same terminology the extension uses.
Old names are kept as aliases.
*:
version endpoint, for extensibility.get, export mirror, etc:
idna module failing to parse some hostnames (#5 on GitHub. Thanks to @philippelatulippe on GitHub for reporting it!).find:
Added --sniff-* options to fix crashes introduced in v0.15.0.
Added tests to hopefully stop this kind of errors.
get, export mirror, etc:
--expr: technically, renamed full_url -> url, though it did not officially exist before.
Added it to the docs.
ftp and ftps URL schemes are now allowed everywhere.
scrub, export mirror:
From now on scrub will simply remove all CORS and SRI attributes from all relevant HTML tags.
This works fine 99% of the time. Smarter handling for this will be implemented later.
Fixed MIME type sniffing of XHTML data.
Also, added some tests for my mimesniff implementation.
Fixed crashes when URL remapper encounters weirdly malformed URLs.
From now on they will be remapped into void URLs instead.
export mirror:
Fixed it skipping regular files given directly as command line arguments.
This was broken since v0.15.0.
scrub, import, Documentation:
Documentation:
scrub, export mirror:
Fixed a stupid bug in MIME detection code that prevented external CSS files from being detected as such.
Added tests to prevent such things in the future.
Fixed CSS formatting with +whitespace set.
Made scrub remove crossorigin attributes from HTML tags for which it remapped a URL.
This seems to have fixed most of issues causing pages produced by export mirror looking broken when opened in a web browser.
Documentation:
export mirror:
import *:
--sniff-* options to fix crashes introduced in v0.15.0.export mirror:
depth value in the UI.not remapping '%s' lines.export mirror sub-command now produces results quite usable in a normal web browser. I.e. it is now comparable to, say, what Single-File produces.
Feature-wise, it reaches a Pareto front, AFAICS, since no other tool I know of can do efficient (with shared page requisites) incremental static semi-open (see --remap-semi option below) website mirrors.
At the moment, scrubbed CSS can get a bit broken sometimes, because hoardy-web leans in favor of its results being safe to use, not them being as close to the original as possible. Also, support for audio, video, and source HTML tags is still a bit quirky. But the current state is quite usable.
scrub, export mirror:
Implemented stylesheet (CSS) scrubbing with the help of tinycss2.
I.e., requisite resource URLs mentioned in stylesheets will now be properly remapped.
I.e., website mirrors will be styled now.
export mirror:
Added --remap-semi option, which does the same thing as --remap-open (which is equivalent to wget --convert-links), except it remaps unavailable action links and page requisites to void URLs, making the resulting generated pages self-contained and safe to open in a web browser without it trying to download something.
I.e., --remap-semi does what wget --convert-links should be doing, IMHO.
Added --root-url-prefix and --root-url-re options.
pprint, get, run, stream, export mirror:
Implemented --sniff-* options controlling mimesniff algorithm usage.
For pprint sub-command they replace --naive and --paranoid options.
--expr, --output:
pretty_net_url, pretty_net_nurl, raw_path_parts, and mq_raw_path atoms.scrub, export mirror:
Changed the way all --remap-* options are implemented. Most of the remapping logic was moved into the scrub function. --remap-* options simply change default values of the corresponding --expr options now.
+styles and +iframes options are now set by default.
Since these things can now be properly mirrored.
Renamed (+|-)srcs options to (+|-)reqs to follow the terminology used by wget.
In documentation, “page resources” became “requisite resources” and “page requisites”.
Improved censoring for IE-pragmas.
Improved +indent and +pretty output layout a bit.
Improved +verbose output format a bit.
export mirror:
Renamed --root option to --root-url, -r and --root options now point to --root-url-prefix instead. The --root option name is deprecated now and will be removed in the future.
Improved progress reporting UI.
It’s much prettier and more informative now.
It ignores duplicate input paths now.
This allows to easily prioritize mirroring of some files over others by specifying them in the command line arguments first, followed by their containing directory in a later argument.
README.md has a new example showcasing it.
It delays disk writes for HTML pages until after all of their requisite resources finished mirroring now.
I.e. newly generated HTML pages can now be opened in a web browser while export mirror is still running, having not finished mirroring other things yet.
*:
MIME type handling a bit, added text/vtt recognition.--expr, --output:
Renamed: path_parts -> npath_parts, mq_path -> mq_npath.
Changed semantics of net_url and pretty_url a bit.
Both add trailing slashes after empty raw_paths now. Also, pretty_url does not normalize raw_path now, i.e. now it only re-quotes path parts, but does not interpret . and .. path parts away.
Documentation:
scrub, export mirror:
Fixed generation of broken file: links for URLs with query parameters.
From now on stylesheet, icon, and shortcut links are treated as page requisites.
Fixed a bug where export mirror with --depth set would forget to mirror shortcut icons and CSS files.
Fixed a bug where export mirror with --depth and --remap-(open|closed) set would fail to remap unreachable URLs properly.
Fixed some places where the code was misaligned with the documentation. Most importantly, scrub and export mirror are --verbose by default now, which documentation claimed they were, but they were not.
Documentation:
*:
pWebArc -> Hoardy-Web.::pWebArc:: error codes into a more consistent naming scheme.Documentation:
*:
wrrarms -> hoardy-web.*:
dumb-dump-server -> hoardy-web-sas.Core, Notifications:
Popup UI, Popup UI:
get:
--expr-fd option, which allows you to extract multiple --expr values from the same input file to different output file descriptors in a single wrrarms call../script:
Improved all the scripts by adding usage descriptions and --help options to all of them.
Added --to option to wrrarms-pandoc script. It allows you to change the output format it will use.
Added wrrarms-spd-say script, which can feed contents of an archived HTML document, extracted from HTML via pandoc -t plain, to speech-dispatcher’s spd-say, i.e. to your preferred TTS engine.
Added --*-url options to wrrarms-w3m and wrrarms-pandoc scripts. They allow you to control how to print the document’s URL in the output.
Modified wrrarms-w3m and wrrarms-pandoc scripts to use --expr-fd option, making them ~2x faster.
export mirror:
--expr arguments.import:
--override-dangerously option.*:
--output formats.*:
Renamed all --no-output options to --no-print.
Edited --output formats, making them more consistent with their expected usage:
Edited the default, short, surl_msn, and url_msn --output formats, replacing a “.” before the num field with a “_”. Because these formats do not mention any file extensions.
Edited surl_msn and url_msn --output formats, replacing and a “_” before the method with “__”. To make these --output formats useful in programmatic usage.
Edited most other --output formats, replacing a “_” before the method field with a “.” and a “.” before the non-standalone num with a “_”. Since these --output formats do use file extensions, this turns the whole wrrarms-specific suffix into a sub-extension.
./script:
wrrarms-pandoc uses plain text --to output format by default now. The previous default was org-mode.*, Documentation:
Improved error messages.
Improved documentation.
export mirror:
--errors option value not only while indexing inputs, but also while rendering and writing out outputs.*:
--output flat_n format.Core, Capture, Popup UI, Notifications:
pWebArc is now officially supported on Fenix (Firefox for Android). It is quite usable there now, so go forth and test it.
Chromium version now has a update_url set in the manifest.json, so if you use chromium-web-store or some such, it can be updated semi-automatically now, see the extension’s README.md for more info.
Implemented User Interface and Accessibility > Verbose option.
From now on, by default, pWebArc will have its most common but annoying notifications mention they can be disabled and explain how.
This is mostly for Fenix users, where these things are not obvious, but it could also be useful for new users elsewhere.
Implemented User Interface and Accessibility > Spawn internal pages in new tabs option which controls if internal pages should be spawned in new tabs or reuse the current window.
It can not be disabled on desktop browsers at the moment, but it is disabled by default on mobile browsers.
Implemented a bunch of new notifications about automatic fixes applied to config.
I.e., it will now not just fix your config for you, but also complain if you try to set an invalid combination of options.
Implemented Generate desktop notifications about ... > UI hints option to allow you to disable the above notifications.
Core, Capture:
pWebArc will CBOR-dump all reqres fields completely raw from now on.
wrrarms learned to handle this properly quite a while ago.
This simplifies the parsing of the results and makes the implementation adhere to the stated technical philosophy more closely.
The dumps will grow in size a tiny bit, but this is negligible, since they are compressed by default by all the archival methods now.
Improved performance.
Popup UI, Internal State pages, Saved in Local Storage page:
Core, Popup UI:
Stash 'in_limbo' reqres option is being inherited by children tabs like the rest of similar options do.Build:
build.sh chromium target to chromium-mv2 in preparation for eventual chromium-mv3 support.Documentation:
Core, Popup UI:
Fixed wrong “In limbo” counts after the extension gets reloaded.
Fixed race conditions in browserAction updates.
Worked around browserAction title updates being flaky on Fenix.
I.e., you can stare at the Extensions > pWebArc line in the browser’s UI now while the browser fetches some stuff and it will be properly interactively updated.
On Firefox, fixed the id of the extension leaking into origin_url field of the very first dump of each session when Workarounds for Firefox bugs > Restart the very first request is enabled (which is the default).
Documentation, Popup UI:
Core, Capture, Popup UI, UI:
pWebArc now runs under Fenix aka Firefox-for-Android-based browsers, including at least Fennec and Mull.
Thought, Export via 'saveAs' archival method is broken there, because of a bug in Firefox. Other methods do work, though.
(Also, it is not marked as compatible with Firefox on Android on addons.mozilla.org at the moment, it probably will be in the next version.)
The above change also added a settings page (aka options_ui).
At the moment, the settings page is simply an unrolled by default version of popup UI, with per-tabs settings removed.
This is needed because on mobile browsers the main screen of the browser is not a tab and there’s no toolbar, so there’s no popup UI button there, and so the extension UI becomes really confusing without a separate settings page.
Core, Popup UI, Toolbar button:
Split in_flight stat into a sum of two numbers.
This makes things less confusing on Chromium, the Help page explains it in more detail.
Added toolbar button’s badge as a prefix to its title, changed its format a bit.
This is needed because Fenix-based browsers do not display the badge at all, so this change helps immensely there. Meanwhile, on desktop browsers this does not hurt.
Improved styling and dark mode contrast of the popup UI.
Documentation:
Improved documentation.
In particular, among other things, added a lot of new anchors to the Help page, most internal links referencing some fact discussed in another section now point directly to the relevant paragraph instead of pointing to its section header.
Core, Capture:
On Firefox, fixed capture of responses produced by service/shared workers.
Also, added a new error code for when it (very rarely) fails because of a race condition inherent in webRequest API and documented all of it on the Help page.
On Firefox, fixed HTTP protocol version detection, requests fetched via HTTP/3 will now be marked as such.
On Firefox, added yet another webRequest API error to a list of those that mark reqres response data as incomplete.
On Chromium, fixed more edge cases where reqres could get stuck in in_flight state indefinitely.
Fixed a bug where force-stopping all in-flight reqres in a single tab could also drop some of the others.
Back and Forward buttons to work properly.Documentation:
Help page.Core, Popup UI
Improved popup UI HTML layout a tiny little bit.
Changed config.history default value.
Core, Internal State pages, Popup UI:
Implemented reqres persistence across restarts.
pWebArc can now save and reload collected but not archived reqres (including those in_limbo) by stashing them into browser’s local storage. This is now enabled by default, but it can be disabled globally, or per-tab.
As a consequence, pWebArc now tracks browsing sessions and shows when a reqres belongs to an older session on its Internal State page.
Implemented two new archiving methods. pWebArc can now archive collected reqres by
generating fake-Downloads containing either separate dumps (one dump of an HTTP request+response per file) or bundles of them (many dumps in a single file, for convenience, to be later imported via wrrarms import bundle),
archiving separate dumps to your own private archiving server (the old one, the previous default, inherited on extension update),
archiving separate dumps to your browser’s local storage (the new default on a new clean install).
As a consequence, pWebArc now has a new Saved in Local Storage page for displaying the latter.
Implemented display and filtering for queued and failed reqres on the Internal State page.
Implemented tracking of per-state size totals for reqres in most states after finished.
As a consequence, popup UI will now display those newly tracked sizes.
Introduced the errored reqres state.
With stashing to local storage enabled, pWebArc will now try its best not to loose any captured data even when its archiving code fails (bugs out) with an unexpected exception. If it bugs out in the capture code, then all bets are off, unfortunately.
Core, Capture:
pWebArc will now track if an error is recoverable and will not retry actions with unrecoverable errors automatically by default.
pWebArc now follows the following state diagram:
(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body received)
| | | |
| v v v
| (no_response) (incomplete) (complete)
| | | |
| \ | |
|\---> (canceled) ----\ \ | |
| \ \ \ |
|\-> (incomplete_fc) ---\ \ \ v
| >------>---------------------------->-----> (finished)
|\--> (complete_fc) ----/ / |
| / / |
\----> (snapshot) ----/ /- (collected) <--------- (picked) <--/ |
/ ^ | |
(stashIO?) <----/ | v v
| \-- (in_limbo) <- (stashIO?) <- (dropped)
v | |
(queued) <------------------\ | |
/ | ^ \ \ \-----> (discarded) <-----/
(exported) <-/ | | \----------------\ \ ^
| | | \ \ |
| /---/ \-----------------\ \ \ |
| | | \ \ |
| v | \ \ |
|\-> (srvIO) -> (stashIO?) -> (failed) | \ |
| | ^ / \ |
| v | v | |
| (submitted) --------------> (saveIO) --> (saved) | {{!saving}}
| \ |
\-------->-----------------------------------------------/Renamed all Profile settings into Bucket, as this makes more sense.
Core, Popup UI:
Improved popup UI layout.
Added more and improved existing debugging options.
Toolbar button:
Documentation:
Core, Popup UI:
config.logDiscarded option as it is no longer needed (pWebArc has proper log filtering now).Core, Capture:
Performed a huge internal refactoring to solve constant sub-task scheduling issues once and for all.
Various small bug fixes.
import bundle:
import bundle sub-command which takes WRR-bundles (optionally gzipped concatenations of WRR-dumps) as inputs. The next version of the extension will start (optionally) producing these.*:
pprint:
Core:
pWebArc will no longer automatically reload on updates, waiting for the browser to restart or for you to reload it explicitly instead.
This way you won’t lose any data on extension updates.
Proper automatic reloads on updates will be implemented later, after pWebArc gets full state persistence support.
Popup UI:
Reverted the split between Globally and This session.
Implementing that split properly will make future things much harder, so, simple is best.
Queued stat moved to a separate line again.
It also shows the sum total of sizes of all dumps now.
Added Scheduled ... actions stat line, showing the names of actions that are scheduled.
It is hidden by default, because watching it closely while pWebArc is very busy can probably cause seizures.
Documentation:
Core, Capture:
Core, Popup UI, Shortcuts:
Implemented DOM snapshots, their popup UI, and keyboard shortcuts.
snapshotTab) and snapshot all open tabs (snapshotAll).Ctrl+Alt+S runs snapshotTab by default now.Core, Toolbar button:
Added a bunch of new toolbar icons for various tab states.
In particular, problematic state as well as mixed-capture states (e.g., disabled in this tab, but enabled and with limbo mode in children tabs) now have their own special icons.
Shortcuts:
Changed some default keyboard shortcuts:
Ctrl+Alt+A and Ctrl+Alt+C run collectAllInLimbo and collectAllTabInLimbo respectively now.Alt+Shift+D and Alt+Shift+W run discardAllInLimbo and discardAllTabInLimbo now.Popup UI:
Toolbar button, Notifications:
Documentation, Help page:
Core, Popup UI, Internal State pages:
Added a dark mode theme.
The extension will switch to it automatically when the browser asks (which it will if you switch your browser’s theme to a dark one).
Implemented some new optional UI-related accessibility config options with toggles in popup UI:
v1.9.0, with slight variations for the new color-coding).Core, Popup UI, Notifications:
Core, Internal State pages:
Added a bunch of tristate toggles for filtering the logs.
Added in-log buttons that open internal state pages narrowed to a tab associated with each log entry.
Core, Popup UI:
Added UI for internal scheduled/delayed actions/functions (e.g., saving of frequently changing stuff to persistent storage, automatic actions when a tab closes, canceling and reloading not-yet-debugged tabs on Chromium, etc):
~ or . in it and change its color, depending on the importance of the stuff that is waiting to be run.Popup UI:
Generate desktop notifications about > ... new problematic reqres option and has custom tabindexes set, for convenience.Core, Capture, Popup UI:
Added config options and popup UI toggles for picking and problematic-marking reqres with various HTTP status codes.
Implemented new config options and popup UI toggles for browser-specific workarounds. In particular, on Chromium you can now set the URL new root tabs will be reset to (still about:blank by default).
Mark reqres as 'problematic' when they finish > ... with reqres errors config option became > ... with reqres errors and get 'dropped', i.e. it is now disjoint with > ... with reqres errors and get 'picked'.
Core, Capture:
Core, Popup UI, Shortcuts, Help page:
Added a few more shortcuts:
Alt+Shift+U by default unmarks all problematic reqres globally now.Alt+Shift+S and Alt+Shift+W by default respectively collect and discard all reqres in limbo globally now.In popup UI, toggles and buttons with bound keyboard shortcuts will now get those shortcuts displayed in their tooltips.
The “Keyboard shortcuts” section of the Help page will now show currently active shortcuts (when viewed via the Help button from the extension UI).
The changes to the code there mean all the shortcuts will be reset to their default keys, but it makes stuff much cleaner internally, so.
Collecting all reqres from currently active tab’s limbo is bound to Alt+S by default now (similarly to how Ctrl+S saves the page).
Discarding all reqres from currently active tab’s limbo is bound to Alt+W by default now (similarly to how Ctrl+W closes the tab).
Unmarking all problematic reqres in the currently active tab is bound to Alt+U by default now.
Core:
Changed some config option defaults (your existing config will not get affected).
Slightly improved performance in normal operation.
Greatly improved performance when archiving large batches of reqres at once, e.g. when collecting a lot of stuff from limbo.
Notifications:
Documentation, Help page:
Core, Capture:
Core, Capture:
Fixed a whole ton of issues.
This release includes so many bug fixes that pWebArc on Chromium now actually works almost as well as on Firefox.
All leftover issues on Chromium I’m aware of are consequences of Chromium’s debugging API limitations and, as far as I can see, are unsolvable without actually patching Chromium (which is unlikely to be accepted upstream, given that patching them will make ad-blocking easier).
archiveweb.page project appears to suffer from the same issues.
Meanwhile, pWebArc continues to work exceptionally well on Firefox-based browsers.
Core, Capture, Popup UI:
Implemented “negative limbo mode”.
It does the same thing as limbo mode does, but for reqres that were dropped instead of picked. (Which is why there is an arrow from dropped to in_limbo on the diagram below.)
Implemented optional automatic actions when a tab gets closed.
E.g., you can ask pWebArc to automatically unmark that tab’s problematic reqres and/or collect and archive everything belonging to that tab from limbo.
Implemented stat persistence between restarts.
You can brag about your archiving prowess to your friends by sharing popup UI screenshots now.
Core, Notifications:
Implemented a bunch of new desktop notifications.
Added a bunch of new configuration options.
This includes a bunch of them for controlling desktop notifications.
Core, Shortcuts, Internal State pages:
Added a bunch of new keyboard shortcuts.
Also, keyboard shortcuts now work properly on narrowed Internal State pages.
Documentation:
Changelog page, which can be viewed by clicking the version number in the extension’s popup.Core, Capture:
pWebArc now follows the following state diagram:
(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body received)
| | | |
| v v v
| (no_response) (incomplete) (complete)
| | | |
| \ | |
|\---> (canceled) -----\ \ | |
| \ \ \ |
| \ \ \ v
|\-> (incomplete_fc) ----->----->---------------------------->-----> (finished)
| / / |
| / /-----/ |
\--> (complete_fc) ----/ /--------------- (picked) <---/ v
| | (dropped)
v v / |
(archived) <- (sIO) <- (collected) <------- (in_limbo) <---------/ |
| ^ | |
| | | |
/------/ \-----\ \--> (discarded) <---/
| |
\-> (failed to archive) -/Terminology-wise, most notably, picked and dropped now mean what collected and discarded meant before.
See the Help page for more info.
Core, Popup UI, Internal State pages, Documentation:
Core, Popup UI, Toolbar button, Help page, Internal State pages:
The version number button in the popup (which opens the Changelog) will now get highlighted on major updates.
Similarly, the Help button will now get highlighted when that page gets updated.
All the toggles in the popup are now color-coded with their expected values, so if something looks red(-dish), you might want to check the help string in question just in case.
Extension’s toolbar button icon, badge, and title are much more informative and consistent in their behaviour now.
The popup, the Help page, and the Internal State pages all had their UI improved greatly.
Documentation:
export mirror:
--no-overwrites, --partial, and --overwrite-dangerously options.export mirror:
Switched the default from --overwrite-dangerously (which is what export mirror did before even if there was no option for it) to --no-overwrites.
This makes the default semantics consistent with that of organize.
*:
Changed format of reqres .status to <"C" or "I" for request.complete><"N" for no response or <response.code><"C" or "I" for response.complete> otherwise> (yes, this changes most --output formats of organize, again).
Added ~= expression atom which does re.match internally.
Changed all documentation examples to do ~= .200C instead of == 200C to reflect the above change.
organize:
--keep -> --no-overwrites for consistency.Documentation:
Core, Capture:
This release contains about half of the new changes in my local branches, so expect a new release soon-ish.
Core, Capture, Internal State pages:
Implemented problematic reqres flag, its tracking, UI, and documentation.
This flag gets set for no_response and incomplete reqres by default but, unlike Archive reqres with settings, it does not influence archival. Instead pWebArc displays “archival failure” as its icon and its badge gets ! at the end.
This is needed because, normally, browsers provide no indication when some parts of the page failed to load properly — they expect you to actually look at the page with your eyes to notice something looking broken instead — which is not a proper way to do this when you want to be sure that the whole page with all its resources was archived.
Added a separate state for reqres that are completed from cache: complete_fc.
Core, Toolbar button:
Core, Capture, Internal State pages:
Renamed reqres states:
noresponse -> no_response,incomplete-fc -> incomplete_fc.pWebArc now follows the following state diagram:
(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body received)
| | | |
| v v v
| (no_response) (incomplete) (complete)
| | | |
| \ | |
|\---> (canceled) -----\ \ | |
| \ \ \ |
| \ \ \ v
|\-> (incomplete_fc) ----->----->---------------------------->-----> (finished)
| / / |
| / /-----/ |
\--> (complete_fc) ----/ /------------- (collected) <--/ v
| | (discarded)
v v / |
(archived) <- (sIO) <--- (queued) <-------- (in_limbo) <---------/ |
| ^ | |
| | | |
/------/ \-----\ \----> (freeed) <----/
| |
\-> (failed to archive) -/Improved performance when using limbo mode.
Popup UI, Internal State pages:
The internal state/log pages are much nicer now.
But the popup UI in its default state might have become a bit too long…
Shortcuts:
Added toggle-tabconfig-limbo, toggle-tabconfig-children-limbo, and show-tab-state shortcuts,
Changed the default shortcut for collect-all-tab-inlimbo from Alt+A to Alt+Shift+A for uniformity.
Documentation:
Core, Capture:
organize:
Core, Capture, Popup UI:
Implemented “limbo” reqres processing stage and toggles.
“Limbo” is an optional pre-archival-queue stage for finished reqres that are ready to be archived but, unlike non-limbo reqres, are not to be archived automatically.
Which is useful in cases when you need to actually look at a page before deciding if you want to archive it.
E.g., you enable limbo mode, reload the page, notice there were no updates to the interesting parts of the page, and so you discard all of the reqres newly captured from that tab via appropriate button in the add-on popup, or via the new keyboard shortcut.
Core, Capture:
pWebArc now follows the following state diagram:
(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body received)
| | | |
| v v v
| (noresponse) (incomplete) (complete)
| | | |
| \ | |
|\---> (canceled) -----\ \ | |
| \ \ \ |
| \ \ \ v
\--> (incomplete-fc) ----->----->---------------------------->-----> (finished)
/ |
/-----/ |
/------------- (collected) <--/ v
| | (discarded)
v v / |
(archived) <- (sIO) <--- (queued) <-------- (in_limbo) <---------/ |
| ^ | |
| | | |
/------/ \-----\ \----> (freeed) <----/
| |
\-> (failed to archive) -/Improved performance.
Reworked the Log page into a multitude of Internal State pages that can now be narrowed to per-tab data.
In other words, from now on, there’s still one global page that displays everything, as before, but now there are also a bunch of new per-tab pages too.
All of those pages also show in-flight and in-limbo reqres now too.
Popup UI:
*:
Documentation:
*:
Implemented scrub --expr atom for rewriting links/references and wiping inner evils out from HTML, JavaScript, and CSS values.
CSS scrubbing is not finished yet, so all CSS gets censored out by default at the moment.
HTML processing uses html5lib, which is pretty nice (though, rather slow), but overall the complexity of this thing and the time it took to debug it into working properly is kind of unamusing.
export mirror:
Implemented a new subcommand for generating static website mirrors from previously archived WRR files.
This is similar to what wget -mpk does, but it works offline and the outputs can be scrubbed and transformed in various ways.
*:
A bunch of --expr atoms were renamed, a bunch more were added.
A bunch of --output formats changed, most notably flat is now named flat_ms.
Improved performance.
Documentation:
*:
Build:
wrrarms to build with newer nixpkgs and cbor2 modules, the latter of which is now vendored, at least until upstream solves the custom encoders issue.*:
Made more improvements to --output option of organize and import with IDNA and component-wise quoting/unquoting of tool-v0.8:
Added pretty_url, mq_path, mq_query, mq_nquery to substitutions and made pre-defined --output formats use them.
mq_nquery, and pretty_url do what nquery and nquery_url did before tool-v0.8.0, but better.
Dropped shpq, hpq, shpq_msn, and hpq_msn --output formats as they are now equivalent to their hup versions.
run:
--expr option now uses the same semantics as get --expr.*:
pprint:
clock line formatting a bit.import, organize:
Fixed an issue which could have caused import to skip some inputs and organize to forget to perform some of its actions.
These issue should have not been triggered ever (and with the default --output it is impossible to trigger) but to be absolutely sure you can simply re-run import and organize with the same arguments you’ve used before.
organize:
organize --output numbering being non-deterministic.Both issues were introduced in tool-v0.8.
organize:
--output flat_n.Toolbar button:
Note that with this release paths produced by wrrarms organize are expected to change compared to tool-v0.6:
with the default --output format you will only see changes to WRR files with international (IDNA) hostnames and those with the above response statuses;
names of files generated by most other --output formats will change quite a lot, since the path abbreviation algorithm is much smarter now.
import:
mitmproxy dumps.*:
Improved net_url normalization and components handling, added support for IDNA hostnames.
Improved most --output formats, custom --output formats now require format: prefix to distinguish them from the built-in ones, like in git.
Renamed response status codes:
N -> I for “Incomplete”NR -> N for “None”organize:
Renamed
organize --action rename -> organize --move (as it can now atomically move files between file systems, see below),--action hardlink -> --hardlink,--action symlink -> --symlink,--action symlink-update -> --symlink --latest.Implemented new organize --copy.
organize now performs changes atomically: it writes to newly created files first, fsync them, replaces old destination files, fsyncs touched directories, reports changes to stdout (for consumption by subsequent commands’ --stdin0), and only then (when doing --move) deletes source files.
*:
*:
--uncompressed option.*:
--no-cbor option to --no-print-cbors.Documentation:
organize:
--quiet, --batch-number, and --lazy options.--output flat and improved other --output formats a bit.get, run:
--expr arguments.*:
Documentation:
*:
*:
--default-profile option.--no-cbor option.Build:
*:
Changed semantics of --ignore-profiles a bit.
Generated filenames for partial files now use .part extension.
Generated filenames now include PID to allow multiple process instances of this program to dump to the same directory.
Core, Shortcuts:
Popup UI:
Documentation:
Core, Capture:
Core, Context menu:
Core, Capture:
Popup UI:
Documentation:
Core, Capture:
document_url and origin_url handling.Documentation:
Core, Capture:
Popup UI:
Core, Capture:
Core, Capture:
304 Not Modified responses.Popup UI, Help page:
Core, Capture:
*:
--ignore-profiles option.All planned features are complete now.
*:
It now prints the its own server URL at the start, for convenience.
Implemented gzipping before dumping to disk.
The extension can now specify a per-dump profile, which is a suffix to be appended to the dumping directory.
Implemented optional printing of the head and the tail of the dumped data to the TTY.
Core, Capture:
*:
*:
… each roughly sorted according to the expected order things will probably get implemented.
Hoardy-Web extensionMerge Pick into limbo and Drop into limbo option into one. I.e., remove all archival-related decision making from the machinery that puts reqres into limbo.
Collect/discard all in limbo and buttons should then become Continue processing all in limbo.
This behaviour would be both simpler to more useful for some use cases.
It will also allow us to simplify the behaviour of Auto-unmark 'problematic' reqres ... when a new 'GET' reqres replaces it ... regardless of its 'in_limbo' state option or just remove it altogether.
Improve integration with hoardy-web serve.
(~25% done) Reorganize tracking- and problematic-related options into config profiles, allow them to override each over.
Implement per-host profiles.
Internal State and Saved into Local Storage UIs.Internal State and Saved into Local Storage UIs.Closed tabs, so that you could easily collect/discard in_limbo reqres from such tabs.DOM snapshots when a page changes.network.proxy.no_proxies_on setting to allow Hoardy-Web archival to an archiving server to work out of the box when using proxies.WARC, so that third-party tools (i.e. not just hoardy-web) could be used for everything except capture.hoardy-web tool*:
DOM-snapshots of extension <= v1.30.0.WRR files. The fact that this is not possible at the moment makes memory consumption in those cases rather abysmal.serve, *:
sqlite database.sqlite index.mirror, organize:
organize or mirror a reqres containing https://web.archive.org/web/<something>/<URL> as if it was just a <URL>.mirror, scrub:
mirror:
mirror --standalone, which would inline all resources into each mirrored page, a-la SingleFile.organize:
WRR files (by URLs and similarity) and packing of related files into WRR bundles.WRR files.un206 command/option, which would reassemble a bunch of GET 206 WRR files into a single GET 200 WRR file.import, export:
HAR and WARC to WRR.WRR to WARC.PCAP to WRR.serve:
--symlink --latest hierarchies on-the-fly when running with --archive-to.*:
Support me via:
