Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, mirroring, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data.
All notable changes to this project are documented in this file. The format is based on Keep a Changelog. This project adheres to Semantic Versioning.
get, export mirror, etc:
Restricted the idna workaround of tool-v0.15.4 to hostnames with “–” in [2:4] character positions.
The previous iteration made parse_url start accepting many malformed URLs.
URL parsing will now strip hostnames of leading and following whitespace, like browsers do.
Mainly, this improves export mirror outputs.
Fixed output formatting when redirecting output to a non-tty destination.
File path parts starting with “.” in profiles (i.e. buckets) given by clients are ignored now.
This prevents escapes from the given --root.
The previous behaviour was not really a security issue, given that the server is not designed to be run with untrusted clients, and filenames are generated by it, not the clients. But still.
Renamed command-line options:
--no-print-cbors -> --no-print--default-profile -> --default-bucket--ignore-profiles -> --ignore-bucketsThis is makes them use the same terminology the extension uses.
Old names are kept as aliases.
version endpoint, for extensibility.get, export mirror, etc:
idna module failing to parse some hostnames (#5 on GitHub).find:
Added --sniff-* options to fix crashes introduced in v0.15.0.
Added tests to hopefully stop this kind of errors.
get, export mirror, etc:
--expr: technically, renamed full_url -> url, though it did not officially exist before.
Added it to the docs.
ftp and ftps URL schemes are now allowed everywhere.
scrub, export mirror:
From now on scrub will simply remove all CORS and SRI attributes from all relevant HTML tags.
This works fine 99% of the time. Smarter handling for this will be implemented later.
Fixed MIME type sniffing of XHTML data.
Also, added some tests for my mimesniff implementation.
Fixed crashes when URL remapper encounters weirdly malformed URLs.
From now on they will be remapped into void URLs instead.
export mirror:
Fixed it skipping regular files given directly as command line arguments.
This was broken since v0.15.0.
scrub, import:
scrub, export mirror:
Fixed a stupid bug in MIME detection code that prevented external CSS files from being detected as such.
Added tests to prevent such things in the future.
Fixed CSS formatting with +whitespace set.
Made scrub remove crossorigin attributes from HTML tags for which it remapped a URL.
This seems to have fixed most of issues causing pages produced by export mirror looking broken when opened in a web browser.
export mirror
import *:
--sniff-* options to fix crashes introduced in v0.15.0.export mirror
depth value in the UI.not remapping '%s' lines.export mirror sub-command now produces results quite usable in a normal web browser. I.e. it is now comparable to, say, what Single-File produces.
Feature-wise, it reaches a Pareto front, AFAICS, since no other tool I know of can do efficient (with shared page requisites) incremental static semi-open (see --remap-semi option below) website mirrors.
At the moment, scrubbed CSS can get a bit broken sometimes, because hoardy-web leans in favor of its results being safe to use, not them being as close to the original as possible. Also, support for audio, video, and source HTML tags is still a bit quirky. But the current state is quite usable.
scrub, export mirror:
Implemented stylesheet (CSS) scrubbing with the help of tinycss2.
I.e., requisite resource URLs mentioned in stylesheets will now be properly remapped.
I.e., exported website mirrors will be styled now.
export mirror:
Added --remap-semi option, which does the same thing as --remap-open (which is equivalent to wget --convert-links), except it remaps unavailable action links and page requisites to void URLs, making the resulting generated pages self-contained and safe to open in a web browser without it trying to download something.
I.e. --remap-semi does what wget --convert-links should be doing, IMHO.
Added --root-url-prefix and --root-url-re options.
pprint, get, run, stream, export mirror:
Implemented --sniff-* options controlling mimesniff algorithm usage.
For pprint sub-command they replace --naive and --paranoid options.
--expr, --output: Added pretty_net_url, pretty_net_nurl, raw_path_parts, and mq_raw_path atoms.
scrub, export mirror:
Changed the way all --remap-* options are implemented. Most of the remapping logic was moved into the scrub function. --remap-* options simply change default values of the corresponding --expr options now.
+styles and +iframes options are now set by default.
Since these things can now be properly exported.
Renamed (+|-)srcs options to (+|-)reqs to follow the terminology used by wget.
In documentation, “page resources” became “requisite resources” and “page requisites”.
Improved censoring for IE-pragmas.
Improved +indent and +pretty output layout a bit.
Improved +verbose output format a bit.
export mirror:
Renamed --root option to --root-url, -r and --root options now point to --root-url-prefix instead. The --root option name is deprecated now and will be removed in the future.
Improved progress reporting UI.
It’s much prettier and more informative now.
It ignores duplicate input paths now.
This allows to easily prioritize exporting of some files over others by specifying them in the command line arguments first, followed by their containing directory in a later argument.
README.md has a new example showcasing it.
It delays disk writes for HTML pages until after all of their requisite resources finished exporting now.
I.e. newly generated HTML pages can now be opened in a web browser while export mirror is still running, having not finished exporting other things yet.
Improved content MIME type handling a bit, added text/vtt recognition.
--expr, --output:
Renamed: path_parts -> npath_parts, mq_path -> mq_npath.
Changed semantics of net_url and pretty_url a bit. Both add trailing slashes after empty raw_paths now. Also, pretty_url does not normalize raw_path now, i.e. now it only re-quotes path parts, but does not interpret . and .. path parts away.
Greatly improved documentation.
scrub, export mirror:
Fixed generation of broken file: links for URLs with query parameters.
From now on stylesheet, icon, and shortcut links are treated as page requisites.
This fixed a bug where export mirror with --depth set would forget to export shortcut icons and CSS files.
Fixed a bug where export mirror with --depth and --remap-(open|closed) set would fail to remap unreachable URLs properly.
Fixed some places where the code was misaligned with the documentation.
scrub and export mirror use -verbose by default now, which documentation claimed they did, but they did not.Fixed some typos.
pWebArc -> Hoardy-Web.::pWebArc:: error codes into a more consistent naming scheme.wrrarms -> hoardy-web.dumb-dump-server -> hoardy-web-sas.Improved all the scripts by adding usage descriptions and --help options to all of them.
Added --to option to wrrarms-pandoc script. It allows you to change the output format it will use.
Added wrrarms-spd-say script, which can feed contents of an archived HTML document, extracted from HTML via pandoc -t plain, to speech-dispatcher’s spd-say, i.e. to your preferred TTS engine.
Added --*-url options to wrrarms-w3m and wrrarms-pandoc scripts. They allow you to control how to print the document’s URL in the output.
get: implemented --expr-fd option, which allows you to extracts multiple --expr values from the same input file to different output file descriptors in a single wrrarms call.
Modified wrrarms-w3m and wrrarms-pandoc scripts to use --expr-fd option, making them ~2x faster.
export mirror: implemented support for multiple --expr arguments.
import: implemented --override-dangerously option.
Added more --output formats.
Renamed all --no-output options to --no-print.
Edited --output formats, making them more consistent with their expected usage:
Edited the default, short, surl_msn, and url_msn --output formats, replacing a “.” before the num field with a “_”. Because these formats do not mention any file extensions.
Edited surl_msn and url_msn --output formats, replacing and a “_” before the method with “__”. To make these --output formats useful in programmatic usage.
Edited most other--output formats, replacing a “_” before the method field with a “.” and a “.” before the non-standalone num with a “_”. Since these --output formats do use file extensions, this turns the whole wrrarms-specific suffix into a sub-extension.
wrrarms-pandoc uses plain text --to output format by default now. The previous default was org-mode.
Improved error messages.
Improved documentation.
export mirror now respects the given --errors option value not only while indexing inputs, but also while rendering and writing out outputs.
Resurrected flat_n --output format.
pWebArc is now officially supported on Fenix (Firefox for Android). It is quite usable there now, so go forth and test it.
Chromium version now has a update_url set in the manifest.json, so if you use chromium-web-store or some such, it can be updated semi-automatically now, see the extension’s README.md for more info.
Implemented User Interface and Accessibility > Verbose option.
From now on, by default, pWebArc will have its most common but annoying notifications mention they can be disabled and explain how.
This is mostly for Fenix users, where these things are not obvious, but it could also be useful for new users elsewhere.
Implemented User Interface and Accessibility > Spawn internal pages in new tabs option which controls if internal pages should be spawned in new tabs or reuse the current window.
It can not be disabled on desktop browsers at the moment, but it is disabled by default on mobile browsers.
Implemented a bunch of new notifications about automatic fixes applied to config.
I.e., it will now not just fix your config for you, but also complain if you try to set an invalid combinations of options.
Implemented Generate desktop notifications about ... > UI hints option to allow you to disable the above notifications.
pWebArc will CBOR-dump all reqres fields completely raw from now on.
wrrarms learned to handle this properly quite a while ago.
This simplifies the parsing of the results and makes the implementation adhere to the stated technical philosophy more closely.
The dumps will grow in size a tiny bit, but this is negligible, since they are compressed by default by all the archival methods now.
Moreover:
Stash 'in_limbo' reqres option is being inherited by children tabs like the rest of similar options do.build.sh chromium target to chromium-mv2 in preparation for eventual chromium-mv3 support.browserAction updates.browserAction title updates being flaky on Fenix. I.e. you can stare at the Extensions > pWebArc line in the browser’s UI now while the browser fetches some stuff and it will be properly interactively updated.origin_url field of the very first dump of each session when Workarounds for Firefox bugs > Restart the very first request is enabled (which is the default).pWebArc now runs under Fenix aka Firefox-for-Android-based browsers, including at least Fennec and Mull.
Thought, Export via 'saveAs' archival method is broken there, because of a bug in Firefox. Other methods do work, though.
(Also, it is not marked as compatible with Firefox on Android on addons.mozilla.org at the moment, it probably will be in the next version.)
The above change also added a settings page (aka options_ui).
At the moment, the settings page is simply an unrolled by default version of popup UI, with per-tabs settings removed.
This is need because on mobile browsers the main screen of the browser is not a tab and there’s no toolbar, so there’s no popup UI button there, and so the extension UI becomes really confusing without a separate settings page.
Split in_flight stat into a sum of two numbers.
This makes things less confusing on Chromium, the Help page explains it in more detail.
Added toolbar button’s badge as a prefix to its title, changed its format a bit.
This is needed because Fenix-based browsers do not display the badge at all, so this change helps immensely there. Meanwhile, on desktop browsers this does not hurt.
Improved styling and dark mode contrast of the popup UI.
Improved documentation.
In particular, among other things, added a lot of new anchors to the Help page, most internal links referencing some fact discussed in another section now point directly to the relevant paragraph instead of pointing to its section header.
On Firefox:
Fixed capture of responses produced by service/shared workers.
Also, added a new error code for when it (very rarely) fails because of a race condition inherent in webRequest API and documented all of it on the Help page.
Fixed HTTP protocol version detection, requests fetched via HTTP/3 will now be marked as such.
Added yet another webRequest API error to a list of those that mark reqres response data as incomplete.
On Chromium:
in_flight state indefinitely.Generally:
Fixed navigation with browser’s Back and Forward buttons to work properly on the Help page.
Fixed a bug where force-stopping all in-flight reqres in a single tab could also drop some of the others.
Help page.HTML layout.config.history default value.Implemented reqres persistence across restarts.
pWebArc can now save and reload collected but not archived reqres (including those in_limbo) by stashing them into browser’s local storage. This is now enabled by default, but it can be disabled globally, or per-tab.
As a consequence, pWebArc now tracks browsing sessions and shows when a reqres belongs to an older session on its Internal State page.
Implemented two new archiving methods. pWebArc can now archive collected reqres by
generating fake-Downloads containing either separate dumps (one dump of an HTTP request+response per file) or bundles of them (many dumps in a single file, for convenience, to be later imported via wrrarms import bundle),
archiving separate dumps to your own private archiving server (the old one, the previous default, inherited on extension update),
archiving separate dumps to your browser’s local storage (the new default on a new clean install).
As a consequence, pWebArc now has a new Saved in Local Storage page for displaying the latter.
Implemented display and filtering for queued and failed reqres on the Internal State page.
Implemented tracking of per-state size totals for reqres in most states after finished.
As a consequence, popup UI will now display those newly tracked sizes.
Introduced the errored reqres state.
With stashing to local storage enabled, pWebArc will now try its best not to loose any captured data even when its archiving code fails (bugs out) with an unexpected exception. If it bugs out in the capture code, then all bets are off, unfortunately.
pWebArc will now track if an error is recoverable and will not retry actions with unrecoverable errors automatically by default.
pWebArc now follows the following state diagram:
(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body recived)
| | | |
| v v v
| (no_response) (incomplete) (complete)
| | | |
| \ | |
|\---> (canceled) ----\ \ | |
| \ \ \ |
|\-> (incomplete_fc) ---\ \ \ v
| >------>---------------------------->-----> (finished)
|\--> (complete_fc) ----/ / |
| / / |
\----> (snapshot) ----/ /- (collected) <--------- (picked) <--/ |
/ ^ | |
(stashIO?) <----/ | v v
| \-- (in_limbo) <- (stashIO?) <- (dropped)
v | |
(queued) <------------------\ | |
/ | ^ \ \ \-----> (discarded) <-----/
(exported) <-/ | | \----------------\ \ ^
| | | \ \ |
| /---/ \-----------------\ \ \ |
| | | \ \ |
| v | \ \ |
|\-> (srvIO) -> (stashIO?) -> (failed) | \ |
| | ^ / \ |
| v | v | |
| (sumbitted) --------------> (saveIO) --> (saved) | {{!saving}}
| \ |
\-------->-----------------------------------------------/Renamed all Profile settings into Bucket, as this makes more sense.
Improved popup UI layout.
Changed toolbar icon’s badge format a bit.
Improved debugging options.
A huge internal refactoring to solve constant sub-task scheduling issues once and for all.
Improved documentation.
config.logDiscarded option as it is no longer needed (pWebArc has proper log filtering now).import bundle sub-command which takes WRR-bundles (optionally gzipped concatenations of WRR-dumps) as inputs. The next version of the extension will start (optionally) producing these.pprint output formatting.pWebArc will no longer automatically reload on updates, waiting for the browser to restart or for you to reload it explicitly instead.
This way you won’t lose any data on extension updates.
Proper automatic reloads on updates will be implemented later, after pWebArc gets full persistence.
Popup UI:
Reverted the split between Globally and This session.
Implementing that split properly will make future things much harder, so, simple is best.
Queued stat moved to a separate line again.
It also shows the sum total of sizes of all dumps now.
Added Scheduled ... actions stat line, showing the names of actions that are scheduled.
It is hidden by default, because watching it closely while pWebArc is very busy can probably cause seizures in some people.
Improved documentation.
Implemented DOM snapshots, their popup UI, keyboard shortcuts and documentation.
snapshotTab) and snapshot all open tabs (snapshotAll).Ctrl+Alt+S runs snapshotTab by default now.Added a bunch of new toolbar icons for various tab states.
In particular, problematic state as well as mixed-capture states (e.g., disabled in this tab, but enabled and with limbo mode in children tabs) now have their own special icons.
Changed some default keyboard shortcuts:
Ctrl+Alt+A and Ctrl+Alt+C run collectAllInLimbo and collectAllTabInLimbo respectively now;Alt+Shift+D and Alt+Shift+W run discardAllInLimbo and discardAllTabInLimbo now.Popup UI:
All SVG icons were edited to not reference any fonts, since those are not guaranteed to be available on a user’s system.
Improved behaviour of new tabs created by clicking buttons on the Internal State page.
Greatly improved documentation.
Implemented dark mode theme. The extension will switch to it automatically when the browser asks (which it will if you switch your browser’s theme to a dark one).
Implemented some new optional UI-related accessibility config options with toggles in popup UI:
v1.9.0, with slight variations for the new color-coding).Improved Internal State/Log UI:
Added UI for internal scheduled/delayed actions/functions (e.g., saving of frequently changing stuff to persistent storage, automatic actions when a tab closes, canceling and reloading not-yet-debugged tabs on Chromium, etc):
~ or . in it and change its color, depending on the importance of the stuff that is waiting to be run.Added config options and popup UI toggles for picking and marking as problematic reqres with various HTTP status codes.
Implemented new config options and popup UI toggles for browser-specific workarounds. In particular, on Chromium you can now set the URL new root tabs will be reset to (still about:blank by default).
Added more desktop notifications, added config options and popup UI toggles for them.
Improved keyboard shortcuts:
Help page will now show currently active shortcuts (when viewed via the Help button from the extension UI).Alt+S by default now (similarly to how Ctrl+S saves the page).Alt+W by default now (similarly to how Ctrl+W closes the tab).Alt+U by default now.Alt+Shift+U by default unmarks all problematic reqres globally now.Alt+Shift+S and Alt+Shift+W by default respectively collect and discard all reqres in limbo globally now.Much of the code working with Chromium’s debugger was rewritten. Now it reports all the errors properly and no longer crashes when the debugger gets detached at inopportune time in the pipeline (which is quite common, unfortunately).
Mark reqres as 'problematic' when they finish > ... with reqres errors config option became > ... with reqres errors and get 'dropped', i.e. it is now disjoint with > ... with reqres errors and get 'picked'.
Improved desktop notifications.
Popup UI, in its default rolled-up state, now exposes Generate desktop notifications about > ... new problematic reqres option and has custom tabindexes set, for convenience.
Changed some config option defaults (your existing config will not get affected).
Slightly improved performance in normal operation. Greatly improved performance when archiving large batches of reqres at once, e.g. when collecting a lot of stuff from limbo.
Greatly improved documentation.
A whole ton of bugfixes.
So many bugfixes that pWebArc on Chromium now actually works almost as well as on Firefox.
All leftover issues on Chromium I’m aware of are consequences of Chromium’s debugging API limitations and, as far as I can see, are unsolvable without actually patching Chromium (which is unlikely to be accepted upstream, given that patching them will make ad-blocking easier).
archiveweb.page project appears to suffer from the same issues.
Meanwhile, pWebArc continues to work exceptionally well on Firefox-based browsers.
Implemented “negative limbo mode”.
It does the same thing as limbo mode does, but for reqres that were dropped instead of picked. (Which is why there is an arrow from dropped to in_limbo on the diagram below.)
Implemented optional automatic actions when a tab gets closed.
E.g., you can ask pWebArc to automatically unmark that tab’s problematic reqres and/or collect and archive everything belonging to that tab from limbo.
Implemented a bunch of new desktop notifications.
Added a bunch of new configuration options.
This includes a bunch of them for controlling desktop notifications.
Added a bunch of new keyboard shortcuts.
Also, keyboard shortcuts now work properly in narrowed Internal State pages.
Implemented stat persistence between restarts.
You can brag about your archiving prowess to your friends by sharing popup UI screenshots now.
Added the Changelog page, which can be viewed by clicking the version number in the extension’s popup.
pWebArc now follows the following state diagram:
(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body recived)
| | | |
| v v v
| (no_response) (incomplete) (complete)
| | | |
| \ | |
|\---> (canceled) -----\ \ | |
| \ \ \ |
| \ \ \ v
|\-> (incomplete_fc) ----->----->---------------------------->-----> (finished)
| / / |
| / /-----/ |
\--> (complete_fc) ----/ /--------------- (picked) <---/ v
| | (dropped)
v v / |
(archived) <- (sIO) <- (collected) <------- (in_limbo) <---------/ |
| ^ | |
| | | |
/------/ \-----\ \--> (discarded) <---/
| |
\-> (failed to archive) -/Terminology-wise, most notably, picked and dropped now mean what collected and discarded meant before.
See the Help page for more info.
A lot of changes to make pWebArc consistently use the above terminology — both in the source and in the documentation — were performed for this release.
Improved visuals:
Extension’s toolbar button icon, badge, and title are much more informative and consistent in their behaviour now.
The version number button in the popup (which opens the Changelog) will now get highlighted on major updates.
Similarly, the Help button will now get highlighted when that page gets updated.
The popup, the Help page, the Internal State aka the Log page all had their UI improved greatly.
All the toggles in the popup are now color-coded with their expected values, so if something looks red(-dish), you might want to check the help string in question just in case.
Improved documentation.
export mirror: implemented --no-overwrites, --partial, and --overwrite-dangerously options.export mirror: Switched the default from --overwrite-dangerously (which is what export mirror did before even if there was no option for it) to --no-overwrites. This makes the default semantics consistent with that of organize.
Changed format of reqres .status to <"C" or "I" for request.complete><"N" for no response or <response.code><"C" or "I" for response.complete> otherwise> (yes, this changes most --output formats of organize, again).
Added ~= expression atom which does re.match internally.
Changed all documentation examples to do ~= .200C instead of == 200C to reflect the above change.
organize: renamed --keep -> --no-overwrites for consistency.
Improved documentation.
(Actually, this releases about half of the new changes in my local branches, so expect a new release soonish.)
Implemented problematic reqres flag, its tracking, UI, and documentation.
This flag gets set for no_response and incomplete reqres by default but, unlike Archive reqres with settings, it does not influence archival. Instead pWebArc displays “archival failure” as its icon and its badge gets ! at the end.
This is needed because, normally, browsers provide no indication when some parts of the page failed to load properly — they expect you to actually look at the page with your eyes to notice something looking broken instead — which is not a proper way to do this when you want to be sure that the whole page with all its resources was archived.
Implemented currently active tab’s limbo mode indication via the icon.
Added a separate state for reqres that are completed from cache: complete_fc.
Renamed reqres states:
noresponse -> no_response,incomplete-fc -> incomplete_fc.pWebArc now follows the following state diagram:
(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body recived)
| | | |
| v v v
| (no_response) (incomplete) (complete)
| | | |
| \ | |
|\---> (canceled) -----\ \ | |
| \ \ \ |
| \ \ \ v
|\-> (incomplete_fc) ----->----->---------------------------->-----> (finished)
| / / |
| / /-----/ |
\--> (complete_fc) ----/ /------------- (collected) <--/ v
| | (discarded)
v v / |
(archived) <- (sIO) <--- (queued) <-------- (in_limbo) <---------/ |
| ^ | |
| | | |
/------/ \-----\ \----> (freeed) <----/
| |
\-> (failed to archive) -/Added more shortcuts, changed defaults for others:
Added toggle-tabconfig-limbo, toggle-tabconfig-children-limbo, and show-tab-state shortcuts,
Changed the default shortcut for collect-all-tab-inlimbo from Alt+A to Alt+Shift+A for uniformity.
Improved UI:
Improved performance when using limbo mode.
Improved documentation.
organize: now works on Windows.Implemented “limbo” reqres processing stage and toggles.
“Limbo” is an optional pre-archival-queue stage for finished reqres that are ready to be archived but, unlike non-limbo reqres, are not to be archived automatically.
Which is useful in cases when you need to actually look at a page before deciding if you want to archive it.
E.g., you enable limbo mode, reload the page, notice there were no updates to the interesting parts of the page, and so you discard all of the reqres newly generated by that tab via appropriate button in the add-on popup, or via the new keyboard shortcut.
pWebArc now follows the following state diagram:
(start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body recived)
| | | |
| v v v
| (noresponse) (incomplete) (complete)
| | | |
| \ | |
|\---> (canceled) -----\ \ | |
| \ \ \ |
| \ \ \ v
\--> (incomplete-fc) ----->----->---------------------------->-----> (finished)
/ |
/-----/ |
/------------- (collected) <--/ v
| | (discarded)
v v / |
(archived) <- (sIO) <--- (queued) <-------- (in_limbo) <---------/ |
| ^ | |
| | | |
/------/ \-----\ \----> (freeed) <----/
| |
\-> (failed to archive) -/The Log page became the Internal State page, now shows in-flight and in-limbo reqres. It also allows narrowing to data belonging to a single tab now.
Improved UI.
Improved performance.
Implemented scrub --expr atom for rewriting links/references and wiping inner evils out from HTML, JavaScript, and CSS values.
CSS scrubbing is not finished yet, so all CSS gets censored out by default at the moment.
HTML processing uses html5lib, which is pretty nice (though, rather slow), but overall the complexity of this thing and the time it took to debug it into working is kind of unamusing.
Implemented export mirror subcommand generating static website mirrors from previously archived WRR files, kind of similar to what wget -mpk does, but offline and the outputs are properly scrubbed.
A bunch of --expr atoms were renamed, a bunch more were added.
A bunch of --output formats changed, most notably flat is now named flat_ms.
Improved performance.
Improved documentation.
Updated wrrarms to build with newer nixpkgs and cbor2 modules, the latter of which is now vendored, at least until upstream solves the custom encoders issue.
Made more improvements to --output option of organize and import with IDNA and component-wise quoting/unquoting of tool-v0.8:
Added pretty_url, mq_path, mq_query, mq_nquery to substitutions and made pre-defined --output formats use them.
mq_nquery, and pretty_url do what nquery and nquery_url did before v0.8.0, but better.
Dropped shpq, hpq, shpq_msn, and hpq_msn --output formats as they are now equivalent to their hup versions.
run: --expr option now uses the same semantics as get --expr.
Tiny improvements to performance.
pprint: fixed clock line formatting a bit.--output flat_n.Bugfix #1:
tool-v0.8 might have skipped some of the updates when importing and forgot to do some actions when doing organize, which was not the case for tool-v0.6.
These bugs should have not been triggered ever (and with the default --output they are impossible to trigger) but to be absolutely sure you can re-run import mitmproxy and organize with the same arguments you used before.
Bugfix #2:
organize --output numbering is deterministic again, like it was in tool-v0.6.
mitmproxy dumps.Improved net_url normalization and components handling, added support for IDNA hostnames.
Improved most --output formats, custom --output formats now require format: prefix to distinguish them from the built-in ones, like in git.
Renamed response status codes:
N -> I for “Incomplete”NR -> N for “None”Renamed
organize --action rename -> organize --move (as it can now atomically move files between file systems, see below),--action hardlink -> --hardlink,--action symlink -> --symlink,--action symlink-update -> --symlink --latest.Added organize --copy.
organize now performs changes atomically: it writes to newly created files first, fsync them, replaces old destination files, fsyncs touched directories, reports changes to stdout (for consumption by subsequent commands’ --stdin0), and only then (when doing --move) deletes source files.
Made many internal changes to simplify things in the future.
Paths produced by wrrarms organize are expected to change:
with the default --output format you will only see changes to WRR files with international (IDNA) hostnames and those with the above response statuses;
names of files generated by most other --output formats will change quite a lot, since the path abbreviation algorithm is much smarter now.
--uncompressed option.--no-cbor option to --no-print-cbors.organize: implemented --quiet, --batch-number, and --lazy options.organize: implemented --output flat and improved other --output formats a bit.get and run now allow multiple --expr arguments.--default-profile option, changed semantics of --ignore-profiles a bit.--no-cbor option..part extension.document_url and origin_url handling.304 Not Modified responses.Help page.--ignore-profiles option.profile, which is a suffix to be appended to the dumping directory.All planned features are complete now.
Help page: it’s much more helpful now.Log page: it’s an interactive page that gets updated automatically now.