software/hoardy-web/./doc/data-on-disk.md

Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, replay, mirroring, data scraping, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data.

Files

Raw Source

The WRR Data File Format

Glossary

Definition of WRR

A WRR dump is a CBOR (RFC8949) encoding of the following structure (encoded in typed almost-Python syntax, with list elements allowed to declare types):

Reqres = ReqresV1

Version 1

ReqresV1: list[Any]
ReqresV1 = [
    # magic
    "WEBREQRES/1": str,

    # a short description of software used to create this `reqres`,
    # e.g. `Firefox/128.0+Hoardy-Web/1.20.0`
    agent: str,
    # networking protocol, e.g. `HTTP/1.1`, `HTTP/2`, etc
    protocol: str,

    # see below
    request: RequestV1,

    # see below
    response: ResponseV1 | None,
    # note that this can be `None` (`null`), when the request got no response,
    # like when experiencing a network issue (archival of such request+response pairs
    # is disabled by default in the extension, see extension's settings)

    # Finish time, milliseconds since UNIX epoch
    ftime: int,

    # a dict from `str` to any `CBOR`-representable value
    extra: dict[str, Any],
    extra = {
        # URLs of containing documents
        document_url: str,
        origin_url: str,
        # both should be interpreted as set to `None` if missing from the dict

        # any errors the agent reported while fetching and processing this `Reqres`
        errors: list[str],
        # an empty list if missing

        # does this request have buggy metadata?
        request_buggy: bool,
        # `False` if missing

        # does this response have buggy metadata?
        response_buggy: bool,
        # `False` if missing

        # was this request actually submitted for sending?
        submitted: bool,
        # `True` if missing

        # was this response served from cache?
        from_cache: bool,
        # `False` if missing

        # was this response generated by the browser/extension/service worker?
        generated: bool,
        # `False` if missing

        # websocket frames
        websocket: list[WebSocketFrameV1],
        # `None` if missing
    }
]

RequestV1: list[Any]
RequestV1 = [
    # reQuest time, milliseconds since UNIX epoch
    qtime: int,

    # protocol method, e.g. `GET`, `POST`
    method: str,

    # the URL
    url: str,

    # request headers
    headers: list[HeaderV1],

    # is the following `body` complete?
    complete: bool,

    # request body
    body: str | bytes,
]

ResponseV1: list[Any]
ResponseV1 = [
    # reSponse time, milliseconds since UNIX epoch
    stime: int,

    # `HTTP` response code
    code: int,

    # `HTTP` response reason, if given
    reason: str,

    # response headers
    headers: list[HeaderV1],

    # is the following `body` complete?
    complete: bool,

    # response body
    body: str | bytes,
]

# a single `HTTP` header
HeaderV1: list[Any]
HeaderV1 = [
    name: str | bytes,

    value: str | bytes,
]

# a single `WebSocket` frame
WebSocketFrameV1: list[Any]
WebSocketFrameV1 = [
    # timestamp, milliseconds since UNIX epoch
    sent_at: int,

    # was this frame generated by the client?
    from_client: bool,

    # Operation Code
    opcode: int,

    # data
    content: str | bytes,
]

Interpreting WRR values

Representation quirks

Note that, unlike how it is with mitmproxy, with WRR the same protocol value is reused by both request and response. Thus, an HTTP protocol switch followed by an immediate server response of Section 3.2 of RFC7540 is represented with a pair of Reqres like:

which, in theory,

but,

So, in practice, this design actually makes WRR simpler and more efficient than a request.protocol+response.protocol design, like mitmproxy, would be.

On the same note, browsers also generate 304 Not Modified answers in a similar fashion:

Grouping Reqres into web pages

If you want to group reqres belonging to a single web page, you can look at their document_url fields.

For the “root” web page, document_url will be unset, the “children” reqres will have document_url set to the URL of the corresponding “root” document.

WRR data on disk

On disk, Reqres can be stored as

Obviously, all of the above has an advantage of making WRR files and bundles easily parsable with readily available libraries in basically any programming language there is. After all, gzip is completely standard, and CBOR is only slightly less supported than JSON, but it is much more space-efficient and can represent arbitrary binary data.

Comparisons of WRR to other web archival formats

Simple directories of non-de-duplicated compressed WRR files (not bundles!) are still more efficient than:

WRR bundles are effectively on-par with the storage efficiency of WARCs. Compared to WRRs, WARCs store less metadata, but WRRs are encoded more efficiently, so, for simple GET requests it’s usually a wash. (Though, WRRs can store more than just GET requests, and WARCs can’t.)

After converting all my previous wget, curl, mitmproxy, and HAR archives into this and with some yet unpublished data de-duplication and xdelta compression between same-URL revisions WRRs are infinitely more efficient than anything else.

For me, it uses about 3GiB per year of browsing on average (~5 years of mostly uninterrupted data collection ATM) but I use things like uBlock Origin and uMatrix to cut things down, and image boorus and video hosting sites have their own pipelines.