by Jan Malakhovski, version 1.0.0, updated , published , created
Note that, technically, each section of this document is a self-contained piece and can be read independently. The latter sections refer to the former ones all the time, however, so it probably would be much more efficient to simply read this document linearly from top to bottom.
(Click me to see it.)
Added #tos-watch and #consumer-guide sections.
As a result, this document now contains everything I wanted it to say.
Edited previous sections in minor ways.
(Note: this section was cross-posted to CRW’s Zulip.)
Let’s decide on a standard machine-readable way of recording all URLs of Terms of Service, Privacy Policy, Terms of Use, End-User License Agreements, Warranty Terms, and similar documents relevant to each company and/or product in Consumer Rights Wiki (CRW). That is, let’s decide how to record such things in CRW’s Cargo templates.
For context, on GamersNexus’s YouTube channel, in a relatively recent video titled “We Could Get Sued for This” around t=00:20:00 Steve talks about how Smart TV manufacturers are likely to quietly start changing their Terms of Service agreements in response to GamersNexus’ video titled “216,000,000 Spy TVs | The LG Smart TV Problem” and how web data hoarders in the audience could help GamersNexus by archiving all such agreements by all other Smart TV companies for future research.
So I went to do that, but trying to perform said archivals I discovered that, at least for some companies, even finding the agreements applicable to the devices you are currently using is a non-trivial task! In fact, it appears that even organizations that work in this area for many years, like ToS;DR, miss these things. For example, at the moment of writing of this, their page on Samsung, refers to two documents:
Meanwhile, I found dozens more such documents (or hundreds more, if you count all the references to third-party agreements you are supposedly agreeing to when you agree to their terms of service, in all their per-country and/or per-language versions) by crawling through Google, Bing, and DuckDuckGo results, and by manually crawling around Samsung’s own websites (each of which produced links to never seen before documents!):
And I’m not at all sure that’s all of them.
Per-product sets of applicable legal documents can also be useful in some cases. For example, health-related agreements from the above list don’t apply to all Samsung devices, at least at the moment of writing of this. For another example, ToS;DR rates Open Camera as “Grade C” because its privacy policy honestly mentions that the author’s website uses Google Ad Sense. Even though Open Camera itself deserves “Grade A” since it’s a GPLv3+ app that comes without ads or anything else evil.
I think the simplest solution here is to add fields containing categorized per-locale legal URLs to https://consumerrights.wiki/w/Template:CompanyCargo and then start referencing them in https://consumerrights.wiki/w/Template:ProductCargo, https://consumerrights.wiki/w/Template:ProductLineCargo, etc.
E.g., in https://consumerrights.wiki/index.php?title=Samsung&action=edit:
{{CompanyCargo
|Description=Large manufacturing conglomerate headquartered in Seoul, South Korea.
|Website=https://samsung.com/
|...
|Legal_General_en_US=https://terms.account.samsung.com/contents/legal/usa/eng/general.html https://www.samsung.com/us/account/privacy-policy/
|Legal_Health=https://samsunghealth.com/terms https://samsunghealth.com/privacy
|Legal_Health_en_US=https://www.samsung.com/us/privacy-policy/consumer-health-data-privacy-statement/
|Legal_TV_en_US=https://www.samsung.com/us/support/legal/LGL10000312/
|...
}}
and then, in https://consumerrights.wiki/index.php?title=Samsung_TVs&action=edit:
{{ProductLineCargo
|Company=Samsung
|LegalCategories=General,TV
|...
}}
and in https://consumerrights.wiki/index.php?title=Samsung_Smartphones&action=edit (does not currently exists):
{{ProductLineCargo
|Company=Samsung
|LegalCategories=General,Health
|...
}}
(Note: this section was cross-posted to CRW’s Zulip and to the CRW itself.)
Let’s create an independent web archive, preferably run by FULU, with the following properties:
it should support easy import and hoarding of snapshots from other web archives and from independent data hoarders, i.e.:
it should be easy to ask it to slurp a list of Wayback Machine URLs into it;
it should be possible for third-parties to submit ZIPs, SingleFile outputs, WARCs (spec), HARs (spec), WRRs, and other similar web page snapshot formats into it;
while it could hoard snapshots of arbitrary web pages, for legal reasons discussed below it would only publicly re-distribute snapshots of consumer-rights-related web pages; which is to say, the list of URLs it would be re-distributing would be vetted;
ideally, it should be used for archival of all the web pages cited by the CRW;
at the very least, it should be used for archival of Terms of Service and similar URLs recorded in CRW’s Cargo templates;
it could also hoard archives of various relevant marketing materials and product listing pages;
but, again, it should not be used a general purpose web archiving service;
it would comply with content removal and DMCA take-down requests from companies legally “owning” those documents by hiding them from public view, as the law requires and like most other web archives do, but, importantly, it would also add any companies making such requests to its watch-list of “companies apparently planning to do something nefarious”, which it would then actively monitor for enshittification, creating CRW incidents and press releases as soon as any enshittification is discovered, and immediately re-publishing all relevant snapshots previously hidden from public eyes, now under the “newsworthy” fair use copyright exception.
For context, on Louis Rossmann’s YouTube channel, in video from 2024 titled “DCS sues Small YouTuber for accurate product review showing battery issues & misleading warranty” Louis discusses how Deep Cycle Systems (DCS) company changed their battery Warranty Terms on their web site without updating the “last updated” date there, removed the old versions of those pages from the Wayback Machine, and then sued Stefan Fischer, a small YouTuber who reviewed their product, for “defamation” and “misrepresenting their warranty terms”.
As far as I’m aware, that story ends happily for the reviewer, but only because Louis found unedited copies of the offending pages in NLA’s web archive and could demonstrate that DCS were lying and backdating their edited warranty agreements.
Meanwhile, on GamersNexus’s YouTube channel, in a relatively recent video titled “We Could Get Sued for This” around t=00:20:00 Steve talks about how Smart TV manufacturers are likely to quietly start changing their Terms of Service agreements in response to GamersNexus’ video titled “216,000,000 Spy TVs | The LG Smart TV Problem” and how web data hoarders in the audience could help GamersNexus by archiving all such agreements by all other Smart TV companies for future research.
Then, back on Louis Rossmann’s YouTube channel in his recent video titled “FetchTV says I’m ‘inaccurate & unfair’… you want to play? LET’S GO! 😤” Louis talks about FetchTV set-top box and subscription TV service company that had recently decided to turn older devices made by them defunct and delete all the data stored on them unless their users are already paying them a subscription fee or are willing to pay them a new levy for it, which is not at all what they have promised in their marketing materials when selling those devices. In that video Louis also asks his viewers to go and archive all the relevant documents referenced in his CRW article on the issue personally because they have a tendency of vanishing from the Wayback Machine, as his previous experience shows.
So, wouldn’t it be great if there was a web archiving service that:
would just automagically archive all things referenced by CRW, including by duplicating snapshots from Wayback Machine’s and other similar web archiving services;
where any independent reviewer could just submit URLs of any relevant evidence pages before publishing their reviews, thus effectively making the DCS bullying strategy defunct;
where any interested bystander could help by submitting more relevant URLs and/or their own snapshots.
As far as I understand, the Wayback Machine and similar web archiving services exist in a kind of legal gray zone that did not shrink into nothingness only because
some of what they do falls under “nonprofit and educational” clauses of fair use copyright exceptions and Article 108(h);
outside of that, the Internet as a whole is younger than current copyright term length, so most of the material such archives store and distribute is in violation of copyright law as it currently written, so such archives have to employ the following defensive strategies:
they have to be very cooperative with copyright title holders and domain host owners and hide their archived data from the public on request quite easily (e.g., https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/, https://trove.nla.gov.au/help/categories/websites-category, etc);
they have to respect robots.txt files and declare themselves in User-Agent strings, thus giving domain owners all the chances to block them from archiving things;
they insist that they are “libraries” even though their operations violate “library physics” of “we only have N copies we can lend” all the time;
which is also the place where they get sued the most, like with https://en.wikipedia.org/wiki/Hachette_v._Internet_Archive;
lawyers use such web archiving services and would rather like keep them around too.
The only public web archive known to me that actively resists removal of old snapshots from itself and ignores robots.txt is https://archive.today/, which is why it’s usually classified as a “pirate” web site, FBI subpoenaed its domains before, and their current set of domains is banned in many jurisdictions, see https://en.wikipedia.org/wiki/Archive.today#History.
But, importantly for this discussion, https://archive.today/ sometimes substantially edit archived pages they serve to the public, which is one of the reasons why Wikipedia banned their usage on Wikipedia pages, see https://en.wikipedia.org/wiki/Archive.today and https://en.wikipedia.org/wiki/Wikipedia:Requests_for_comment/Archive.is_RFC_5#Evidence_of_altering_snapshots.
In other words, at the moment of writing of this, if you want to legally preserve some evidence against a company you feel might try to screw you over in the future, you can either - archive those pages yourself, - ask Save Page Now and/or a similar feature of another web archiving services to try and archive those pages, hope it works (which it will not for at least some web pages that lazy-load their contents on user inputs) and then hope that the company in question won’t just request removal of those archived snapshots later.
Imagine for a second what would have happened in the DCS case if DCS were more careful and asked to remove their pages not just from the Wayback Machine, but from all public web archives too, which they easily could have done as Wikipedia has a helpful list of all of them:
Firstly, everyone could have been gaslighted into believing DCS’ claims.
Secondly, even if the original reviewer successfully resisted the gaslighting (e.g., by archiving some of the original materials themselves), to defend themselves against DCS they would have needed some independently collected evidence, which means they would have needed to counter-sue DCS, sue the original domain host owner where those pages were stored as the part of their lawsuit against DCS, and then subpoena all web archives for their copies of those pages, including the ones hidden from the public.
Needless to say, the latter option is both costly and inapplicable for investigative journalism purposes.
The same observations apply to the FetchTV case. Personally, without all those archived pages I would’ve been gaslighted by their e-mail to Louis. Their arguments there make sense until you look at their original marketing materials.
In other words, the current state of these things is sub-optimal, but a web archive operating as described above would resolve all these issues.
From the legal side of such an operation:
From a copyright standpoint, I think it would be quite hard to argue in court that a Terms of Service, a product listing, or a marketing material is “subject to normal commercial exploitation” like a movie or a song is, but it would be quite easy to argue that all such materials are designed for as wide distribution as possible anyway, are “nonprofit and educational”, with some of them also being “news”, as noted above.
While some argue that when Terms of Service of a website forbid its archival, then to ignore those them and archive it anyway is a violation of Computer Fraud and Abuse Act (CFAA), which would make some of the above operations illegal.
However, I don’t think this interpretation makes physical sense: CFAA forbids “access” to computer systems “without authorization”, and the “access” part happens while you are fetching the data, not while you are saving it to disk. So, in theory, if you are not “hacking” anything while accessing the data, you should be in the clear.
E.g., I don’t think that downloading a YouTube video is a violation of CFAA even if YouTube’s Terms of Service forbid it, which, of course, they do, to quote https://www.youtube.com/t/terms?hl=en:
The following restrictions apply to your use of the Service. You are not allowed to:
- access, reproduce, download, distribute, transmit, broadcast, display, sell, license, alter, modify or otherwise use any part of the Service or any Content except:
- as specifically permitted by the Service;
- with prior written permission from YouTube and, if applicable, the respective rights holders; or
- as permitted by applicable law;
However, while the above ideas appear to be legal to me and, with enough automation, all of the above can be implemented and run by a single person, the unfortunate reality is that you can’t be seen doing such things as a lone computer enthusiast, especially while putting your real name on it. E.g., most notably, see the case of Aaron Swartz. The fact that OpenAI, Anthropic, and other “AI” companies are essentially doing what Aaron did, but not just for published research, which should be in public domain anyway, but for all the data they can get their grabby hands on, on a scale Aaron didn’t even dream of, and now this now appears to be perfectly legal and the federal government wants to invest in them instead of prosecuting them… makes me think really unkind thoughts.
Thus, assuming FULU is a nonprofit (the website does not appear to say either way), such a web archive should probably be run by FULU for legal ass-covering reasons.
Though, of course, all these understandings need to be verified by an actual lawyer.
From the technical side, if we are to follow the mainstream opinions, then the best current tools for web archival are
However, personally, I think that the WARC format is both overly complicated and also not powerful enough for its stated web archiving purposes:
See WARC’s spec for more info.
The only supposedly good thing about it is that it’s supposedly standard, but it’s a bit of a lie because WARC outputs produced by some tools following the spec fail to be parsed properly by other WARC tools also supposedly following the spec. E.g., notably, data generated by ArchiveBox fails to be parsed by most other WARC tools. See there for discussion.
Additionally, with so many companies getting publicly caught enshittifying their products via their archived web pages I expect many such companies will soon start
thus making their sousveillance using conventional tools as hard as physically possible. After all, many news websites already do this in response to LLM companies scraping them via web archives, see there and there.
Thus, for all of the above reasons, personally, I think other technical solutions are preferable.
Personally, between 2016 and 2022 I’ve been running a personal web archival workflow by making most of my web traffic go through mitmproxy HTTP proxy, with some further processing of the resulting mitmdump files done by custom scripts. However, in 2022 CloudFlare learned to fingerprint and block mitmproxy usage, I assume as a side-effect of them learning to recognize situations where the network-level behaviour of a client does not match its User-Agent, to fight bots. This feature was then copied by most other CDNs, which quickly rendered my mitmproxy setup defunct. So, between 2022 and late 2023 I’ve been trying out various alternative methods and tools for this, eventually becoming dissatisfied with all of them, which is how I ended up developing hoardy-web (also on GitHub, licensed under GPLv3+), which is the tool I use for all my web archival needs now.
So, if it was me implementing this public web archive idea, I would rather simply add the missing bits to hoardy-web because it’s technical design work quite well for a low-maintenance web archiving service because it can generate static website mirrors via its hoardy-web mirror sub-command. hoardy-web mirror produces outputs similar to wget -mpk, but generated offline from already archived data, allowing you to tweak various rendering options without re-downloading anything.
In other words, a web archive that periodically archives a list of URLs and then renders them into dated static website mirrors can be built on top of what hoardy-web already has, plus a tiny harness connecting a browser running hoardy-web’s browser extension to cron or something. Moreover, a service built on top of hoardy-web would only really need to keep its normal GNU/Linux systems and its HTTP daemons up to date. Unlike with all other solutions, hoardy-web itself doesn’t even need to be installed on the machines that would be facing the public, thus reducing possible attack surfaces.
(Note: this section was cross-posted to CRW’s Zulip.)
Let’s make a tool similar to ALPRRadar (or its predecessor ALPRWatch), but for the consumer-protection-related documents.
That is, let’s make a tool that periodically:
and then allows:
Additionally, it could also semi-automatically edit per-company CRW pages using those human-approved notifications, adding notes about Terms of Service changes.
Most of those documents say the company may change them at any time, and while most companies notify users of those changes, not all of them do.
Additionally, it’s quite easy to become subject to these agreements without signing anything or giving those companies your contact information, in which case it’s trivial to miss them changing those terms and conditions under you.
After all, for example, as Louis Rossmann repeatedly notes in his videos, it’s entirely possible to become a subject to Terms of Service on a smart fridge without ever seeing the box they are supposedly printed on.
Additionally, having an explicit log of such Terms of Service changes would be quite useful for investigative journalism purposes, like what GamersNexus do.
I think both this project and ALPRRadar/ALPRWatch should be using FULU’s Web Archive internally, for test and reproducibility, if nothing else.
Purely technically, however, this project only really depends on CRW’s Legal Cargo, and most of it can be initially be implemented without explicit web archival support. For instance, the fetcher could be configured to transparently run all its traffic through mitmproxy, warcprox, or similar and the resulting dumps could then be added to FULU’s Web Archive later.
(Note: this section was cross-posted to CRW’s Zulip.)
Let’s create tooling that provides consumers with useful and user-configurable guidance instead of just making them depressed and teaching them to be helpless.
That is, when an Internet user visits a company website or an Internet store page (Amazon, EBay, etc), instead of ToS;DR’s tosdr/browser-extension showing them a bad Grade and FULU/CRW’s FULU-Foundation/CRW-Extension listing all the relevant evil Incidents, let’s make a tool that suggests better alternatives according to user’s own preferences instead.
Because bad grades and lists of incidents are not actionable for users while suggested alternatives are.
“But we don’t want to maintain lists of ‘better alternatives’ for each company/product on CRW! What it even means to be ‘better’? Better for whom?”
Indeed! Let’s make a tool that does this properly. That is, when a user visits a company/product web page, it should show them that company/product among its alternatives in all its service/product categories, with entries rated and sorted according to user’s own preferences.
The above can be implemented as follows:
Categories or some such field to CRW’s CompanyCargo, ProductCargo, ProductLineCargo, etc cargo templates.CRW-Extension to display alternative companies/products when said companies/products have a category in common with the company/product the user is looking at.To implement the last step:
Firstly, note that ToS;DR’s Grades are supposedly based on ToS;DR’s Cases, which are standardized statements about legal documents ToS;DR works with, like:
(The above says “supposedly” because ToS;DR’s mapping from Cases to Grades does not appear to be at all consistent to me. Also, ToS;DR’s Cases are a plain list, not a tree/graph, unfortunately. However, when you ask an LLM to summarize a ToS and it works I suspect that a large part of such LLM outputs is based on training data taken from ToS;DR.)
Therefore, with CRW’s Legal Cargo plus Categories feature described above, FULU’s Web Archive, and (optionally) ToS Watch projects implemented, we could easily make an alternative ToS;DR’s Cases-like database that
is structured as a tree/graph (which will be useful both for adding new entries and for computations described below);
attaches each Case to each separate statement in each archived Legal Cargo URL (ToS;DR’s Cases are almost this, but not quite, so some human input and then automated data massaging would be needed);
which could then be used to make a tool that could
reduce each legal URL into a vector in Cases-space;
assign such vectors to all companies/products using their CRW Legal Cargo URL fields (by taking point-wise min/max over each dimension of Case-space);
ask users for their preferences, e.g., by starting with a unit vector in in Cases-space and then editing it by asking them to rate “Privacy vs. Social Influence”, “Security vs. Convenience”, for various things, or by allowing them to rate the importance of each Case dimension separately;
compute a numeric score for a company/product by taking a scalar product of the per-company/product vector and the user preference vector;
compute such scores not just for the given company/product but for all companies/products in the same Categories when doing queries and display all the results sorted using the resulting scores.
Note that the above is reminiscent of what (Another) Consumer Rights Wiki Injector (also on CRW’s Zulip) does, except instead of such a tool just augmenting Amazon, EBay, and other Internet store pages by marking the products which you probably shouldn’t buy, this tool could also show you
In theory, such a tool could also augment store pages like the Injector does, but with custom user-defined labels, like
by using those CRW cargo exports and other data sources.
However, at the moment of writing, doing such things is actually legally risky. Facebook and Amazon regularly sue tools that modify their pages into oblivion by claiming they are in violation of their Terms of Service and, thus, Computer Fraud and Abuse Act (CFAA). As noted above, I don’t think this interpretation of CFAA is valid, since all the processing would be done on user’s own computer, but litigation to this end would be expensive.
Thus, I think that augmentation of Internet store pages should be a low-priority feature.
Also, it would be nice to eventually have a smartphone app version of this tool so that you could scan barcodes while shopping in physical stores too, but that appears to be a much lower priority task to me.
(Also, generally speaking, such a tool set is a subset of what I’m trying to achieve with my personal exocortex software system, except my preferred UIs are CLI and Emacs, while I expect most laypeople would prefer a web browser extension instead.)