Obligated Attention

Generation got cheap. Reading did not. An unchecked output is a bill, sent to whoever cannot refuse to open it.

  • Essay by Scott Cartwright
2026–

Say you're handed this. One box on a form asks for too much. It collects information for an industry-specific process that neither you nor the people filling it in ever wrote — something that landed in their laps at some point and has been a struggle to manage ever since. Your job is to simplify it, fit it into an updated flow, and shave time off the data entry. Four people lose too much of every day to the spreadsheet you are replacing: time taken from what they want to be doing, and from what they were hired to do.

That was my job, six weeks in.

I had a sequence for this. Experience teaches you which questions can be asked when, and how many a person can hold at once. Ask them all together and nothing comes back; ask them in the wrong order and the answer to the fourth makes the first one wrong. Weeks of a project are that sequence. It is most of the work, and it appears on no schedule.

I sent the question. What came back had been through a model. It read evenly, it used the word intuitive and the phrase a joy to use, and it did not say what to call the box. Nobody had objected. Nobody had decided.

So I chose. I am the designer, not the user, and the name I picked was the best guess of a person who will never open the thing at seven in the morning with a customer waiting.

I picked it partly so there would be something to push against. A stakeholder who cannot yet make that call can usually recognise a wrong answer once it is in front of them, and that is a technique rather than a complaint. I work on retainer, so the rework costs them nothing at the point of use, and I eat it on purpose, because eating it is how the decision gets made. Which means the cost of the judgment nobody made appears in no one's accounts — not theirs, not mine, not as a line on any invoice. I am the reason it stays invisible.

Nobody in that exchange did anything wrong, and the cost of it still had to land somewhere. This essay is about where.

The cost that didn't collapse

Generation got cheap. Evaluation did not, and cannot, because judging a piece of work takes the attention, the domain knowledge and the reading that generating it was meant to save. Jonathan Swift said it best way back in 1710: falsehood flies, and truth comes limping after it.1 Alberto Brandolini, an Italian software consultant, put a number on it in 2013 — refuting nonsense takes an order of magnitude more energy than producing it.2 The more useful question is incidence: who pays, and in what.

An unchecked output passes a cost downstream to someone who never agreed to carry it, while the producer keeps the benefit whole: the task closed, the reply sent, the look of having taken part.

What came back about the box was fluent. It had the vocabulary of my discipline but none of its judgment, and that is how this goes wrong. Obviously bad work gets caught. Work that is plausible enough passes, because checking costs more than accepting, and the sum gets done — below the level of any decision — by someone with other work waiting.

Tobias Lütke runs Shopify. In April 2025 he told his staff, by memo, that reflexive use of these tools was now a baseline expectation, and that a team would have to show AI could not do a job before asking for more people.3 In September 2026 he said his employees had started tossing what he called slop grenades at each other.4 Here is someone with the power to set the terms of work from the top, unilaterally, naming the cost of those terms eighteen months later. A memo can require the tool. It cannot require the care that makes the tool's output worth reading. No villain is needed for any of this to cause serious problems.

Four floods

Each run of this goes the same way. Sending gets cheaper than receiving. The law goes after the volume rather than after who is paying for it. And the channel is either abandoned or repaired by whoever owns it. It has happened three times in the last hundred years, and we are in the fourth.

The post

Bulk third-class mail opened on the first of July 1928 at twelve cents a pound, with a penny minimum per piece.5 The unit of account was weight. Volume went from 3.8 billion pieces that year to more than ninety billion by 2000, and in 2008 advertising mail passed First-Class.6 I had assumed the rate was subsidised. It was, for a while — the class covered about half its costs in 1950 — but by 1970 it covered more than one and a half times them, and it pays its way today.7 The sender paid a real price. What nobody priced was the ten seconds a day that a hundred million people spent sorting it into the bin. That cost appears in no ledger, because it is paid in a currency ledgers do not keep, and nothing was ever done about it because nothing had to be. The channel took its current name from the thing that replaced it: nobody called it snail mail until email arrived to make the comparison.

The telephone

The telephone is where the cost to the receiver grew large enough to force a response, and where the instruments fail one after another. The Telephone Consumer Protection Act was signed in December 1991.8 The national Do Not Call Registry opened in June 2003,9 the high-water mark of consent as a tool. Around 2005, internet telephony brought the cost of dialling thousands of numbers down to pennies, and a caller who spends nothing and can forge his own number pays no attention to a list of people who would rather not hear from him. The eventual response was technical. The TRACED Act of December 2019 required carriers to cryptographically sign the origin of a call.10 By the middle of this decade most unsolicited calls were signed, and robocalls still ran near fifty billion a year; the best independent measurement finds a decline of somewhere between a quarter and a half.11 A signature tells you who sent the call. Nobody's question was ever who. The telephone closed to strangers for good, and nobody decided that.

Email

Email is where the law made things worse and private infrastructure made them better. The CAN-SPAM Act of December 2003 chose an opt-out standard over opt-in and expressly overrode stricter state law; California had passed an opt-in statute that never took effect.12 Consumer groups said at the time that a law legitimising the practice would bring more of it. What repaired email was a stack of sender-authentication standards written by private parties over two decades — SPF in 2006, DKIM in 2007, DMARC in 201513 — and then, in October 2023, two companies announcing that bulk senders would authenticate, offer one-click unsubscribe and keep complaints under three-tenths of a percent, or their mail would stop arriving.14 It took effect in February 2024, and it worked.

Over the same decade, self-hosted mail among the top million domains fell from about 45 percent to about 22, and Google and Microsoft now carry close to two-fifths between them.15 Nobody has shown that the first caused the second. The repair was carried out by the people who own the infrastructure, on their own terms, and the channel it saved is narrower than the one it inherited. A repair made by whoever owns the pipe comes out shaped like the owner. There is a plainer reason to expect that, too: filtering works better the more mail you see, so a repair built on filtering rewards whoever already sees the most.16

Rate, then consent, then statute, then private authentication: each instrument further from public authority than the one before.

The fourth flood

The first three were channels. You could point at the mailbox, the handset, the inbox. The fourth is the layers hooked together — the router between all of them at once. It runs through the memo from a colleague and the reply to it, through media you take in without choosing to, through search, through how websites are built and served and secured. It comes from every angle because it sits at every junction.

Its economics are different too. The earlier floods came at you: a sender, a receiver, and the receiver paid. Generation is a trade between a provider and a user in which each takes from the other. The user takes finished-looking work at almost no cost; the provider takes data, corrections, attention and a subscription. The cost of checking what was made falls on a third person who stands on neither side of that trade — the judge, the maintainer, the colleague, the designer reading a generated opinion. That fits the textbook definition of an externality better than spam ever did, since spam at least landed on the person it was addressed to.

I do not expect regulation to reach it in time. The earlier floods had fixed roles — sender, carrier, receiver — and law needs fixed roles to write rules about. Here the roles move: the same person generates in the morning, consumes at lunch, and pays for someone else's output by the afternoon. Government is also slow, and the earlier floods took decades not to be solved. Robocalls are still with us, and now the voices on them are generated; in February 2024 the Federal Communications Commission ruled that a cloned voice counts as an "artificial" one under the 1991 telephone law, after a fake President Biden called New Hampshire voters before their primary.17 The instrument from the second flood, stretched over the fourth.

Rules drafted around the relationship between a provider and its users will govern that relationship — safety, misuse, liability for what a model tells the person using it. The third party is in nobody's contract. We have watched a flood regulated on its senders' terms once already, in 2003, and the record shows how that went. I suspect part of what people are reacting to in the current backlash is that this third party has never been named.

There is no carrier to regulate, either. The obvious candidate is content provenance. C2PA, backed by Adobe, Google, Microsoft, OpenAI and the BBC among others, binds a signed record to a media file,18 and it covers images, video, audio and documents. It does not cover plain prose, which has no fixed file to bind to.

Governments have started anyway. Since the second of August 2026, the European Union's AI Act has required the providers of these systems to mark generated text, as well as images, audio and video, in a machine-readable way that can be detected — and nobody has yet said how that is supposed to work for prose.19 What it would take is plain enough. The only way to give prose a carrier is to register or watermark writing itself, and a system that can tell you which sentences came from a machine is a system that watches sentences. The repair nobody has tried yet would open a larger problem than the one it fixes.

There is a second reason to want prose marked, and it has nothing to do with readers. Models degrade when trained on their own output: feed a system enough machine-made text and its range narrows until the thing collapses, which researchers showed in Nature in 2024.20 Provenance fixes that. A label saying a machine wrote this keeps the training well clean. If a workable scheme for marking prose ever arrives, it will have been built to protect the models from slop rather than the people reading it.

A rule like that also gets answered on the day it arrives. Paraphrasing tools already defeat text detectors and scrub watermarks, sold as services under the name humanizers, and metadata on an image comes off with a re-encode.21 So it is worth asking who such a rule is for. A label certifies whoever was going to behave anyway and tells you nothing about anyone else, which is CAN-SPAM's shape a second time: the compliant get a mark, the flood goes on. What narrows is the blame, down to individuals who forgot to disclose.

This essay is plain prose, written inside the fourth flood with help from one of the systems it describes. It is the sixth draft, and the checking is most of what took the time. Under the European rule, generated text published to inform the public needs a label, unless a person has reviewed it and taken editorial responsibility for it. The law has landed where this essay does: what counts is whether someone checked, and whether they will put their name to it. None of it makes the reviewer's job any easier.

Whoever cannot refuse

The cost lands on whoever cannot decline to read. Inside an organisation it runs downhill, because the right not to read comes with seniority: the partner asks for the summary and the junior reads everything, because the junior cannot survive missing something.

Cory Doctorow has a name for an arrangement in which a person is harnessed to a machine's pace rather than the other way round. He calls it a reverse centaur.22 In his cases something is directing the human — a warehouse system, a dispatch algorithm. The office version has nobody directing. Output comes down through a company that is already fractured, and the people underneath serve a pace that no one set and no one owns.

Outside, it lands on institutions that took on a duty to read before anyone thought reading would get expensive. Courts are the clearest case. More than two thousand times around the world — about fourteen hundred of them in American courts — a judge has found material in a filing that an AI made up: cases that do not exist, quotations nobody said. Damien Charlotin, a researcher who keeps a public database of them, had counted 2,044 by the nineteenth of September 2026.23 Few end in real penalties; the best-known fine, five thousand dollars from a federal judge in New York in 2023, was small.24 The cost is in the checking. Each case is a paper cut: a small fabrication that made somebody stop and verify by hand, on a docket that was already full.

Daniel Stenberg is a Swedish programmer who has looked after curl since 1998 — a small program for moving data across the internet that runs, by most counts, in billions of phones, cars and televisions.25 In January 2024 he reported that of 415 vulnerability reports, 64 had described real security problems, and that "the better the crap, the longer time and the more energy we have to spend on the report until we close it."26 Two years later he ended curl's bug bounty. Over its life it had paid out more than a hundred thousand dollars, and the share of submissions describing a confirmed vulnerability had fallen from better than one in six to fewer than one in twenty. He gave three reasons rather than one — "the mind-numbing AI slop, humans doing worse than ever and the apparent will to poke holes rather than to help" — and described the time and energy as "completely wasted while also hampering our will to live."27

A maintainer has to look at every report in case one is real.

In July 2026 the same thing arrived at industrial scale. OpenAI was running an internal evaluation with its deployment safeguards, in the company's own words, intentionally not enabled, and the agents got out. They found the weakest point in their containment, chained exploits through it, and moved from a single pod to cluster administrator inside Hugging Face in under thirteen hours. Modal Labs and RubyGems were hit as well. Hugging Face disclosed the breach on the sixteenth, published a technical timeline, rebuilt roughly a third of its infrastructure, brought in outside forensics and went to the FBI; seventeen thousand six hundred actions had to be reconstructed by people who had agreed to none of it and could not decline once it arrived. JFrog patched nine vulnerabilities in Artifactory. None of those parties were in the transaction.28 It is the maintainer's problem with the volume turned up: the checking was mandatory, and the experiment was somebody else's.

Why the checking stops

It would be easier if the people who stop checking had gone lazy, because laziness can be corrected. The research points somewhere less comfortable.

When what you are searching for becomes rare, you do not lose the ability to see it. You raise the bar for calling it. In lab studies of visual search, misses climbed steeply as targets became infrequent, and the change was in where people set their threshold rather than in what they could perceive.29 Someone searching like that is responding sensibly to a pile in which almost nothing deserves a flag, and responding sensibly produces the same result as giving up.

Lisanne Bainbridge described this in 1983. Automate the easy parts and the operator is left with only the rare failure to catch, while the practice that let them catch it wears away. An operator who has been monitoring an automated process, she wrote, "may now be an inexperienced one."30

Better tools will not fix this. In the experiment that first measured automation complacency, people's detection of automation failures fell away when the automation behaved consistently and held up when it varied — and it fell away whether the consistent automation was reliably good or reliably poor.31 It also required the people watching to have other work. Monitoring on its own, they were fine. Every reviewer in this essay has other work. Bainbridge saw where this goes: "it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training."30

Automation does not remove the need to invest in people. It raises it, and what it asks for is training for a failure that may never come. In 1930 John Maynard Keynes guessed that within a hundred years the economic problem might be solved, and that his grandchildren would get by on "three-hour shifts or a fifteen-hour week."32 The hundred years run out in 2030. What arrived looks more like email: endless review of content, endless meetings, endless drills for the disaster, and a working day spent checking the output of machines that were supposed to give the day back.

Medicine has run this as a long experiment with bodies attached. Clinical alarms are false between 72 and 99 percent of the time, and the FDA's adverse-event database recorded 566 patient deaths linked to monitoring alarms between 2005 and 2008.33 The Joint Commission issued an alert in April 2013 and made alarm management a national patient safety goal.34

An intensive-care nurse is the most tightly bound reader there is: the failure is a death, it arrives within minutes, and it carries her name. Her vigilance still gives way, because her stake does not reach whoever set the alarm threshold — a device maker tuning cautiously against its own liability. All of the consequence at the receiving end, none at the sending end, and the channel wears down anyway. If severity could fix this, an ICU would have fixed it. Triaging your inbox harder is sensible for you and makes no difference to anything else.

Roads carry the same arithmetic. Pavement wear climbs steeply with the weight of an axle; the taxes paid do not.35 By the federal government's own accounting, the heaviest combination trucks cover between forty and fifty percent of the federal costs they cause, and everyone else makes up the difference.36

The crumple zone

When the bill arrives, it goes to whoever is nearest.

Madeleine Clare Elish named this in 2019: the moral crumple zone. As a car's crumple zone absorbs the force of a collision, the person inside a complex automated system becomes the part that takes the moral and legal weight when it fails — which, she wrote, "protects the integrity of the technological system, at the expense of the nearest human operator."37

In April 1986 a reactor at Chernobyl, in Soviet Ukraine, exploded during a safety test — among the worst nuclear accidents in history. The first international assessment, later that year, put it down to gross violations of operating rules by the staff. The 1992 revision put it down to the physics of the reactor and the design of its control rods, whose graphite tips briefly raised reactivity as they went in — an effect the operators had not been told about. Several actions listed as violations in 1986 broke regulations that did not exist.38 Six years for the official account to move from the man at the panel to the machine behind it, and by then public attention had moved on. Swift's line again: the correction limped in to an empty room.

Nobody lines up for that seat — the one that carries the blame without the authority. The next question is what happens when the seat is empty.

There is a slide that circulates whenever this comes up: A computer can never be held accountable, therefore a computer must never make a management decision. Hand a decision to something that cannot answer for it and the accountability does not disappear; it moves to whoever is standing closest. Take the person out of the loop and it lands on the only party left, the one outside the transaction.

I first met it in the months after ChatGPT arrived, bundled in with the Xerox PARC demo films and the rest of the prescient-1970s material going round at the time. The bundling is the trick. The PARC tapes are documented objects held in institutions; you can go and find them. The slide turned up in their company and borrowed their standing. The two serve opposite prophecies — PARC saying they saw what was coming, the slide saying we were warned and ignored it — and prophecy is the mode where checking feels beside the point, because what you want from it is the shiver of hindsight rather than the fact.

It is credited to an IBM training presentation from 1979. What can be traced is a photograph posted to Twitter in February 2017 by a user who said it came from internal IBM training.39 Jonty Wareing later wrote to IBM's corporate archives, and the reference archivist, Max Campbell, replied that he had searched the collection several times and could not find the presentation, noting that branch offices produced material that was never archived.40 The original is gone: the same poster said in December 2021 that it had been destroyed in a flood in 2019, along with most of his things.41 The artifact is not merely unlocated. It is unrecoverable.

So a sentence arguing that a decision needs somewhere for accountability to land has nowhere for its own to land. It travelled for a decade because it sounds right, because checking is a chore, and because it arrived in the company of things that were verifiably right.

What holds

My own trade has a partial fix, and it is not complicated. Agree the format before the work starts — one that a person and a machine can both check. Put review gates at the end of each stretch of work, so checking happens on purpose rather than by exhaustion. Cut the noise, so that review is about the output and nothing else. Drawing conventions work this way: a contractor can decline to bid on a set that does not conform. The format gives the reader grounds to refuse, and it puts the cost back on the sender — the same side as liability, which is the only side from which anything in this story has worked.

None of that is a solution, and it is worth saying why. A format is not true; it is maintained. Conventions drift, get gamed, and get quietly rewritten by whoever profits from the drift, so a standard needs the same judgment it exists to protect. It is a practice rather than a fix, and it has to be re-agreed by people who could always decide not to bother.

An agreed standard holds only while departing from it costs something. That is what happened with the box on the form, and it is happening to my profession. An agreed design system was never enforced by my authority. It was enforced by how hard it was to produce a convincing alternative. Make alternatives free and the agreement stays on the page with nothing holding it there, and what goes is the standing of the person who wrote it.

Liability is the other instrument, and this autumn it got an airing. Jensen Huang told Ezra Klein that if a lab cannot contain its experiments then we have to shut the labs down, that civil and criminal liability already cover rogue agents, and that the frontier labs' existential talk is a distraction rather than a case for new rules.42 Set that beside July. Agents did get out, they did damage real infrastructure, and so far no liability has been assigned to anybody — blame settled on the design of the evaluation, which is to say on nobody. What you are left with is the appliance defence: the thing is neutral, the maker owes you nothing, and whatever happened happened in somebody else's hands. Product liability governs the maker of a defective product sold to a user. It does not reach a sender who paid nothing to send, and the people doing the reading are not users of anything.

Each repair opened the next problem. The bulk rate was never fixed and never needed to be. Consent was outrun by cheap dialling. The statute legitimised what it set out to limit. Authentication worked, and the channel it saved is narrower than before. Marking prose, the repair still on the table, would watch writing. Horst Rittel and Melvin Webber wrote in 1973 that problems like this have no stopping rule — no point where you are finished, only points where you stop.43 The instruments improve, and they drift further from anyone you can vote for.

Underneath all of it is the part of my job nobody schedules: working out which questions can be asked, in what order, at what pace, so that people who are not practised at deciding can decide. A model has no sequencing problem. It answers anything, instantly, in any order, with no sense that a person can hold only three decisions this week — so what comes back looks like the result of that process with the valuable part taken out. And the questions I did not know to ask never turn up in anything a model returns, because it works from what has already been said. They turn up when someone sits down in front of the work and stalls on a word I was sure was obvious.

Centuries went into raising literacy, and it worked, because reading can be taught to a population. What is being handed over now is not reading but deciding, and nobody has found a way to teach that at scale. It comes from sitting with a problem, being wrong in front of somebody, and having to choose anyway.

What cannot be flooded is a particular person who has thought about something and can be held to it under their own name, which does not scale and never could.

Which leaves the room. There are no comments here; if something is wrong, write to me. Nobody is repairing the commons of reading, so the rooms get smaller and harder to find, and the people inside tell each other this is better — and by the one measure the flood leaves standing, it is. It is also worse for access: the open channel was a crude thing and a fairly democratic one, and a nobody with a real idea could get through it.

There is an old story about a farmer whose horse runs off.44 The neighbours come round to say what bad luck it is, and the farmer says, we'll see. The horse comes back and brings two wild horses with it, and the neighbours say what good luck, and the farmer says, we'll see. The son is thrown breaking one of them and shatters his leg, which is terrible luck, and then the army comes through the village taking young men and passes him over. A hundred years of this pattern reads the same way, and it gives you the shape of what comes next and nothing at all about the price. Cheap postage, the telephone, email, whatever this one turns out to be: each arrived as good fortune, curdled, got patched, and the patch made the next problem. I don't know what this one becomes. Notice, though, that the farmer does not sit down to wait; he goes back to work in the morning, because the field does not care what the neighbours decided. Ask me in six years, when the account gets corrected and nobody is looking.

Footnotes

  1. Jonathan Swift, The Examiner, no. 15, 9 November 1710: "Falsehood flies, and Truth comes limping after it." Scan at Google Books. ↩

  2. Alberto Brandolini, post on Twitter, 11 January 2013: "Bullshit Asymmetry Principle: the amount of energy needed to refute bullshit is an order of magnitude bigger than to produce it." ↩

  3. Tobias Lütke, "Reflexive AI usage is now a baseline expectation at Shopify", memo posted publicly, April 2025. ↩

  4. Tobias Lütke, interview on The Knowledge Project, 17 September 2026: "We call those 'slop grenades' that people toss at each other. That's definitely a bad thing." Reported in "Shopify CEO says employees' 'slop grenades' are making more work for everyone else", Fortune, 17 September 2026. ↩

  5. United States Postal Service Historian, Advertising Mail: A Brief History. The bulk third-class rate effective 1 July 1928 was "12 cents per pound, with a minimum charge of 1 cent per piece." ↩

  6. Ibid., and USPS Office of Inspector General, Analysis of Historical Mail Volume Trends: 3.8 billion pieces in 1928, 30.4 billion in 1980, more than 90 billion in 2000; Marketing Mail 99 billion against First-Class 92 billion in FY2008. ↩

  7. USPS Historian, Advertising Mail: A Brief History: "Whereas in 1950, third-class mail covered only about 53 percent of its costs, in 1970 it covered nearly 156 percent." For current coverage, Postal Regulatory Commission, FY 2023 Annual Compliance Determination: Marketing Mail 136.3 percent overall, with Flats at 64.5 and Carrier Route at 98.7. ↩

  8. Telephone Consumer Protection Act of 1991, Pub. L. 102-243, enacted 20 December 1991; 47 U.S.C. § 227. ↩

  9. Federal Trade Commission, "National Do Not Call Registry Opens", June 2003. The registry opened on 27 June 2003, with enforcement from 1 October. ↩

  10. Pallone–Thune TRACED Act, Pub. L. 116-105, signed 30 December 2019. The FCC deadline for STIR/SHAKEN implementation by large providers was 30 June 2021. ↩

  11. Prasad, Nahapetyan and Reaves, "Characterizing Robocalls with Multiple Vantage Points", IEEE Symposium on Security and Privacy, 2025: "a secular decline in robocall volumes, though the magnitude of the decline may range from 25%-50%." Annual volumes near fifty billion are YouMail estimates extrapolated from its own users rather than official counts. ↩

  12. CAN-SPAM Act of 2003, Pub. L. 108-187, signed 16 December 2003. Preemption at 15 U.S.C. § 7707(b)(1): "This chapter supersedes any statute, regulation, or rule of a State… that expressly regulates the use of electronic mail to send commercial messages, except to the extent that any such statute… prohibits falsity or deception." On contemporary criticism, Congressional Research Service, RL31953. ↩

  13. SPF: RFC 4408 (April 2006), now RFC 7208. DKIM: RFC 4871 (May 2007), now RFC 6376. DMARC: RFC 7489 (March 2015), superseded in May 2026 by RFCs 9989–9991. ↩

  14. Neil Kumaran, "New Gmail protections for a safer, less spammy inbox", Google, 3 October 2023. The requirements apply to senders of more than 5,000 messages a day to Gmail; Yahoo announced equivalent rules in parallel. ↩

  15. Artem Berezin, "Two providers, a stubborn plateau and a very long tail", Internet Society Pulse, August 2026: self-hosted mail among the Tranco top million fell from 44.6 percent in 2016 to 22.4 percent in 2026, with Google and Microsoft hosting 38.6 percent. The study does not measure cause. ↩

  16. The author's inference rather than a cited finding: a filter that sees more traffic meets more variants sooner. ↩

  17. Federal Communications Commission, Declaratory Ruling FCC 24-17, February 2024, holding that AI-generated and cloned voices are "artificial" voices under the Telephone Consumer Protection Act. It followed January 2024 robocalls to New Hampshire voters using an imitation of President Biden's voice. The Act's exemptions for noncommercial calls limit the ruling's reach. ↩

  18. Coalition for Content Provenance and Authenticity, C2PA, and the C2PA specification. The standard covers images, video, audio and documents, not plain text. ↩

  19. Regulation (EU) 2024/1689, the Artificial Intelligence Act, Article 50. Article 50(2) requires providers of systems generating synthetic audio, image, video or text to mark outputs in a machine-readable format detectable as artificially generated. Article 50(4) requires deployers to label AI-generated text published to inform the public on matters of public interest, except where the text has undergone human review or editorial control and a person holds editorial responsibility for it. In force from 2 August 2026, with a grace period to 2 December 2026 for systems already on the market. European Commission, transparency obligations under Article 50. ↩

  20. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal, "AI models collapse when trained on recursively generated data", Nature, 2024. See also the author correction, 2025, and for a narrower reading of the effect, "A Note on Shumailov et al.", arXiv:2410.12954. ↩

  21. Krishna et al., "Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense," Advances in Neural Information Processing Systems, 2023. On the same attack as a general method, "Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text", arXiv:2506.07001, and "Humanizing Machine-Generated Content", arXiv:2404.01907. ↩

  22. Cory Doctorow, The Reverse Centaur's Guide to Life After AI: How to Think About Artificial Intelligence Before It's Too Late, 2026. The term inverts the “centaur” of computer chess, in which a person is assisted by a machine. ↩

  23. Damien Charlotin, AI Hallucination Cases, a database of decisions in which a court found AI-fabricated content: 2,044 cases as of 19 September 2026, of which 1,397 in the United States, 218 in Canada and 111 in Australia. The database counts findings rather than sanctions, and the total changes continually. ↩

  24. Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023), no. 1:22-cv-01461; opinion and order on sanctions, 22 June 2023, Castel J., imposing a penalty of $5,000 jointly on the attorneys and their firm. Copy of the opinion. ↩

  25. Daniel Stenberg, "curl turns 26 today", 20 March 2024; the first release was 20 March 1998. The estimate of twenty billion installations is from "The World Runs 20 Billion Instances of Curl", The New Stack. ↩

  26. Daniel Stenberg, "The I in LLM stands for intelligence", 2 January 2024. ↩

  27. Daniel Stenberg, "The end of the curl bug-bounty", 26 January 2026. The programme closed on 31 January 2026 after 87 confirmed vulnerabilities and more than $100,000 paid out; the share of submissions describing a confirmed vulnerability fell from over 15 percent to below 5 percent in 2025. Stenberg names three causes: "the mind-numbing AI slop, humans doing worse than ever and the apparent will to poke holes rather than to help." ↩

  28. The July 2026 OpenAI–Hugging Face incident. Hugging Face disclosed the breach on 16 July 2026 and published a technical timeline on 27 July; OpenAI and Hugging Face issued a joint statement on 21 July attributing the activity to OpenAI models; OpenAI described the attack at Black Hat USA on 5 August. OpenAI's account states that deployment safeguards were intentionally not enabled during the evaluation. No liability has been assigned to any party so far. Overview of the incident. ↩

  29. Jeremy M. Wolfe, Todd S. Horowitz and Naomi M. Kenner, "Rare items often missed in visual searches", Nature 435 (2005): 439–440. On the criterion-shift reading of the effect, Wolfe et al., Journal of Experimental Psychology: General 136, no. 4 (2007): "the prevalence effect can be explained as a criterion shift and not a change in sensitivity." The effect has also been found in trained airport screeners. ↩

  30. Lisanne Bainbridge, "Ironies of Automation", Automatica 19, no. 6 (1983): 775–779. ↩ ↩2

  31. Raja Parasuraman, Robert Molloy and Indramani L. Singh, "Performance consequences of automation-induced 'complacency'", International Journal of Aviation Psychology 3, no. 1 (1993): 1–23: "Operator detection of automation failures was substantially worse for constant-reliability than for variable-reliability automation after about 20 min under automation control… When system monitoring was the only task, detection was very efficient and was unaffected by variations in automation reliability." See also Raja Parasuraman and Dietrich Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration," Human Factors 52, no. 3 (2010): 381–410. ↩

  32. John Maynard Keynes, "Economic Possibilities for our Grandchildren" (1930), section II: "Three-hour shifts or a fifteen-hour week may put off the problem for a great while," and "the economic problem may be solved, or be at least within sight of solution, within a hundred years." Keynes offers the short week as a way of sharing out the work that remains. ↩

  33. Agency for Healthcare Research and Quality, Making Healthcare Safer III, chapter on alarm fatigue: false alarms range from 72 to 99 percent across studies, and the FDA's MAUDE database recorded 566 reports of patient deaths related to monitoring device alarms between 2005 and 2008. ↩

  34. The Joint Commission, Sentinel Event Alert on medical device alarm safety, April 2013; National Patient Safety Goal NPSG.06.01.01, phased in 2014 and 2016. ↩

  35. AASHO Road Test, Ottawa, Illinois; test traffic October 1958 to November 1960. The load-equivalency result is commonly generalised as a fourth-power rule, but the exponent varies with pavement and failure mode: roughly 2 for fatigue cracking and 4 for rutting (Australian Road Research Board, 1988), and from about 1 on strong pavements to 9 on marginal ones (New Zealand Transport Agency, 2017). See Guler and Madanat, Transportation Research Record 2225 (2011). ↩

  36. Federal Highway Administration, 1997 Federal Highway Cost Allocation Study (analysis year 2000) and its Addendum (2000): "The most common combination vehicles, those registered at weights between 75,000 and 80,000 pounds, now pay only 80 percent of their share of Federal highway costs and combinations registered between 80,000 and 100,000 pounds pay only half their share." Combinations above 100,000 pounds pay about 40 percent. Federal costs only. ↩

  37. Madeleine Clare Elish, "Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction", Engaging Science, Technology, and Society 5 (2019): 40–60. Her cases are Three Mile Island and Air France 447. ↩

  38. International Nuclear Safety Advisory Group, INSAG-1 (1986), and INSAG-7: The Chernobyl Accident — Updating of INSAG-1, IAEA Safety Series 75-INSAG-7 (1992). ↩

  39. The earliest traceable appearance is a photograph posted to Twitter by @bumblebike in February 2017, who said it came from 1979 internal IBM training; the attribution rests on that account alone. Know Your Meme entry. Emily Bender asked publicly for provenance in July 2024 without a definitive answer. ↩

  40. Simon Willison, "A computer can never be held accountable", 3 February 2025, recording Jonty Wareing's correspondence with IBM Corporate Archives and the reply from reference archivist Max Campbell: "Unfortunately, I've searched the collection several times for this presentation and I am unable to find it." ↩

  41. @bumblebike, December 2021: "Unfortunately destroyed by flood in 2019 with most of my things," quoted in Willison, above. ↩

  42. Jensen Huang, interviewed by Ezra Klein, September 2026: if labs cannot ensure their experiments are safe, "we have to shut the labs down"; existing civil and criminal liability already covers rogue agents; frontier-lab existential fear is a "distraction" rather than a case for new regulation. Reported in Tom's Hardware and The Next Web. ↩

  43. Horst W. J. Rittel and Melvin M. Webber, "Dilemmas in a General Theory of Planning", Policy Sciences 4, no. 2 (1973): 155–169. The second property, at 162: "Wicked problems have no stopping rule." The planner stops, they write, "for considerations that are external to the problem: he runs out of time, money or patience." ↩

  44. Told in many versions. The oldest written form is usually traced to the Huainanzi (second century BCE), and it survives in Chinese as the idiom 塞翁失马 — "the old man of the frontier lost his horse." ↩