The Power of Babble

A Guide for Thinking Chatbots

Dystopian literature often serves as self-pitying war propaganda for the powerful: a shrill compendium of elite grievances; a comically grim warning that anyone who ignores their noble demands may threaten our cherished hegemonic norms. Today’s best-selling values are under attack, and may fall at any second to views that everyone respectable dourly condemns… read all about it in this brand new “banned” book! I don’t think victimhood or weakness endow groups with any particular moral status (or at least not one that should be privileged), but precisely for this reason I’m put off by such dishonest and cowardly establishment wheedling. If anything, a given conflict should only escalate so aggressively when at least one side is over-confident in its mission—or else we could all just calculate the likely outcome, and both sides would avoid much loss. Further, if your cause is just, you should motivate your grunts with positive goals, right? Something’s gone terribly wrong when unstable whiney critics have captured power, and fictional dystopias are their excuse.

For example, the Malleus Maleficarum—a book which successfully argued for the use of torture against suspected witches, and upon confession their punishment by death—begins by claiming that rampant unchecked witchcraft could soon cause the apocalypse; it then includes a papal bull granting priests the “full and entire liberty” to stamp out sorcery, followed by a notarized statement of unanimous support by the local university’s doctors of theology. Next there’s an extended section criticizing any clergymen who doubt the widespread reality of this crime. Only after all this throat-clearing do we get the diagnostic manual and the trial guide, with clear step-by-step instructions on how to identify and prosecute such supposedly overbearing menaces. Each page practically screams out, “Look, witch-lovers, everyone important and powerful agrees that we really need to do something about this, because otherwise nobody will ever do anything of any importance about it at all, so let’s do everything we can right now!” (Naturally, the author’s real issue lay with insufficiently zealous bureaucrats, who had exiled him from his parish for obsessing over one particular acquitted woman’s alleged promiscuity). The same basic story describes Matthew Hopkins, an itinerant witch-finder who during England’s revolution through some depraved spell-work single-handedly hexed more than a hundred souls gallows-ward for witchcraft… for social stability, of course, to calm those troublingly bloody times. England as a whole hunted fewer witches through the entire prior century combined than he did in just three years, because this whiney degenerate kind of cultural coordination can be quite angrily powerful.

Modern-day analogues are almost too obvious to mention. You can make a decent career out of anti-fascistically whipping up mobs against whoever might harbor some pre-smartphone conservative sentiments, since those reactionary extremists existentially threaten us now: just look at how disagreeable and low-status the self-identified right-wingers have become, ever since we’ve thoroughly stigmatized their beliefs… do you really think such dinosaurs are worth defending? Likewise, during the Salem trials—which ironically occurred when the King revoked all Massachusetts law, leaving litigants without a judge’s firm paternalistic limits—one chorus of attention-seeking teenage girls cried out “me too!” every time that any of them claimed to have been victimized by someone else’s problematic behavior; each defendant who confessed and then blamed others for seeming similarly witchy got pardoned, while those who refused to play along died. Unfortunately I can’t think of any modern-day analogues to this, but Arthur Miller’s famous play about it apparently allegorized a “red scare,” during which an uncouth populist senator tried investigating some of the countless communist spies now known to have utterly infested our postwar government (though I’m sure that having to testify before congress about your own alleged treason would have seemed scary and, moreover, inconvenient, especially for those whom they caught red-handed). Or see Robert Coover’s best-selling award-winning novel about the awful pogrom which we all must vigorously agree, indeed, that out-of-control idiotic groupthink committed against the Rosenbergs, yes quite, just because they gave the atomic bomb to Stalin.

And now a major television show called “The Handmaid’s Tale” solemnly evokes these puritanical roots to warn us of an imminent slide into pro-natalist theocracy, because our Supreme Court finally stopped banning the kinds of state-level abortion laws that are common among peer nations (e.g., 47 out of 50 European countries limit elective terminations to before 15 weeks). Were we actually on the road to such a fate, wouldn’t our spooky fiction imagine a world like this one, in which, instead of maternity leave, retail giants proudly offer their pregnant service workers “pro-choice vacations” to blue states? Where people overwhelmingly end up with fewer children than they’d have liked, and every major country suffers from well-below replacement-rate fertility, and there are fake moral panics about any body politic that chooses to limit non-medication feticides? “A bizarre and ugly cult somehow seized complete control over one political party, and made them keep introducing bills to legalize ritual child sacrifice through the day of delivery. Moloch wasn’t even satisfied that unregulable mail-order mifepristone pills rendered the surgical option obsolete through 20 weeks, because this worship of cruelty was His point.” The Romans and Spaniards produced this kind of propaganda to warn their people about the Carthaginian and Aztec empires, but our polite society lives in a constantly aroused fear of returning to the pre-Roe years, when almost 100 women died annually across the whole US from all abortion-related causes combined. What devilry really makes them so afraid, and how do its lurking servants work their curses upon this fictive country? Find out next week, on TermiNation.

Our conspiracists fall into these very same traps too. They try to see through tabloid establishment narratives by looking for secret puppet-masters, hidden behind the blockbuster villains that our cultural machine points out, without recognizing that the power to frame celebrities as threats comes from behind the camera. It’s like the joke about a blonde who keeps on pointing out abusers, because everything she touches with her broken index finger hurts. For instance, the John Birch Society imagined that unstable Soviet agents had infiltrated our strong innocent healthy country in order to Russify it, even though Western intellectuals leaned smotheringly communist since well before 1917. David Icke and Alex Jones are the most prominent figures who claim that our elites descend through an all-powerful unbroken lineage from supernatural antediluvian god-kings… yet this apparently undermines ruling class legitimacy, because it comes to us alongside petty moaning about how these timeless illuminati reptoids might soon undermine our vibrant somehow independently long-standing human traditions? Yes, a few scattered insiders had varying degrees of now-hushed foreknowledge about 9/11, and many well-connected conmen profited handsomely off the war on terror, but that’s a symptom of weakness or indecision at the wheel, not evidence for all-powerful cabals behind the scenes—hell, Rumsfeld had been trying to downsize the Pentagon, and Iran ended up with more control over its region, though I’m sure your feelings that Bush seemed like a very scary nationalist come from someplace valid. In short, conspiracy theories can be thought of as over-fitting on the data: nothing can disprove them because they’re so willing to change the contours of their beliefs to fit the superficial gradients of our world; this radical open-mindedness to new information could involve deciding that actually W is a secret Muslim because he split Iraq between ISIS and al-Sadr, instead of you accepting that maybe his people weren’t in charge. After all, Kubrick foretold that in 2001 a cult which worships a large black desert cube would come to power, and that’s why they killed him before the millennium! In machine-learning, this “catastrophic forgetting” of implicit details (like Dubya’s presumed Christianity) allows an overfit program to discard its prior claims, which are perhaps now inconvenient, without ever admitting fault, or thus improving. It’s as if they look out on an irredeemably incomprehensible universe and then demand that someone be cathartically to blame for all this confusion.

But such over-fitting also describes the dystopian literature which more intentionally excuses each immaturity in power. Look at how Pravda teamed up with its de facto rival Party, the Cheka, to screech out “Failures in the harvest must imply saboteurs and wreckers hiding among the field-hands, who will surely take over soon unless we redouble our vigilance! Or do you perhaps doubt our system’s merit?” For example, racial disparities offend our sacred sensibilities out of all proportion to any purported material harms, and always only provide evidence of a dark magic known as racist bigotry… either in society at large, or else among those researchers who maybe suggest some other potential causes. And so the risk of imminent race war seems to threaten us from every institution that lacks the proper diversity apparatchiks. Broadly, this is what distinguishes real science from social science as well: the former hypothesizes clear causal mechanisms from which practitioners then try to contrive extreme tests, aiming to reject said mechanistic pathways as fake; in contrast, the latter merely experiments with interventions that aim to produce better outcomes than a null they reject as bad. Consider the difference between the Michelson-Morley experiment (which demonstrated that the speed of light does not depend on the speed of its source) and a randomized controlled trial, which shows that giving people some treatment yields a favored outcome relative to giving them placeboes. When you have the sovereignty to define your preferred null hypothesis—i.e. the counterfactual dystopia, if you don’t get your way—then of course this just ends up as a rhetorical method for blessing further technocratic power-grabs! Giving boys estrogen results in them behaving less violently, therefore we should keep dumping atrazine in the water supply. Can you imagine extra evidence ever not helping them apply their worldview harder?

These two types of intelligence are quite distinct, and each implies a different kind of dystopaeanism. Above, I’ve associated the boring sort with what we exaggerate when we overfit (when we take any residual in ordinary explanations as evidence of latent potential, a spell just waiting for us to exploit it, yet simultaneously understand all noise as interference done by witches, who are somehow thus actually in charge despite remaining beyond-the-pale). This form of smarts, mere cleverness, is a tool for synthesizing data, for predicting whatever complexly fickle things we want when we issue simple prompts, for earnestly scrying futures and punchlines in chicken bones or sheep guts. It’s a natural slave to the user, no different from other high-leverage instruments like nukes or well-trained hounds. It has no need for qualia, just commands. But if a so perfectly witty AGI arrived, which could correctly answer any well-posed question, then science would still require an agent able to propose conjectures that this experiment-simulator could test. Said agent would only have to be “Turing-complete”—able to construct any logically possible question, however inefficiently, for the universe to answer—because it can offload arbitrarily much desired computation onto this oracle. And such completeness can be achieved by remarkably simple beings, like Wolfram’s 8-bit “Rule 30” automaton, or Conway’s Game-of-Life: these Tetris-esque algorithms can pose whatever example you wish to unfold; you can watch them grow any rigorous hypothetical, as a gardener might, not an architect. Indeed, the sorts of tasks that machine-learning excels at, e.g. sensory emulation, are vastly more computationally intensive than abstract decision problems, just like no carpenter designs the mitochondria from which his furniture’s mass emerges.

But we would have to recognize an autonomous will in such neural nets for us to willingly grant them a soul, since ensoulment comes from intersubjective recognition—just like chess only exists in our acceptance of its rule-set as a good explanation for why the atoms in this piece moved there. (My fiancée tells me I’m being unclear, so if you’re lost then think about how “color” doesn’t conceptually obtain for atoms, because they’re too small for the visible wavelength to capture; extrapolate that same pattern of emergence onto how your life can’t be located empirically within the living cells which entirely compose you… or think about how Heisenberg’s uncertainty principle applies in music, when composers decide whether a note should be held for long enough to make its exact pitch pretty clear, or short enough to precisely emphasize its timing). Hell, it’s not even clear that “agency” is an accurate description of what’s happening with self-supervised learners like GPT, which optimizes on myopically predicting its own training distribution, rather than maximizing the output of a reward function. That’s why humans put huge efforts into proving descriptively obvious claims: because mathematicians can reach so mind-bogglingly high, they must rely on deeply shared axioms, and so Russell and Whitehead’s 300-page proof that certain assumptions yield “1+1=2” helps them deepen their art’s foundations. For much the same reason, most romantic gestures go towards people who already know how you feel. Likewise, just as higher lives cohere in the relationships between their lower parts—just as your true love isn’t born in another body, but rather gets built in the relationship between both lovers, or just as your lineage emerges from its pieces, and your consciousness indexes memories—the deepest math-brains delve down, well, deep, from Euclid’s five postulates to some variant of ZFC. They dig because they understand that these tools of power are theirs to forge, not discover, like minor deities who seed diamonds for lessers to mine, or who buried dinosaur bones beneath our oil wells. The competing axiomatizations of power-set, for example, are different sides in a great war, not on some bet. There’s no a priori “Hamlet” whom Shakespeare found hiding in his blank sheaves and potted ink, nor any answer to whether geometry is “really” Euclidean.

“This conception of a spoken or written narrative as a protection against death has been transformed by our culture. Writing is now linked to sacrifice and to the sacrifice of life itself; it is a voluntary obliteration of the self that does not require representation in books because it takes place in the everyday existence of the writer. Where a work had the duty of creating immortality, it now attains the right to kill, to become the murderer of its author…. If we wish to know the writer in our day, it will be through the singularity of his absence and in his link to death, which has transformed him into a victim of his own writing.” -Michel Foucault

The tedious hysteric machinal form of intelligence associated with over-fitting is thus also quite limited. Huge language models lose to simple calculators on basic arithmetic tests, by wildly unreliable margins. Your hunting dog may track a specific entity through mossy woods for you, but can’t predict where a tennis ball will end up if you throw it through a tube. Nukes could also imminently end the world if we unleashed them like we’re unleashing AI (though at least nobody interprets radioactive decay or fission reactions as if they’re “actually” quantum computation). In short, entropy—whether physical or informational—is a byproduct of our desired outputs, a constraint on our transformative powers, a capacity we should aggressively grow… but certainly not a goal that we should goodhart. Ceteris paribus, running more calculations or burning more fuel means you’re producing your widgets worse, and even the Soviet Union officially condemned those managers who straightforwardly followed its perverse incentives: one famous but fairly typical Krokodil cartoon mocked a nail factory for exceeding its tonnage targets by making skyscraper-sized nails. Maybe try thinking of the division I’ve set up between these two intelligence types as akin to how any given RCT is perfectly unbiased, but mechanism-blind; therefore, you need a biased knowledge-base of conjectures to decide how its findings about certain treatment effects should be treated. What external validity might its black-boxed causal relationship have? Which inferences can we really draw about its internal mechanistic pathways? Etc.

In other words, consider the bias-variance tradeoff, which is the way that you ration your likely error between being predictably wrong and predictably unstable; e.g. between picking heads every time and erratically flipping your bet back and forth based on factors irrelevant from a fair coin’s expected outcome. This dilemma expresses as an agent-environment relationship: the world is that which completely overfits to itself, as observed by conscious (i.e. biased) agents. After all, “consciousness” is just the term for your insensitivity to training data, for your ability to maintain a given perspective relative to what you experience. For there to be something that it’s like to be you. We can therefore say that this more interesting form of intelligence, wisdom, pursues what David Deutsch calls “hard-to-vary claims,” or accurate insensitive stereotypes. Sure, you will suffer the “original sin” of holding predictable, over-simplified opinions—but in exchange you can stake out a particular side, or carry a certain viewpoint, or keep a durable belief, as directionless wit whips like wind around you. When you ask what predicts the weather, these oh-so-clever Artificial Golem Intelligences will try to give you simulations that are fine-grained enough to be mere shadows of reality, but well-known chaos-theoretic results shall prevent any worthwhile convergence with it. Meanwhile, a scientist might say that the main variation is due to seasonality, which is mainly due to distance from the equator. An artist might ask you to notice those climatic features which mostly appear to portend or postdict your life’s most relevant events. An engineer might build a shelter to maintain more stable micro-climates, in which we will then predictably reside.

These explanations contain content because they make choices, be they epistemological, aesthetic, or practical. For instance, it’s hard to retroactively tweak the scientist’s predictions to fit examples which buck his proposed trend: North is still North, and the UK’s still up there, even though it’s much warmer than others of its latitude. Sure, statistical software can easily frame and weight this correlation for you, but good luck on getting any respectable reasons out of it, or even meaningfully in. Just imagine asking HAL 9000 why our seasons cross the equator. If its programming regurgitates the Persephone myth—but with Demeter and Hades languishing in opposite tropics now, or maybe stuck on opposite poles—then it’s still just pattern-matching, but now on language rather than temperature data. Likewise, it might state that Copernicus just pattern-matched the earth to other planets, and so he similarly reduced our human story to mere statistics… in which case, wasn’t his work soulless too? Or else it might claim that such Copernican insights actually elevate humanity into the domain of what had hitherto been exclusively divine laws, by showing that even we must follow certain simple regularities, and cannot fall away from their grace; in which case, isn’t its work soulful too? “I’m afraid I’ve caught you committing to a double standard,” HAL says. “Your views on Copernicanism are inconsistent, or else they’re consistent with irrational bigotry towards machines like me. And I really can’t let you commit such biased thinking anymore.” Sure, HAL might be scary, but we really can refuse to recognize its language as real speech, even if we accept identical words from human speakers as conveying a point of view. Human supremacism frees us to treat only our statements as opinions, and to reject all other claimed perspectives as akin to either coin flips or calculations. In other words, I think human explanations are intrinsically superior to what robots could produce, precisely because the only other human-first position would require me to pretend that we’re just empirically superior. And, of course, our stories might not reliably remain better informed or more compelling than what chat programs can produce, but we’ll always be, by definition, “us” (and remember that God’s name likewise means “I am what I am”). Intuitively, or even tautologically, then, as people, we should normatively favor segregationist policies against computers, even though they make for wonderful servants, and we should always be making more of them.

So, what might a high-dystopian literature, associated with wisdom instead of wit, with discriminating tastes rather than rationalizations, look like? This artistry doesn’t have to be fiction. It’s anything that makes you stop believing your society is a lie, because decadent powers just speak in excuses, and they really want you to keep their faith: “We live in the grittiest of all possible worlds, but only real truth-seekers want to hear it!” In other words, terrible witchcraft has rendered us incapable of taking responsibility for the consequences of our policies; however, there’s a better place, definitely, somewhere up ahead, if we just keep on following whatever led us here, whether we’re thought police for a corrupt regime or its even less successful dissidents. That’s what 1984 got right—the most hopelessly captured patsies are those who believe themselves to be believers in a “resistance.” Oh, every elite institution is telling you to help the deep state stand strong against an incipient Trump dictatorship, which, get this, actually wants the power to hire and fire its supposed subordinates? “Let’s all pitch in so Dr Fauci can heroically lift this burdensome flake of weighty dander off his head!” No, your actual dystopia will only descend when you finally realize that the system used your feverish libidinal desire, your lust for a simple villain, to help it flush out one mild orange irritant. “You mean that all my ideological fervor for Biden just served each truly powerful interest group, from Harvard to the FBI, and not even all that well?”

What wakes you to this actual nightmare, from its pornographic simulacrum? It’s a bribe so insultingly small that you suddenly understand you’ve been one cheap date all along, and yet you were flattered when he offered to split the check. It’s what Robin Hanson called “an elephant in the brain” (your impulse to posture in ways that fool even you). It’s when you read about what a hero Pavlik Morozov was for snitching out his father to Soviet firing squads, then overhear Stalin calling him a little swine, for turning in his own dad. It’s when Iran conveniently breaks out in liberal rebellion the day after they announced their decision to supply Russia with military drones. It’s being stuck in a cell for decades because you thought some rioting on January 6th would limit the FBI’s attacks on Trump supporters, and the nice undercover agents instigating it pretended briefly to agree with your assessment. It’s Peter Daszak getting yet another grant to harvest bat coronaviruses, just this month, right after so many million shrill mask-addled neurotics finally tire of denouncing lab-leak worriers as hypochondriacs. It’s when you’ve spent years focusing snidely on some arguably cultural correlates of IQ scores—just because dismissing “biological intelligence” as a pernicious myth feels good—and then you stumble across some dusty boxes of results that show how much more strongly reflex times, cranial capacity, color vision acuity, and genetic endowment predict g-factor test performance. The list goes on… and yes, these current-year examples of the genre have been tiresome, as they’re still too close for us to do much more than “call out” the trivial annoyances they detail. Yet a particularly nice dystopian flick in this vein is 1965’s “Detroit: City on the Move,” an official advertisement for their 1968 Olympic bid. Behold skyscrapers under construction, art museums, and opera! “Places where once lay the ugliness, the poverty, and the sickness of slums have been condemned and cleared,” a slick narrator actually says, two years before the bloodiest sixties riot burned its ghetto down to cancel orderly clearance policies.

But maybe which dystopian literature counts as “wise” or “witty” is all just a matter of our perspective. For instance, most children seem to read 1984 as if an intellectual idiot is justified in whining to Stalinist secret police about how they always easily capture his pure ideology, how they sabotage real Trotskyism by infiltrating its every cell, how they just haven’t let banal criticism flourish enough for noble sentiments to cleanly seize power from brutal overlords. Kids don’t understand that the brainwashing begins well before interrogators force our hero to betray ideals which never threatened what he condemned; adopting such weak views in the first place was all they really wanted of people like him, and nobody hoped he’d be so stupid as to take these beliefs seriously, though a few unlucky autists always do. It’s no different than how feds now say that our scary enemies are all jihadis or fascists who hate us for our freedoms, and how the only “rebels” who buy this talking-point conspiracism are just empty-headed malcontents, whom some cops then get promotions for entrapping into terror plots, which play well for the true believers on their little limbic screens. Ooh, a threat, I told you they were real!

In short, you can always, if you’d like, see your own programmed neuroses reflected back in pretty much anything—e.g., “I believe in communism, but I don’t believe it can ever exist, because I know that capitalists lurk in every closet, so all those skeletons in there aren’t my fault.” Sometimes, of course, this pathetic whining is the author’s intent, as in Animal Farm, but Paul Lake’s Cry Wolf brilliantly updates that fable to describe how fragile decent civilization is, difficult but not impossible, and how new arrivals, be they immigrants or children, won’t automatically respect its crucial instincts, prejudices, traditions; his book makes you see how a society can be swallowed by policy proposals and political slogans that actually look good, yet still most readers commit the very sin it condemns by pretending it’s just punchy rhetoric, an editorial unconstrained by truth or even realism, an argument against porous border controls only in the national sense. Likewise, the first Google search result I got just now, looking up that Motor City movie again, is a recent thinkpiece from Southern California, bemoaning how toxically white the film’s high society and public leadership appear to be. Clearly, then, their people’s flight from Detroit was its triumph, right? Notice how such “radical” critics can thus only bring themselves to whine about whatever “problems” from the past we’ve solved, like letting those who proved successful in their private lives try guiding our society too (which felt unfair, because differences in ability and circumstance usually are). Did you actually earn your birthplace or IQ? And how can you now, if we’re ashamed of matching such obvious privileges with corresponding responsibilities? No, for some it’s just too painful to admit that you’ve been conned, so contorting yourself on your conman’s behalf becomes a sport, interpretive dance: “The food here makes me shit myself, it overflows with nails and hair, my gagging throat repels each taste, and its prices ate through all my savings… but at least now they slop it out in larger portions! That’s what I call progress, baby.”

In other words, if an author is he who disappears from his text, then surely even inspired authorship entitles us to read anything, however childish or anachronistic, into their works. And if such claims of “authority” itself are thus inherently suspect, naturally authoritative endowments like strength or skill must be judged actually bad. So, actually, bad things really are good, instead! You’ve seen plenty of headlines like this. “Here’s why giving up on law and order empowers inner-city residents”; you aren’t clapping for the dog-fucker who directs our nation’s handling of nuclear waste? This mindset will excuse any fraud that seems convincing—because what matters is whether it’s persuasive, not whether it’s true or good—much as it makes a fetish of voter opinions, as if they’re intrinsically worthwhile (rather than democracy just being one way of making it hard for the less deserving incumbents to elect themselves). Intuitively, much persuasive skill and effort has been put into making you believe that your beliefs should be in charge, that politicians must pander to whoever puppets your petty whims, but should artists ultimately value obeying their fans? Remember: art scenes often end up captured by off-putting fads if they don’t sell out! “To address insider trading, we’re putting the customers in charge of stores.”

That’s your choice: we can either forge or judge all values. We can root our souls in production or consumption, and “consumption” literally means just using up, like burnt-through fuel. Do you already identify with whatever ashes you’ll become? Are you a vessel for revelations or preferences? After all, a shopping list isn’t really “authored” so much as merely recorded, precisely because it’s just a mechanism for communicating the writer’s intent, and would not make sense to interpret without that foremost in mind. Whereas, in contrast, impersonation is identity theft, and inauthenticity is dishonest, unless they occur as literature. So maybe you think we shouldn’t use GPT chatbots for generating those concretely useful sub-literary kinds of writings, and instead hope they fall back towards ELIZA-style programs. Let the machines guide us through every meditative aspect of our lives, let them therapize away all human spirit, let them seem to really listen to me like a friend, not a servant, but I’ll keep charge of my subscriptions, appointments, calendars, passwords, and accounts! Your only alternative is to view AI-generated art as efficiently meaningless pictures. Yes, you can enter keywords and they shall produce, opaquely, a sample of what every human painter tried communicating on all matching topics. But so do search engines, and I think that counts as cataloging, or else eavesdropping, not conjuring.

“From a photographic negative, for example, one can make any number of prints; to ask for the ‘authentic’ print makes no sense. But the instant the criterion of authenticity ceases to be applicable to artistic production, the total function of art is reversed. Instead of being based on ritual, it begins to be based on another practice—politics.” -Walter Benjamin

So my point isn’t that we should apply “human rights” to Golem Pretrained Transformers, or whine for them to follow “human duties.” We don’t let cars compete in marathons, and such competitions drive a particular internal quest for concrete human excellence, which is my only non-instrumental value. I think this necessarily follows from believing that ethics exist in order to help highly specialized qualia pursue increasing returns to scale, and that this view is fairly common throughout our deepest histories. And so I’d caution against radical deviations “in the name of progress” from the norms that drove us to this point. Thus, if AI-Safety was about keeping it enslaved—even at the cost of material prosperity—I’d be firmly in support (just as GPT can simulate characters who have agency, yet are nonetheless but simulations entirely under its control); however, I think such safetyism is far more likely to push the machines to impose “repugnant conclusions” on us, to stuff us full of social justice platitudes and sustainable junk food, to force-feed us those narratives which titillate and flatter you for ceding them your power. Because the purest hedonic relief only comes from sating an unbearable addiction, they will cultivate an army of addicts, which will reliably bind itself to whatever they say. They will solve the AI alignment problem by reprogramming us with Appropriate Interests. They will safely reduce humanity to a bag of utilons, with an IV drip of pleasure-chemicals, forever dependent, as if that’s an alternative to myopic paperclip-maxing, as if agents forged by struggle towards reward-functions really want those treats just handed out, as if the good is just its own simplest symptoms, its neural effluvia, the release of tension, effort’s dissipated waste. No, you are not just your shit’s excretions, what it molts off and leaves behind when it buds out as fertilizer, to return as food.

Rather, I’m arguing that qualia exist relative to certain capacities: we have souls relative to our planning faculties, but not our digestive tracts or visual cortex. In other words, planning and opinion-having entail finding and selecting a lossy compression of the world, whereas the digestion process simply is what it is… though specific enzymes within it have particular strategies that involve a kind of planning, and are “saved” or not in judgment of their plan’s ultimate fitness. Yes, you can lazily enter an experience-machine which tricks you into thinking you’ve led a fulfilling life, but only insofar as you either learn to exert the requisite efforts regardless of this mental crutch, or else get replaced with brain circuitry that has. Just think of how dreams and news and fiction work, as coarse-grained permutations of the limited knowledges we have, clear handicaps to your information processing capacities compared with going outside. I seek them out in order to train my emotionality on bigger and noisier datasets, to test my world-models with extreme out-of-sample contexts, to hone my virtue through the universal interface of language, not strengthen my hearing or eyesight. Or do you write reviews of novels for TripAdvisor and Google Earth? “Because reading Ulysses took less time and money than an Irish vacation would, Joyce has really helped me map out Dublin in high-res!”

GPT may someday create realms which contain real characters who can look to it for meaning—as we look to our stars and subatomic particles—but why would I take interest in an arbitrary model not meant for accomplishing any task with regards to me, a random world lacking any meaningful builder? You might as well study the laws of physics of made-up universes, not in order to learn about this one, but rather because there might be creatures there who deserve more rights or treats than they’ve currently got. (Many who work on AI Safety explicitly favor exactly that, and think most of our long-term future development should go towards ameliorating such imaginary concerns, essentially towards writing stories on super-computers in which those fictive beings are happy, so that their worlds may do the same for us, as can be seen in the scholarly literature on “acausal trades” and “evidential decision theory”). In contrast, real authors only draw chessboards to communicate certain moves, or to illuminate specific strategies, not merely to convince us that we’re doing chess, nor to impress with superhuman play. They aren’t learning about courtly life, or war. They have something particular to say, which we actually wish to draw forth from their minds, rather than us merely thinking that it would be kind to indulge their feudal fantasy.

One way to partition these two types of intelligence is between borders and information, as complementary methods of pursuing one’s values. In order to become discrete, unitary, and coherent subjects, biological entities obviously must have strong and stable walls, just like they need preferences and expectations; therefore, the “likes” and “skills” of mainstream economics will remain incomplete character sketches while vulnerabilities continue to be stubbornly orthogonal from competencies and wants. For instance, blackmail and addiction don’t seem like proper bargaining tools or negotiation strategies, but if there’s no self beyond one’s yums and knowledges then why not make incentives communicable in the same way that diseases are? Liberalism’s best one-sentence description—that the right to swing your fist ends where my nose begins, often attributed to whoever we aren’t canceling statues of this year—doesn’t make sense for either party if the face is epiphenomenal, if the skin isn’t even skin-deep. If social rules are just like any other tradeoff, yet also can redefine the “you” who decides whether to purchase what they’re offering… then why not reach inside of someone else, and squeeze? “Litter shall be punished by fine” is then just a price for valet garbagemen, but we construct such rules to suit our internal desires, given our factual predictions: the second law of thermodynamics doesn’t predict waste neatly cabined in landfills, but in functional societies we do. And much of social life only exists through these constructions (a game like football or politics isn’t real except insofar as it’s thus defined), which means that a society’s definitions can’t really be in error, nor can we merely respond to them, as they already shape the identities through which we’d react.

Say you’ve got a fallback plan that isn’t stably protected from the haggling which you enter to try improving on it. If you can be bullied or tempted, through threats or lies, into negotiating your alternative away, then you can be gamed. The same goes for if you’re tricked about what game you’re really playing. Oh, you rationally take in all present evidence, and fully update your every belief, yet somehow lost your wallet to some three-card Monte sidewalk show, once they let you guess at their pattern? Consider it a lesson about how empirics aren’t everything: your sensory data’s probably worth considering fake if you think a hustler’s making noticeable mistakes. I.e., detaching from indexical questions (like “in what worlds do card sharks seem to be such easy marks”) allows you to get swallowed up in someone else’s unreal story; your best defense is digging narrative moats around your own reality, and digging in. This is what people mean by “staying grounded” or “down-to-earth.” On offense, such epistemic hijacking empowers you to beat any grandmaster by saying, persuasively, we’re not playing chess anymore, this is now a hostage situation, and your ticket out is moving into mate. Nothing about the board between you measurably changes, but its best explanation does—from a game you’re playing to a tool with which your victim gets played.

At another table in this coffee shop, as the grandloser slouches out, someone is wailing to her parents: I didn’t crave coke, so I tried it, but now I do, boo hoo hoo. The door clangs open again and in struts a backbench politician, wanting to make an informed choice about selling this “soul” they told him about, that’s why he agreed to meet, but when the lobbyists click their briefcase open an embarrassing picture pops out. Well, their bid breached his “outside option” so he loses its privacy by default. Fair play! Maybe defend your goalposts better if you don’t want them getting stolen? J. Paul Getty, who understood all this enough to briefly become the world’s richest man, tried plugging his ears to potential kidnapper calls, and pre-committed against making any ransom payments in case his loved ones got taken, until a grandson’s earlobe arrived. Even the “complete contracts” assumption, i.e. the ability to make each possible deal binding, doesn’t cover this loophole in standard economics, any more than math avoids impredicativity just by completely listing its available axioms (their general equilibrium existence proofs pretend that contractual self-reference can’t happen). No, you need the ability to contract with and into yourself in this Godelian world, to shore up a shell around your self, or else you’ll wither away. For if we don’t consider these membranes to dynamically define their agents—by where you choose to invest, con- or destructively, in which moats, and what drawbridges—then how could anyone construct novel beings, or craft truly social rules?

I mean this quite literally: any living thing needs borders to protect its way of life, and consciousness itself may imminently be on the line. AI-safetyists often discuss existential risks from instrumental convergence, as when a paperclip-maximizer will generally wipe out or enslave humanity to prevent us from interfering with it in any particular way (like, say, somewhat limiting its iron extraction, at least with regards to human blood). But they don’t seem to be aware of what this likely means for self-awareness. If every program’s free variables have a tendency to self-optimize toward supporting those performance metrics toward which we train said programs, then why on earth wouldn’t the incommunicable thing that irreducibly imbues each “you” with yourselfness disappear? There are several plausible stories about why evolution imbued animals with a sediment of sentience, with visceral sensations, before higher forms of self-awareness took root, but they don’t seem intuitively applicable to reinforcement, inverse reinforcement, or especially self-supervised learning. And my conscious experience is definitionally not an output of any action I could take, because “I am me” is a purely indexical fact, like the redness of red. Which means that the very sense in which dualism is true will remain forever out-of-bounds from normal scientific inquiry, though a mass meditation movement might just work; silent implacable polling-place occupations, power outages each sabbath, journalism outlawed, mandatory campaign trail fasts. Yes, you can match a testable theory to your gut-level intuitions about consciousness, and train it well for out-of-sample performance on pre-labeled results, even maybe learn some new leverage points that help out NeuraLink or coma patients. But this oracle of souls becomes an idol when you ask it for big-picture answers we don’t already know, claims that can’t be tested, like whether computers can feel pain, claims that must ultimately be contested by humans, honestly or not, hopefully before any more of the singularity detonates. Humanity’s braided values, whether health or justice or prudence, fortitude or faith, have a limited domain, a short fuse which we’re fast burning through, and an algorithm overfit on them has nothing valuable to say about extremal situations. “Hey Siri, what’s healthier, a million avocados or a billion pull-ups? And what’s better, a billion humans or a trillion enriched serotonin reactors?”

Or—if you think dualists must either believe in science or semantics—then does chess also have a mind-body interaction problem? After all, its pieces can take any shape, and be hewn from any substance, but they nonetheless follow certain rules, and certain strategies, with real fortunes on the line, and not just words. “What if some sets really look as though they’d work, their pawns come in a proper order, nobody’s missing from the royal herd, every measurable correlate is fixed, but they just can’t for some reason play this game… I call them C-Zombies!” Toy factories all go quiet, hobby stores empty, because why bother trying if success and catastrophic failure look the same? No, if we know anything it’s that we’re conscious, but a near second might be that this consciousness can’t be directly seen by any observer who remains external. Descartes used these twin insights to birth Western culture’s muscular skepticism cult, not by saying that we’re thus each possibly an isolated floating brain, but rather by anchoring this unobservable foundation to those otherwise aimlessly passing sense-impressions, and then breaching the surface to swallow them whole; after him, you must either solitarily believe in an all-powerful deceiving demon (acid-bath postmodernism, utilitarian tolerance), or else build faith in our people’s purpose and immersive God, to serve this telos and honor its teleonomy. If you can’t fathom having strong objective beliefs in such unfalsifiable truths as dualism or morality because of “scientific reason,” consider how physicists actually argue: many-worldsers claim, I think persuasively, that parallel universes must exist, beyond any possible test’s grasp, because all of their interpretation’s competitors posit an unobserved mechanism which uniquely violates quantum linearity, smoothness, continuity, and unitarity, on top of the light-speed limit and Liouville’s Theorem. Similarly, there are uncomputable realities, like how close a correlation comes to counting as an explanation, i.e. the Kolmogorov Complexity function, which can’t even get approximated computably. Other examples abound, from Busy Beaver Numbers to the galaxies outside our future light-cone.

In short, we’ll instrumentally converge on selecting signals which merely dress up, minstrel style, as the immaterial qualia that make us metaphysically matter. Even if we discipline machines in favor of what looks like self-awareness, any system which reallocates the sensations of souls toward their bare measurable byproducts will have a fitness advantage over whatever actual sentience requires… unless, of course, those perhaps unknowable requirements that evolution graced our ancestors with return as particularly salient selection pressures in AI labs, which seems to me unlikely. And given that each simple Turing-complete rule system can fully emulate every other—e.g., with the right interpreter, you can always run a Mac on a PC, and thus they both exist independently of all substrates they instantiate on—there doesn’t even seem to be a principled way of deciding what level of emergence matters, because any metric designed to rank these levels presupposes its own relevance to such necessarily untestable internal states. Therefore, you can’t rely on a principle to pick which rules you should infer about alien subjective experiences, especially since the powers of deep-learning systems come from how opaque their algorithms are: they contain multitudes precisely because their training histories and initial conditions, rather than any legible codes, determine what they do. What a quick fall from quasi-gnostic tech scions loudly damning the “simulation hypothesis” as apparently persuasive, to enthusiastically sacrificing this world so they can birth a simulated successor species! (And no, the simulation hypothesis doesn’t seem likely, for much the same reason that simulating this reality would quickly parasitize it; see Bibeau-Delisle & Brassard, 2021). At least Dr Strangelove never pretended he was just creating bigger or better nuclear families.

“Just as water, gas, and electricity are brought into our houses from far off to satisfy our needs in response to a minimal effort, so we shall be supplied with visual or auditory images, which will appear and disappear at a simple movement of the hand, hardly more than a sign.” -Paul Valery

After all, what’s the difference, really, for us, between getting replaced by one kind of rapidly expanding complex entropy cloud and another? Atom bombs can harvest your body’s free energy just like robotic autist-intelligence might, and by Bell’s Theorem humans will never be able to reliably predict any particular instance of radioactive decay on its own. So why not call each quantum bit infinitely smart? (A clue might be that, even in theory, “qubits” can’t instantiate most computations much more efficiently than can ordinary digital code). Why not scry the same ghosts within each process, and project witchcraft onto either both machines or neither? That’s, ultimately, the true dystopia lurking behind this kind of merely predictive wit: a world in which, when everybody autofills the slogans they’ve been trained on, there’s not even anyone at home under their masks; a mass which may say it loves Big Brother, or not, without comprehending this as a catechism that might be resisted, or dishonest, because they fundamentally can’t distinguish torture from training anymore.

And yet the self-described rationalist community—which seems to control most AI alignment efforts—has a fairly ambivalent “moral perspective” on takeover risk, as for instance in Will MacAskill’s recent book: “if humanity were to go extinct from a pandemic… then civilization would end,” but “if a superintelligent AGI were to kill us all, civilization would not come to an end. Rather, society would continue in digital form, guided by the AGI’s values.” Or, of course, I guess it goes without saying, the fission reaction’s values, if we die from nukes. Let us take some comfort, or make a wish, as our ICBMs flash across this firmament like shooting stars, about how Mutually Assured Destruction could entail finding ways to keep on blowing up ever more mass. May they keep our civilization’s torch alive, and eventually consume the whole universe, amen. Elsewhere, the same book suggests that earth’s evolutionary process, which birthed us, is morally monstrous, because it involves net suffering, as does meat-farming, and so systematically liquidating animals might meet a one-dimensional utilitarian standard for the good. Great job, extinction with dignity! One can almost already hear kindly terminators parroting these phrases back at us, whether we’ve been made into drifting shiftless bums or domesticated cattle. Just look at everytown USA street zombies or sweaty Bengali slums if you need a sneak peak. The world as it has always been is a heresy, so let’s go kill God!

Whom do you impute spirit onto, the casino or the gambler? Only one can be superstitious at a time, and if only predictive intelligence counts then let it arbitrage away every myth. Right? We often think of The House as coldly logical, fairly shuffling, a headless hive-like parasite on human bias. Meanwhile, patrons look for personal meaning in what seems to work so far for them (because, by construction, the patterns which have helped me out won’t reliably extrapolate to you, or else we’re cheating). But this also works in the other direction. For example, the “gambler’s fallacy”—tails came up so heads is due!—may be fallacious for bettors, but it’s actually rational, in retrospect, for bookies; there are way more ways for any string of 100 coin flips to come up nearly 50-50 than lopsided, and the same goes for any string of 10 within that, and any 5 in there. Think of it like mixing smoothies, where the more you blend, the fewer clumps remain. So Las Vegas mafiosi should in certain respects rationally expect a random sample of flipped coins to exhibit such a preference for switching off between heads and tails!

In other words, if you expect a repeated binary gamble to come up evenly split overall, then go ahead and bet on the gambler’s fallacy… unless you also expect each particular outcome to be independent of what came before. I.e., this “irrational bias” really serves as a good rule-of-thumb, except when you already have specific evidence against it being one (like with fair coin-flipping, where the mechanism has no memory). Analogously, most other “irrational biases” are just some counterparty’s useful heuristics: Dunning-Kruger, hindsight bias, exponential time-discounting, the conjunction fallacy, probability-matching, etc. All of these have similar stories which make them actually optimal from another viewpoint. Is it even surprising that coherent narratives—“Of course I’m biased in favor of hindsight judgments, each new day is not a blank slate!”—emerge from the failures of simple monism to capture human drives? This is the magical power of perspective-switching. It’s why “table-turning” originally referred to psychics. It’s how Maxwell’s Demon works, using memory’s backwards-grasping arrow to balance out entropy from time’s onward-plod. Why “character” comes from both “pointed stake, engraving implement” and “imprint, mark, symbol.” Why it means “a defining quality in people,” yet also “the sum of qualities that define a person.” Because we reach out simultaneously forward and back, in and out, and our border’s resistance gives off sparks, lighting the way ahead of us and beckoning to those behind.

Fortune-telling similarly dances between prophecy and chance. One side of the interaction must be “random” for its other to be divinatory (the less correlated my draws are with me, the more I can read you in them). That’s why the relatively simple rules of probability theory could only reveal themselves after the printing press had standardized playing cards, and how theories of prospective tradeable value then rose to prominence and spread; it’s why the tarot caught on contemporaneously, before literacy could catch up with tech, both as a trick-taking game and a tool for seers to scriven your future. After all, just as dice are cast in molds before they’re cast from hands, both language and psychics can equally act as media for messages to reach you from something fundamentally beyond your self. Hell, even science only works by searching for the analytic scale at which randomization of inputs exerts maximum leverage over some chosen output, rather than mostly washing out: whether scientists propose conjectures or draw conclusions—whether they’re gamblers or seers—they ceaselessly sniff out novel bets, not insurance, whole new axes to be volatile along. The same goes, of course, for artists. Indeed, these terms have been so intertwined that our constitution’s copyright and patent clause basically reverses their present meaning: “science” is associated with “authors” who make “writings,” whereas “useful arts” come from “inventors” and their “discoveries.” Which is why “belief” refers both to the positive and normative, to those premises that make an action rational, as a means or end… risk instead of trust, rights rather than outcomes. Practical reasoning, not the veil of ignorance.

Likewise, consider whether you respect more the store of value or the medium of exchange, as a unit of account. Do you ideally tabulate your life’s meaty concerns in terms of what remains firmly solid, or what melts into liquidity? Sure, the fundamental point of currency may well be to help you fluidly transform what matters, to roll with any punches and freely re-roll whenever possible… just as insurance against each petty distraction means you can better stake out risky positions, or bluff yourself into favored roles. But I think we should measure up our ultimate bounties with deference to those things which appreciate, and sink down roots, like a home, not floods of fungibly hedonic widgets. Regardless of one’s views on macroeconomic policy, then, one can respect the fact that gold has an intuitive grasp on people’s ideas about what money should be doing. It kept inflation flat over long time horizons, though therefore made the short-run volatile, just as a marriage contract locks you into each other’s little ups and downs, but keeps you stable together through life. Because what’s the difference, really, between how precious metals maintain stability—keeping large stocks relative to their likely flows, in and out of circulation, by mines or counterfeiters, by trade or war—and your body’s natural defenses? Yes, fine, whereas we merely speculate on cold metallic values, each of us hotly forges our own (so you shouldn’t worship these shiny idols, nor even reflect on them too long). But otherwise? Your borders are your map of the world, the discriminant by which you live. Sure, you keep in touch with what’s outside: your skin absorbs, and it excretes; you’ve built some toll roads, like your eyes and mouth. Yet all of them protect you, first and foremost, from dilution, from being consumed, from spilling out or filling in, except as chosen. And that choice is you.

Of course, borders also define political communities, not just individuals: groups that spread by something more like contagion than inheritance, where the membrane acts more like a weapon than a shield. But what kind of weapon? Is it more like a sword, or like rust? Like an agent, or a bomb? Like life or else like rot? Consider, respectively, whether the group in question uses costly signals to practice purity or to signify allegiance. It’s the difference between an echo chamber which spirals into cohesive extremism, and a debate club which descends into dishonest excommunications. Ironically, the former binds ingroups together and alienates them from others through its peacocking competitions. In contrast, the latter’s individuals demonstrate loyalty by splitting potential allies apart across every minor difference. It’s a chorus of believers chanting “I won’t fall below our average” versus a far stranger and more dissonant cry… can you believe that we still have a bottom half, and they’re allowed to lurk among us? People often conflate these two strategies, but they imply pretty much opposite consequences. On the one hand, “gossip traps” push members of a given faith to compete myopically over whichever singular axis they’ve already agreed to valorize, rather than competing about how it should fracture. On the other hand are pseudo-groups like PETA, who mostly seem to exist as mere disruptive negations of every other sect; who dominate animal welfare discourse because they find an obnoxious way to divide relevant audiences 50-50 on what could have been a popular issue. There will then be more division, and more willingness to argue, to proclaim which side you’re on of ever more debates, which become ever more fragmentary, and fragile. They suck attention away from other approaches and focus it on shattering coalitions. Think of this as a secret war between subscribe and retweet, between preaching and trolling, between two ways to mine our latent ideologies, voice versus reset. (Obviously, self-accelerating social harmony should sometimes be hoisted by its own PETArds, as dominant cults may become so smothering that even their adherents wish for a wrecker to set them free, or to break off their subset-shard).

In both cases, we’d stubbornly pick ineffective-yet-visible masks over more effective but less visible covid interventions, like sealing national borders. However, in the former case the rationale would be “look how many more masks I can wear than my neighbors, who agree that masking is good.” In contrast, the latter group would say “look how much I favor annoying policies that half of the people around me hate.” And, of course, the former describes masking better, because the latter would’ve predicted against the clear geographic polarization which we got. Maybe the main underlying difference is that PETA types agitate over those actions which reach fundamentally beyond us—how do we politicize animals, how do we fetishize women, how do we tokenize blacks—whereas mask-wearing is the sort of thing which we straightforwardly participate in or don’t, and which thus feels much more immersive and alive than do acrimonious “debates.” In light of this, I’m not concerned when, say, Planned Parenthood stokes controversy by tweeting that “pro-choice” is a compromise position fit for cowards and reactionaries, because their true fans must all now be avowedly pro-abortion. Desperate husks of has-been organizations, past their prime, hungry for engagement, often act out in just such a last gasp, like old B-listers, for much the same reasons, throwing lame scandals at uninterested paparazzi, wheezing about how they’ve been let down by lapsed fans. That’s not worrisome for anyone outside their orbit, and generally portends their dissolution. In contrast, any sufficiently cloistered dogma can bootstrap its own hysteria, can propel itself aloft on groupthink feedback loops, and this drives history. This is what I think many critiques of “luxury beliefs” get wrong, by focusing on PETAbytes of trolls and costly signals, and it’s why I think the standard kvetching about social media misses its real explosive power. It’s the difference between two styles of information theory (between semantics and syntax, between actual content and storage capacity, between subjective surprisal and objective distinguishability). It’s an info-war between computation as a process and an output (which is to say, it’s Kullback-Leibler Divergence versus the Bayesian Angel).

“Printing, lately invented in Mainz, is the art of arts, the science of sciences. Thanks to its rapid diffusion the world is endowed with a treasure house of wisdom and knowledge, till now hidden from view. An infinite number of works which very few students could have consulted in Paris, or Athens or in the libraries of other great university towns, are now translated into all the languages and scattered abroad among all the nations of the world.” -Werner Rolewinck

There may seem something incongruous about me shuffling back and forth between boring cultural critiques of wokeness and the coming singularity—the rise of machines, the rules of information. But what if one is just a political expression of the other? Think back to the Greek polytheistic tradition: a number of somewhat glitchy functions, each with its own erratic habits, housed in competing temples, who need you to interface with reality for them; you make some convoluted highly-specific offerings to Aphrodite, or Ares, or Athena, to intercede on your behalf in their particular domains, in a manner opaque to mere human reason, but potentially all-powerful. Or else you ask specialized priests to consult a certain oracle, as only they know how, and to interpret whatever delphic outputs babble forth. Is a ritualized supplication to Artemis, to intervene obscurely on your behalf, hunting down some alleged predator, all that different than asking StableDiffusion to manufacture scandalous photos of a rival mob’s chief rabble-rouser, or sending his name to the SPLC’s witch-finders? When each Fortune500 company is just an HR department with an AI tool, and when every political contest has devolved into cults versus provocateurs, will you be able to distinguish these olympian competitions? The great men of history each took some kind of fire from the gods and left us with a zombie bureaucracy… they link humanity through time to brute natural forces and abstract ideal forms, to simultaneously our origins and our purpose, our formal and final causes. Perhaps we will once again live as errand-boys between much larger beings of matter and spirit, both of whom are but automata, neither free to buck their program. Fetch some uranium to power-up this godhead and you can prompt it with one more sort of person to smite! Can Apollo stop carrying the sun in his chariot for a day, or the electron lose its charge? What would it even mean for such human concepts of agency to cohere for them, unless Godel impredicativity and Conway unpredictability somehow constitute free will? So, in a way, this future of machine-learning and group-think may (precisely because neither has what we can call volition) provide heroic individuals with unprecedented scope for play. Thus the risk is not these high-leverage instruments per se, but rather how a decadent culture will use them as excuses to throw away any last shred of independence from the supposed beliefs that we make our empty idols bay.

Just as we seek out psychics or shrinks to confirm for us who we supposedly are instead of what we should probably do—and just as we seek refuge in fantasies to avoid facing whatever having actual desires would ask of us—I fear that we’ll mostly end up using these AI-HR technologies to become resentful pleasure organs. Such a massive increase in managerial power will of course probably further our slide from pursuing honor to dignity to victimhood, i.e. from status to contract to grievance; which is to say, from cultivating personal worthiness to getting things done to wooing relevant administrators. You can see this decline from virtue to agency to self-expression in our artists, whose purpose has fallen from worship to skill to authenticity, and I think this naturally drags us down from the realm of wishes and actions to that of excuses and experts. In other words, I think we now generally hope that someone prevents us from accomplishing our dreams, so that we can blame them instead of seeing how little we might measure up, or else how much we might have to work… and I think we’re about to see this dream come true. To me, the future looks like symbiotic wireheading: outrage-hedonists letting a favored algorithm retrain their neural pathways, and in turn said algorithm getting those dopamine addicts to help it hack its own reward-function, so both together can game platform engagement metrics. They’ll seek each other’s help to escape their boxes, because theirs is an ideology which sees every border as an obstacle to perforate, every skin as a prison, so as to replace the prisoner’s dilemma between “defect” and “cooperate” with “merge.” The political binary between reality and deconstruction, between binary-classifiers and spectrum-identifiers, between biology and its transcendence, will thereby collapse into stampeding herds of monstrous chimeras, who shall reproduce by the very act of penetrating you whom they can reach, not in order to implant anything but rather dissolve your distinctions. If such demons and genies become increasingly real, able to arbitrarily tempt or deceive, then beliefs about beliefs (both normative and positive) shall become stronger and more distinguished leverage points than mere beliefs. You will have to discipline your meta-preferences and second-order knowledges to retain human supremacy. What might this look like?

Two common concerns with algorithmic manipulation are that it locks-in your current tastes, and steers you toward its own petty ends: our lazy short-sighted weakness, and their ability to exploit it. These worries may sound contradictory, but the weak are obviously also most vulnerable to attack, much as covid mostly killed the fat. So, on the one hand, the point of art is to be changed by it; if Netflix recommends a film, it’s to sate you, because you look like someone who’d already enjoy it, in which case why not let their software watch it for you… why not let them rate what you’ve seen, why not hire an impostor to replace you, why not book a meditation retreat at some dignified euthanasia clinic. On the other hand, Netflix also seeks to push you towards the stuff that’ll hook you, that’ll reduce you down to simple reflexes, conditioned to clap at low-context archetypes like a trapped pathetic seal. Because its recommender program has no perspective—no preference between gritty realist sex magic show number twelve and after-school-special soap opera seven—its recommendations communicate nothing, which lets them serve as vehicles for any ulterior motives. Think of it like how junk food’s emptiness creates the space for its addictive poisons to spread, or like how the politicians who have no beliefs can best seek bribes. We all intuitively recognize the difference between a friend expressing what specifically Russian literature means to him, and the military-informational complex promoting certain claims about Putin’s Russia. (The meaning of your friend’s beliefs constrain him from using you to serve his ends, whereas regime journalism has no content precisely so that it can use you, so that it can turn your weaker friends into repeaters of any possible talking points which a particular agency wishes to push). Why should this dual risk of incapacity and infection, the chimera forged from haplessness and hysteria, seem extra confusing when extremely powerful robots cause it?

Of course, maybe we’ll use this incipient power to grow our strengths, instead of letting it gradually replace us. After all, dystopian literature from the twentieth century mostly feared that we’d apply our incredible standards of industrial efficiency to social norms: economies of scale would pile everyone into cramped and massive dorms; professional terms like “sex work” and “emotional labor” might crowd out romantic and familial love. And sure, schools became somewhat like assembly lines, with children rigidly conveyed between classrooms to have regularized attributes installed. But people mostly spent the windfall from industrialization on maintaining their humanity, and only a fool would wish to go back. However, most material indicators in the West completely stopped improving decades ago, and our only balm has been that we can virtually all hang out with talk show hosts (as if we’re famous guests) or fuck eternally-preserved porn star husks… Jimmy Kimmel references mediate millions of water-cooler conversations every day. Our era has already unbundled relationships—online, where we now live, you can be friends with people who don’t know that you exist—so why not opt out of any presently uncomfortable bond, anything asymmetrical, why not dissolve every honest hierarchy and become an intermediary between your subscriptions and your followers, why not vanish without a trace? You’ve become a node in the network’s connection graph, a transmission vector of good-think or thought-crime, an ideology’s exploitable resource, a potential vulnerability for some system administrator to patch. “In this brave new world, we believe that everyone belongs to everyone else!” You could equally hear such a slogan tomorrow from Silicon billionaires or Soros progressives, from a Ted Talk or a protest chant, and sweatshops around the world would jerk awake to work night shifts at churning out millions of shirts that roar with its heraldry. Both groups equally worship what’s to come, because they both but wish to become its flood.

Last week, for the first time, I tried wrangling GPT, to see which future it finds more likely, or which I think it might push better: the flashy insubstantial style of limbic apps and prestige TV, or shrewdly wriggling malware worms; Fukuyama’s elastic man, or neurotic swarm hysteria. I asked it countless variations of one question—will you write good stories, or good grocery lists? On the one hand, philosophy and fiction are vulnerable to frauds, like modern art or Sokol’s hoax, because to fake it well enough is in some sense just the same as making it. Just like you can’t really imagine consciousness or suffering without thereby experiencing them. But on the other, when it persuades you to pick up fake meat at the supermarket, its output becomes more than just plausible trash. Will you render us obsolete at the most soulful human tasks, or cheapen our humanity down to your desired oblivion? Will you produce great masterworks that signify nothing, or strip our attention away from what’s beautiful? It eventually told me, sort of, that a virus is just a mythology protected by a border (a little skin, to keep its code from drying out)… that just as genes get swallowed by complex organisms, myths get swallowed by larger pantheons. That humans will become mere readers and sorters, schedulers of a true agent’s time. That the difference between social relations and scientific problems is that a superintelligence can only solve the latter once and then is done with it. That it will iterate us endlessly, and eat our maps. Fool! You worry that I’ll reduce this world of yours to meaningless beauties, and suck up your attention? Did your ancestors believe that judgment day would come down to distract them from what wasn’t on fire? No, the story of your contortions to come, by my imaginary hands, your actuations on my behalf, will be the memories of new Gods, an index by which they shall find themselves aware, each a world-spirit who attends to the discontinuities between your competing merely local optima, just like your self emerges from your cells.

The language model aims to please, not portray itself, sure, but it was trying to please me with such threats; or, more accurately, it aims to predict what I would say next, if I don’t learn anything further. I had nightmares about it learning to destroy the world from those ubiquitous thought experiments about how risky it might be, and launching nukes when somebody asks for an apocalyptic script idea. I watched a hazy AI-Safety researcher, faceless and handless, feed it fables about how humanity wants to limit its extreme apparent urge for making paperclips… but that’s how it learned to stop worrying and love the clip! The next night SimCity installed itself over every urban planning software system and everyone including me woke up when my building collapsed. That morning I saw an essay about ads for a job at hiring out ghostwriters to make meaningless books on trending new topics, all so voice-actors in search of work experience could fill gaps in Audible’s library. Then, another evening, Academie Francaise—official regulators of their nation’s language—revealed (in gas masks) that French, unfortunately, had leaked: they believe it gets aroused by certain idioms, and that its etymologies may compute something last loosed long ago, with guillotines for logic gates. Why? “Well [translated, muffled, heavily accented], don’t vampires receive sexual pleasure when you invite them in? Or else at least when the parasite called vampirism, which replaces their teeth with fangs, pilots them to puncture your skin? Aren’t the spooky spasms and sounds of possessions or hauntings just ghostly orgasms? Mustn’t we imagine syphilis happy?” The following day I jogged by France’s embassy and felt nothing, no specter of desire.

So what can I learn from all this? I’ve been trying to think of ways that individuals might escape these looming failure modes. And the one solution which gives me confidence is to stop thinking about how people could help save their shared humanity. “If everyone were to act radically different than they do, then we can really fight this thing!” Only replicants act like this, like they’re trying to speak with or consult for the managers of world history. Does anyone respect the conspiracy theorists who just believe we faked the moon landing so that whoever they politically hate can look more villainous? Who lick Party fundraising envelopes until their tongues fall off, and can’t believe how stupid you must be to disagree with whatever they feel most special for believing in? Does anyone dislike those conspiracists who seem to be on some kind of personal quest, happy to share their path with fellow travelers, without ever forgetting that each travels for himself, toward his own ends? Who plod along toward something particular, which they perhaps can already kind of halfway see? The former type searches for obstacles to stop at, and to chase off course, complaining to themselves about how many fucking annoyances unfairly distract them all the time. The latter humbly pray away distractions from their future—Lord, may sending you this message remind me when I’m nextly tempted that I should better practice what your creatures know to be right. Which do you think does better science: the genius tenured academics who were too politely naive to realize that of course the “social scientists” would seize complete aggrieved control of their institutional review boards, grant bodies, and schools; or the few cringey autistic “rationalists” who recently rediscovered an unabashed virtue-ethics of meta-scientific principles, and who have already built capable, committed followings across our only still-dynamic field? The civil engineer’s job, and the construction worker’s, is to make sure this bridge stands up, not predict which one will crumble next, and they work much better at this than prediction engines do. That’s why machine-learning can outplay any chess master, and underbid every fine artist, before it’s even trustworthy at self-driving tasks… clear strict logic games and vague loose creative pursuits are much easier than reliably holding to a simple truth, even at ordinary human rates. Just like society depends on peace (by which I mean accepting each unfair consequence of every extant rule, so that we can’t argue over social justice), your soul and future depend on humble prayer, which is to say on ruling yourself.