Transcript of Attractors in the Space of Mind Architectures and the Super Cooperation Cluster, a 45min conversation between Aneesh Mulye and Romeo Stevens.
Machine transcript (whisper-large-v3 with speaker diarization). The words are unedited; the speaker turns were corrected by hand. Timestamps mark the start of each turn.
Aneesh [0:00]: To begin with, I wanted to get a clearer idea of the context of both of these.
Romeo [0:05]: Okay, so specifically the complexity prior and the homeostatic prior?
Aneesh [0:11]: No, the complexity prior and the supercooperation cluster. So it turns out that you said humanity is grappling with the Darwin meme.
Romeo [0:19]: Yes.
Aneesh [0:19]: And these seem to be the only two things that even remotely, I don’t know, but the vibe by God was that seem to have the heft to deal with it?
Romeo [0:30]: Okay, yeah, so the way I would put it is that I see most people as doing one of two things. They’re either sort of like reaching up from the bottom, like so it’s like a, if you start off with a materialist reductionist sort of stance, then you’re kind of trying to recreate all of our meaning structures, you’re trying to like bridge that is-ought gap, it’s like okay we have all this stuff that is going on, and then we’re trying to start from the is. Yeah, they start from the is, they try to get to the ought. When people start start from the ought, sort of ex nihilo, right? They come up with a lot of wacky stuff that’s just, you know, I mean, there’s two things it’s predicated on. One is just whatever humans have a tendency to believe. The other is like what humans in the past have believed. Right, like whatever people have come up with that’s survived over centuries, right? Various religions and other moral frameworks that someone, people read it and they’re like, oh, that’s pretty clever, that seems consistent, or it seems like I’m able to apply that in a sort of fuzzy, heuristic way, right? utilitarianism, deontology, virtue ethics, whatever it is, stoicism, you know, particular schools of thought. It’s like this is a basic framework for acting.
Aneesh [1:32]: They pick something that seems, given their life experience and intuitions, to be workable, pragmatic,
Romeo [1:39]: yeah.
Aneesh [1:40]: Or even resonant.
Romeo [1:42]: Yeah, and then people get frustrated with the sort of that these things are under-specified or not ultimately grounded, right? You have the Baron Munchausen trilemma, right? Either infinite regress or unjustified axioms or a circular justification. So I’m attempting to do something like…
Aneesh [2:03]: The Munchausen Trilemn of Grounding, an ought that is purely an ought in a sense.
Romeo [2:09]: Well, so it’s informed by… I am trying to like do down from the ought instead of up from the is, is, but it’s informed by sort of a descriptivist agenda of like, first, like, see what is in terms of, oh, humans have these behaviors around meaning structures and some of them seem to operate well. Like people feel like their actions have meaning and they feel like they are oriented towards like useful levers in their world. And other people don’t. They’re handed one, but they find it insufficient, and then they abandon it, and then they cast around for replacements. They don’t have good meta-heuristics for, like, how do I select an ethical system? That’s why often meta-ethics is the term that gets thrown around.
Aneesh [2:56]: Often people have issues with the concept of given their ought sense. Often they have a problem with the concept of owning the fact that there is, in fact, a selection of ethical systems that they’re doing. because if you do that you lose the internalistic aspect of things meaning if
Romeo [3:14]: oh that’s true
Aneesh [3:15]: yeah so they can’t admit they have to frame things in terms of well you see i thought that was correct but it turns out that actually as opposed to just no i’m doing a search procedure right
Romeo [3:28]: yeah and it’s probably the uh for for secular people the ancestral environment and the darwin meme sort of serve that purpose but very poorly right and so they wind up with nihilism but yeah I would agree it’s
Aneesh [3:40]: something more pernicious than nihilism actually it’s not that things don’t mean anything it’s more that everything you know and love its actual meaning is in fact horror not anything good
Romeo [3:54]: well it’s just it’s all it all reduces to impersonal forces yeah which is not commit doesn’t resonate with the internal experience and so ultimately it It feels like it’s an unsatisfying explanation. And obviously we have, so we have Mars levels of distraction, which tell you why this isn’t a satisfying explanation, right? So a person comes to you and asks for an explanation on the intentional level of how should we orient towards the world? And you give them an algorithmic or a physics based response of like, well, you know, it’s all just mechanistic systems. It doesn’t, it’s not, you didn’t actually answer them on the level of intention.
Aneesh [4:27]: Right. I mean, I would go one step further, as I said, like this more than just like, it is is a sort of of all the possible overlays right that you could give on your sensory experience the darwinian overlay is a particularly cynical one because as it says it confuses intention with the causal mechanism that led to the beings having intention in that particular way but humans are not good at maintaining this distinction and so they end up thinking well bob is doing this to maximize his fitness and so it makes a wide variety of human experience just straight up unavailable. Like I am loved is cognitively impossible to believe if you think it’s just other people pretending to love you because that will maximize their fitness even if they are actually just genuinely give a shit. Right,
Romeo [5:18]: they’re having an internal experience that is totally in line with a more intuitive sense of like oh yeah there’s a felt sense to caring about people and being cared for that feels lost.
Aneesh [5:29]: Right and if if all the all the time you ascribe the meaning to that of this machiavellian person is you know doing this was like okay this is one it’s not true two you’re making yourself miserable and three this is an extremely difficult trap to get out of because you can always point to well but the causal process like it’s it’s more like a salience trap than a than a actual you know content trap
Romeo [5:59]: Yeah, and I think this is part of why the Azathoth meme or just sort of reifying anthropomorphizing impersonal forces is an effective or not ineffective, but it’s at least a partial antidote, right? So if I posit the existence of Azathoth, I can sort of gesture at, well, yeah, Azathoth is Machiavellian, but my goals are not necessarily the same as Azathoth, even though I’m running on hardware that was optimized by Azathoth. Azathoth being the personification of evolution.
Aneesh [6:25]: Right. I mean to be entirely honest choosing Lovecraft was not a neutral decision for this Pantheon because the reason he was picked is because of the sense of horror people felt. And I don’t think it’s a good idea to pick Azathoth because it fundamentally just fixes in place the emotional tone of horror as the dominant as opposed to alien-ness as the dominant. because alienness brings with it the possibility of wonder. Horror does not. It fixes the tone in place.
Romeo [6:57]: That’s a good point. Interesting.
Aneesh [6:58]: So to summarize, just so that I’ve got it correctly. People start with the is and end up in all kinds of wacky places because the space of the is is vast. People start with an ought and find it unsatisfying either because conditions have changed tremendously since most of these oughts developed or because the Darwin meme renders a lot of these oughts not workable. there’s eternalism, the problem of not being able to just acknowledge directly that yes, you’re looking for an ethical system because your current one is unsatisfactory. But I think I’m missing some core component. What would you say is the core piece of the problem with the starting with ought like you have with these?
Romeo [7:36]: So I view a lot of descriptively, if we’re just going out looking at what humans are doing when they do sort of moralizing, what I see is an attempt to systematize the moral intuitions, right? In the hopes that, so what do we accomplish when we systematized, well, you render a sort of predictive and extensible and modular. It’s like, okay, I’m going to be able to figure out using some sort of rule-based simple rule-based system, how to apply this system and also to apply it to novel situations and maybe create some like coordination points because like if it’s systematized, we can like more easily communicate it, talk about it, all coordinated about around it, like, you know, different interpretations of law, right, are sort of like philosophical arguments rendered into these Byzantine
Aneesh [8:18]: Let’s pretend that the bureaucracy is sufficient in such a way that it already encodes all our shared understandings. So all fights over so that all our fights about deep theology translate into the minutiae of procedural rules. No way that could possibly lead to problems.
Romeo [8:37]: Yeah. So. So, OK, so we have this agenda of we want to be able to systematize our moral intuitions. But like, where do you start is the question. Right. Right. So one of the core ideas, well, let’s see, this is a different, there’s a couple different slices on this. One is the navigational schema. One is the super cooperation cluster itself. And the other is maybe like the homeostatic and complexity priors. And they all relate ultimately.
Aneesh [9:08]: Okay. So to answer, so this, to set the context is the odd spry from a top-down view doesn’t work for various reasons, right? The is-is start from a bottom-up view also runs into very weird problems. What are the problems that is-people run into? And why is Darwin relevant? That seems to be the crux here, right? Why does that make both of these approaches unworkable, whereas they seemed kind of sort of workable at a small scale before?
Romeo [9:35]: They’re only workable as long as you don’t examine their foundations.
Aneesh [9:39]: Okay. The is or the ought or both?
Romeo [9:42]: Both. So traditionally, So this is actually a good additional context. Traditionally, the is-ought problem is seen as a bridging problem. Okay, you have the is, and you have the ought, and you need to bridge between them. This is just, you know, if you examine experience, this is obviously false. You have direct access to neither the is nor the ought, right? So you have indirect realism. You don’t actually have direct access to how things are. You’ve never seen an electron, right? So what you have is heuristics that return is-like answers. Like, oh, like physical laws, consistency, uh, invariances, you know, et cetera, et cetera. And then you have other heuristics that return ought like answers. They return like, oh, the interest. Yeah. These are our moral intuitions. Is those this, this group’s moral intuitions, how the game theory for these people interact, um, how do we feel about laws, et cetera, et cetera. So you are always this heuristic bridge and one end sort of reaches towards is one end sort of reaches towards ought. and you are already solving the is-ought problem on a daily basis. When you get up to go eat an apple instead of an orange, you’re solving the is-ought problem using just fuzzy heuristics.
Aneesh [10:49]: There is an apple. Should there be an apple? Yes, it should be in my stomach. Okay, so then why is there a gap like this in the first place? I know this takes us slightly afield, but what necessitated the heuristic bridge?
Romeo [11:07]: Because there are things that are the case that you don’t have direct control over, and you have to decide how to act in relation to those.
Aneesh [11:16]: Yes, but why is there such a thirst for systematization in the sense that we do not find, or at least my suspicion is we don’t find these kind of same problems in genuinely small tribe human societies. There is no is-out gap.
Romeo [11:31]: I disagree. So the pack that evolution initially found was animism, right? Treat the physical world as if it’s intentional, as if the is’s are all oughts, right? And then you get things like naturalistic fallacy, et cetera, et cetera. But this is a fairly consistent, like it can be made consistent by just telling stories. The stories have enough degrees of freedom that you can sort of fit stories to various things. Well, the clouds rain because they want to, right? Because then you have extended stories. And whoever’s the best storyteller, you know, and what counts as best, right? Right, well, the version that best compresses the sensory data of people’s actual experience.
Aneesh [12:06]: Okay. And that’s not workable in modern times merely because of the fact that the complexity is not easily fittable into a narrative framework?
Romeo [12:16]: Yeah, and also the amount of… So the complexity on the is side, but then also complexity on the ought side, because there’s so many intentions in the world all clashing at the same time at all different levels of abstraction, and no one’s really keeping track. And the way humans work is, you know, there’s, we’re not forced to stick to a particular level of distraction when we engage in our sort of memetic warfare. Right. Mimetic warfare, it probably has some ripple.
Aneesh [12:41]: Mimetic, mimetic or both?
Romeo [12:43]: Both. But yeah. Yeah. So one, one question that we can ask is, so we know sort of where is comes from. Like here it all is right. There’s rocks. You can throw them at things. Other things happen as a result of what those rocks hit. So where do moral intuitions come from? Right. And so one answer for what moral intuitions, where they come from, is sort of just a basic homeostatic system can be said to have oughts that it’s instantiating on the environment. So a bacteria that can only survive between two temperatures and has a temperature sensor, and the evolution of that bacteria is constrained by, well, bacteria that go outside of this temperature gradient die. So any
Aneesh [13:23]: homeostatic system has an ought overlay that it can impose.
Romeo [13:26]: You can interpret it as sort of imposing an ought on its environment.
Aneesh [13:31]: Okay. So any homeostatic system can be thought of as imposing an ought both on its environment and presumably its response to those environmental…
Romeo [13:41]: Yeah. And so we do have a question. We have a research agenda in the sense of it’s not known how do more complex goals in in a nervous system of a simple bacteria or up to a mammal, how does it build up higher order goals out of the very low level goals? And we seem to be creatures that are able to, we sort of do a type of speculative extension where we’re able to just define our own intermediate goals. We’re like, well, what if I tried maximizing this thing for a while, this measure? I’ll invent my own measure, go off and try to improve it and see if does that meet my underlying needs well or no? And when we sort of shop for different value systems, when we see people choosing different strategies, we sort of set up these intermediate goals. It’s like you’re generating a new thermometer based out of this whole bank of thermometers, like temperature sensors, pressure sensors, various, and then your own internal sensors for how well your organism is handling things, how hungry you are, blah, blah, blah, your appetites. So you have all these sensors, and we seem to construct speculative sort of measures. What if I tried doing this thing that increases this and then see if that sort of meets my needs?
Aneesh [14:49]: Oh, so we can experiment. A bacteria cannot experiment, but we can just poke at stuff and say, what if I tried to just, I mean, this is unlikely there were diet fads in the Paleolithic, but what if some paleo is like, yeah, I’m going to go on a carnivore diet because I’m really, really rich. Let’s see what happens.
Romeo [15:10]: So, yeah, the homeostatic prior, you could see it as sort of an extending, you can see the sort of you’re attempting to try to see how you could live in like a higher variance under higher variance conditions, like a broader range of environments.
Aneesh [15:25]: So what you’re saying is that we are not constrained by, well, except that, it’s not that we are not constrained by, but that the constraints allow us a far greater degrees of freedom in what we…
Romeo [15:40]: Well the bacteria can’t figure out to try putting on a fur coat to be able to survive in a wider range of environments.
Aneesh [15:45]: Exactly, yeah.
Romeo [15:46]: But we can and then some of those creatures survive and now there’s selection pressure on expanding your homeostatic range. Right. So I think there’s something around the homeostatic and complexity prior there of like…
Aneesh [15:57]: Ah, I see.
Romeo [15:58]: Yeah. Okay. Although the complexity prior covers a lot more than that.
Aneesh [16:00]: So the idea is that if you can survive both in the Arctic and in the desert and in Saudi Arabia or something and one hopes also in the intermediate climates because it’d be really weird if you couldn’t then in some sense you are more robust as a race than you know some hominid that.
Romeo [16:20]: Right. So this is all still sort of reaching up from the is. This is all scaffolding.
Aneesh [16:30]: Right. Homeostatic is is very clear. Yeah, exactly.
Romeo [16:33]: Well, it’s sort of how nature answers the question of there being an ought, or why there would be an is-ought in the first place. So what I’m doing is, with the super-cooperation cluster idea, is…
Aneesh [16:48]: What’s the conflict here? here, like the conflict that you’re trying to solve, that the fact is that these heuristics no longer work or that the grounding process doesn’t work or like what exactly if you had to characterize it is the issue here, where the is people cannot build up to a consistent and proper ought and the ought people cannot ground out in a thorough is,
Romeo [17:09]: sure. Yeah, yeah, exactly.
Aneesh [17:11]: But why is that a problem?
Romeo [17:12]: So from is to ought there’s nihilism and from ought to is there’s like crazy religious nonsense that doesn’t… It’s not a problem.
Aneesh [17:20]: It ignores causal factors.
Romeo [17:20]: You can imagine a healthier religion. Like Buddhism probably comes the closest in terms of like it’s fairly compatible with science. You know, the Buddhist leaders, blah, blah, blah. Everyone’s sort of already familiar with that. But in practice, most religions seem to calcify around particular worldviews. And they don’t seem to be very good at inculcating sort of adaptive strategies to their flock, unfortunately. Yeah. So I can also, so then I’ll try to cut in from another angle and see if they converge somewhat.
Aneesh [17:46]: So what is the positive goal? A healthy religion, one might say.
Romeo [17:50]: That going both ways is fine. That people can go up and down the stack, no problem.
Aneesh [17:54]: And it’s fine.
Romeo [17:55]: Yeah. That people have some self-awareness of what they’re doing when they move up and down the stack. And that explanations of the world at these multiple levels are in concordance with one another instead of dissonant with one another.
Aneesh [18:07]: And also in concordance with, I hope, what leads to a happy and virtuous or healthy human life, right?
Romeo [18:12]: Well, that’s the odd.
Aneesh [18:15]: Okay, so what’s your is here and what’s your ought here?
Romeo [18:19]: Alright, so I’m getting to it. So the super-partners imposter is the ought. So let’s back up and try to explain the super-partners imposter.
Aneesh [18:29]: Otherwise you wouldn’t be doing anything. There is an is that you think ought to be different.
Romeo [18:34]: Yeah. So, okay. So in game theory, you have this concept of third-party punishment. Right. And so for people that don’t know, third-party punishment means that, okay, you have some people cooperating, you have cooperate, cooperate, sort of equilibria. You have defect, defect, equilibria. You have defectors sort of trying to free ride on a cooperation norm. You have tit for tat equilibria.
Aneesh [18:58]: Yeah,
Romeo [18:58]: yeah. And so third-party punishment means that there are people who are observing the interactions that are happening, and they are noticing when defectors defect, and then they’re telling everyone or getting everyone or punishing or doing something to cause people to defect against the defectors. They’re like, okay, it’s not enough to just, we all cooperate with each other. We also have to make sure we defect against the defectors so that defection itself is a losing strategy, and the defectors will be incentivized to switch over to a cooperate-cooperate strategy. And so there’s been a lot of research on what causes this, how do you get stable cooperate equilibria and what can tend to undermine them. Can you antidote those? And this already happens in society. Obviously this is happening all the time.
Aneesh [19:40]: There’s a beautiful example. I remember some Atlantic article or some other article which talked about how very often bureaucracy is going from corrupt to non-corrupt or non-corrupt to corrupt can sometimes happen in what seems like relatively quick phase shifts and I think this might explain it. Like if you have even this effective third-party punishment for corruption then I think once the probability of that goes above a certain threshold it just completely flips the calculus and so within a few years it’s like boom even hitherto corrupt people just become non-corrupt because now it’s in this this effect is not true.
Romeo [20:20]: And the phase shift has actually been super well characterized in, in, in simulation at least. So they found that if a small portion of the network is super cooperators, then the network can flip over from a defect effect equilibrium to a cooperate one.
Aneesh [20:33]: Now this brings, this seems to give me some intuition for what a super cooperator is, but you’ll tell me what it is.
Romeo [20:39]: Yeah. So the surprising thing is, is that it goes both ways. They discovered empirically in actual tests with people that there’s also super super defectors that will punish cooperators. And there’s been a bunch of, there’s been a bunch of followup research to try to figure out why this would be, this seems very counterintuitive.
Aneesh [20:57]: You’re saying that in simulation anyway, they have found that super defectors can in fact create super defect, defect, defect equilibrium and enforce it. Yeah. Yeah. Even if they have to punish cooperators at some small cost to themselves. Right. Okay. And do real people, do real human beings do this?
Romeo [21:17]: Yes. That was the surprising thing.
Aneesh [21:18]: In what context?
Romeo [21:20]: In the context of… so it’s public goods games. Iterated. It’s sort of like an iterated games in which people can choose to contribute some proportion of a chunk of money in each round. And if enough people contribute, there’s some sort of multiplier. And so if few enough people contribute, then it winds up net negative for each individual person because it gets distributed so widely that you don’t gain. but if enough people contribute then everyone actually gains more because you know you’ve all multiplied your money together and so defectors can just sit there and free ride and so they have to and so people try to you know coordinate to punish the defectors and then like i said this this surprising dynamic emerges so so the intuition with cooperation is that what we’re looking for is we’re looking for what are sort of the the high level parameters that make systems more meta-stable uh in terms of cooperation so they generate cooperative equilibria and they tend to incentivize the players in those systems to maintain those cooperative equilibria.
Aneesh [22:18]: Okay, and is there any incentive to players outside the system to also?
Romeo [22:23]: So yeah, so there’s a whole thing around coalitions and what the rules are about coalitions and how costly you make joining coalitions and how the players signal that they’re in a coalition and making those signals like unfakeable or costly. But that’s like a bunch of complicated game theory stuff. But the short answer is yes, there’s a lot of thinking about how do you incentivize people to want to join cooperative clusters rather than try to do something else, free ride or defection clusters or just attack and dissolve the cooperation cluster.
Aneesh [22:56]: That was the historical dilemma, right?
Romeo [22:58]: Well, so there’s a whole thing around the relative balance of offense and defense, right? So when defense is more powerful just due to the technology at the time, then obviously you’re going to skew towards capital formation and when offense is favored then you go towards banditry
Aneesh [23:14]: because historically Central Asia was the repository of horse nomads just randomly blowing people up.
Romeo [23:24]: So we combine this with the last thing I was talking about about that we’re sort of these extensionalist creatures or we sort of extrapolate we create these speculative things. We try shit out. So it’s extending It’s taking this idea of cooperation and extending it out beyond our current understanding of saying, okay, we have some current level of understanding about cooperation. And presumably, so I understand more about cooperation than my past self did, right? I’m able to cooperate with a broader range of people under a broader range of circumstances because I sort of understand the structure of cooperation and I understand how other people work. And so I can extend that into the future and say, okay, my future self is going to understand more about cooperation than I do. be better at coalition building, et cetera, et cetera. And we can also think about that in sort of a scale free way, right? So like you already are a giant colony of super cooperators, right? So like all of your cells figured out the jump from single cellular organisms to multicellular took like a billion years or something crazy like that. Right.
Aneesh [24:23]: Immune system also runs a totalitarian regime that keeps everything in check and murders those who don’t. So.
Romeo [24:30]: Right, right, right. Yeah. The immune system is not, not fucking around.
Aneesh [24:32]: Yeah, exactly. Exactly. There’s super cooperators and then there’s, you know, big daddy.
Romeo [24:38]: So it’s scale free. It’s operating both at lower levels, you being a colony of super cooperators, but also at higher levels, right? That’s, you know, you can see that civilizational, there’s certain civilizational parameters that might contribute to being better or worse at sort of generating these cooperative equilibria. And so we can extrapolate that into like our own future, but also just more broadly. Like you can imagine a turning into a, you know, super intelligent civilization that is able to do way more with this. I have way more understanding than we do of cooperation. And you can also just imagine like the space of all possible super intelligent civilizations that, and so you can extrapolate out and, you know, whatever larger means, right. Even, even, even the concept of larger is sort of like, well, that’s at my current level of development. I have this certain notion of what it means to be like larger and more powerful. Right. And maybe that’ll change because it turns out the universe is quite different than I thought, just like our understanding is very different from the past. But you can imagine that there’s some sort of largest or most powerful cluster, individual civilizations, what have you, Jupiter brains, that are able to cooperate with each other, able to coordinate with each other and the hypothesis goes that this largest cluster of cooperators should beat the largest available clusters of defectors simply by the nature of defection and cooperation itself so you know defectors can build coalitions but they’re always limited by trying to figure out what the best level of abstraction to defect on is right Right, so, you know, Cancer is not very good at coordinating with itself to not just kill the creature that it’s growing in. And so it screws itself over, right? Because then once the creature dies, it dies. You can imagine that if it could coordinate better, it would want to, so that it could survive longer and maximize Cancer or whatever.
Aneesh [26:33]: I mean, at that point, its interests just straight up line up with the creature’s, right?
Romeo [26:37]: Mm-hmm. So this also cuts back to the complexity prior, right? So the super cooperation cluster, it doesn’t want to just tile. It doesn’t just take whatever the best, presumably doesn’t take whatever the best configuration is and just tile the universe, whatever available resources with that. The whole point would be that there’s a diversity of minds exploring different parts of the space of possible minds.
Aneesh [26:59]: Cooperation is only necessary if you’re not in fact trying to convert the universe into rat brains in a sense.
Romeo [27:03]: Yeah, yeah, exactly. Exactly. So you’re cooperating with things, with other processes that you think are able to explore a different part of the sort of space. That might be the physical universe or it might be the space of conscious minds, right? Space of possible conscious experiences. And you’re like, well, okay, I might learn something that I would not have been able to figure out on my own by interfacing with these other processes doing other things. All right, that’s the intuition. Yeah, I can talk about sort of the Buddha nature.
Aneesh [27:40]: Okay.
Romeo [27:41]: And tractors. Yeah, tractors in mind space. Okay, so we had the thing before about being the bridge between is and ought, that you already are this thing. There’s the famous ship of Theseus, right? So if you replace each part of the ship, then are you still the same ship? Blah, blah, blah. It’s just board games. names. There’s also the ship of Neroth, which is another philosopher who named this… Otto? Yeah, Otto von Neroth. I don’t know if I’m pronouncing the name right. But he said, you’re a ship of Theseus at sea. So you’re replacing parts and this ship has to remain seaworthy the whole time.
Aneesh [28:14]: Right.
Romeo [28:14]: Right. And Quine called this the ship of Neroth. So when we transplant to this bridge between is and ought, then I’m saying it’s sort of like you’re the bridge of Theseus. So you’re replacing your heuristics for how how you figure out what is the case and what ought to be the case. But in the ship metaphor, the nice thing about that is this idea of navigating, of trying to figure out where you should be aiming your ship. And so this super cooperation cluster idea, to me, is sort of like a lighthouse. And what that means is that you don’t steer directly towards lighthouses. That’s not the point of light. That’s not how lighthouses work.
Aneesh [28:58]: You really go to that.
Romeo [29:01]: But you sail. You can use a lighthouse to navigate, and hopefully it gets you far enough that you can spot the next lighthouse off in the distance. So the supercarburetion cluster, the idea is that that’s only the idea that I can conceive of from where I’m standing now. So you can imagine that the multicellular organism is not able to imagine all of the things that humans are going to be able to accomplish. once you have these giant colonies of trillions of cells being able to coordinate all these fantastic ways and generate something like a mammalian human brain, it can learn things that, you know, the bacteria is completely beyond it. So similarly, the idea here is we don’t need the absolute best version of this idea. We just need to intuit that there is a coherent navigational target out there and see if it gets us far enough to, you know, think of an even better version of of the idea. So it’s a kind of bootstrapping process.
Aneesh [30:00]: And the constraint is that the bootstrapping process has to, I mean, it’s a bootstrapping process all the way to the end, right? Until…
Romeo [30:09]: Yeah. So actually that brings up a great point. So all the way to the end, like, what does it mean? Is there a final target? And maybe there is, but from where we are now, when we talk about values, sometimes people get into this thing of they’re trying to figure out what are our real values right and it’s like they’re trying to figure out where’s the specific location we’re trying to get to yeah and so to me the the idea of what’s called buddha nature in in buddhism it’s kind of like a vector it’s not a specific place you’re trying to get to it’s a direction that always points out of where you are where you currently are and points you to like the they’re being more like good stuff out there so if you’re in the hell realm it just sort of points out of the hell realm right and if you’re in the human realm it just like points you like onwards so so a vector that so let’s say that we take as our principle you know freedom or safety or something all right or like the balance between freedom and safety you know whatever meta principle people come up with there’s some legal legal theory that still sort of grounds out in our current understanding and that’s the equivalent of choosing a vector that like we sort of know winds up in a sort of like a whirlpool like a black hole like an attractor in the space of possible values. And so I’m saying what we can do to specify the Buddha nature vector is we just take the vector that never does that. Within our horizon, as far as we can see, it just threads through all of the attractors. Maybe it gains energy from some of them, right? Maybe there’s good things to be taken from the various tractors, but it just never gets caught in any of them. And so…
Aneesh [31:40]: Oh, so the heuristic for the navigation schema is that it it shouldn’t take your ship into a fucking whirlpool.
Romeo [31:45]: Yes, yes. And so you remember the super defectors, like the super defectors, they’re always trying to figure out, okay, yeah, but when are we going to really defect? Like, you know, it’s all for the sake of like, you know, you play a board game or something and you cooperate with other players for a while, but then ultimately at some point you have to defect. And then one of the ideas of the super cooperation cluster is that you have knighting uncertainty. You have indexical uncertainty about like where you are in the journey and like what’s even happening. Like what’s even the real, you know, the is right. Like what even is the world I’m navigating? I don’t know. So therefore there is no, there never is going to be high enough certainty that you’re gonna be like, oh, this is the level I should defect at. I should just maximize freedom or whatever. Right. That’s like sort of choosing a final answer. And so this is, this is avoiding that and saying, you just never defect. Like all the way down and up the stack as, as you head, as you head up towards more complexity and more cooperation, you just don’t defect, you just keep assuming that there might be more uncertainty
Aneesh [32:41]: which can be demonstrated to be true about what level you’re working at right
Romeo [32:45]: well it’s like it’s like maximizing is like turning yourself into cancer for that particular thing it’s like i’m going to be paperclip cancer right i’m just going to decide that paperclips are the thing and now i’m going to tell the universe of paperclips and so again we can extrapolate to the past right so if you were to take whatever your values were 15 15 years ago and put yourself into a holodeck that maximizes those values.
Aneesh [33:08]: That sounds…
Romeo [33:08]: It’s like, okay, sure. Maybe the creature that emerges from that is like, yeah, I’m pretty happy. This seems good. But from your current perspective, you can see that that’s like a stunted creature. It’s like not actually… All the dimensions of value that you’ve discovered since then are just not served at all. So similarly, we can assume that as we sort of head towards this and as we increase in complexity that we’re going to discover entire new avenues of value to explore. So, you know, a superintelligence that preserves option value seems good. So another way of saying that is, you know, the superintelligence has the Buddha nature.
Aneesh [33:47]: So I mean this without minimizing your achievement, but this is by far the most rigorous and well-grounded derivation of God and heaven that I have ever come across?
Romeo [33:59]: Yeah, thank you.
Aneesh [34:00]: Like, yeah, this is not one of those, you know, disses where, haha, you just rediscovered heaven. There’s like, actually, no, this is well done. And you can imagine that there is a cluster out there somewhere. You can make it even more real. Instead of the super cooperation cluster being just a theoretical construct, you can imagine, no, they’re out there somewhere.
Romeo [34:19]: Yeah, so what I didn’t mention is time and space right so out there some when somewhere oh it’s in the it’s off in the future no no those are those are also concepts right so like if you’re familiar with a causal trade as soon as you become aware that this largest cluster possibly exists somewhere some when um somehow you know maybe not even in this universe it’s just the obvious thing to do to like yeah the thing that i do is i attempt to coordinate with the super corporation cluster because that’s… I don’t know, that’s the thing.
Aneesh [34:52]: Presumably they have some heuristics about coalition building. And it would be nice if when we encountered them, we met those heuristics.
Romeo [35:02]: Right. Yeah. Yeah, the litmus test for when you have first contact is, did you already figure out the first contact protocols?
Aneesh [35:14]: I see. So this is an interesting eschatology. Like we don’t know how long we have until we meet God, but what we do until that time determines what happens when we do.
Romeo [35:28]: Well, or like you already have to have the key by the time you reach the gates. It’s also in the super-corporation cluster’s best interest to aggressively go after and root out the worst defection clusters. So we should expect that if there are hells that, well, what does the super-corporation cluster do with all of its time and all of its infinite computation, assuming that it’s actually competently operating? Well, probably Helldive, right? Like figure out, okay, I need to develop some sort of compact payload, like, you know, von Neumann probe that like I can fire into a hell and it sort of like unfolds itself in such a way to like optimally antidote that hell realm. Hell realm being sort of a, one of the whirlpools out there for like configurations of mind space that’s just like terrible for the minds that are in it. And there’s nothing to say to preclude that that’s not what this simulation is, if this is a simulation.
Aneesh [36:34]: That’s not what this simulation is meaning?
Romeo [36:36]: This could be a simulation intended to do a search on how do you antidote a certain class of hell generators, like unfriendly AIs, or just other types of hell realms in general. It could be that this is a simulation to sort of build.
Aneesh [36:54]: Figure that out.
Romeo [36:55]: Yeah.
Aneesh [36:56]: If you could write this up poetically, this would be one of those little probes that you could throw in.
Romeo [37:02]: Huh, that’s an interesting way to wrap it on itself. I like that.
Aneesh [37:05]: No, this would be one of those just literal, you know.
Romeo [37:09]: Oh yeah, I do like the idea of Moloch as sort of the inverse of Buddha nature. It’s the thing, it’s the vector that points into attractors and each step along the attractor it tells you, oh yeah, going even farther into the attractor would be better.
Aneesh [37:23]: Right. The whole Carthaginian nonsense. Yep. Yeah. I mean, though we do not look kindly upon the Romans or the Spanish, right, for their cruelties, you have to say they did in fact get rid of literal human sacrificing pits.
Romeo [37:36]: That’s true. Yeah. Well, what have the Romans done for us lately? Well, they got rid of Carthage. Carthage was really fucking bad. Yeah. Yeah. So I mean, you asked what the problem to solve is. One of them is is the problem of a process that sort of generates externalities onto minds, like needlessly. It’s like, let’s just generate, you know, 100 billion copies of this mind that its architecture is kind of shitty. That’s a form of tiling. And tiling is like inherently bad because it’s, of course it leads to more zero-sum competition than non-tiling.
Aneesh [38:14]: Right.
Romeo [38:14]: By definition. Right,
Aneesh [38:16]: right, right, right, right.
Romeo [38:17]: It’s processes that want the same thing.
Aneesh [38:19]: Literally by definition. This might even be a prior, the diversity prior.
Romeo [38:23]: So I include this as part of the complexity prior. Okay. So the complexity, you have to have non-similar, otherwise it’s just symmetry. Right.
Aneesh [38:32]: Like if there were a species that could only live on planets that were exactly like Mercury, then they don’t fight with humans.
Romeo [38:37]: Yeah. Yeah. The bad gods, they don’t care about tiling, right? They’re perfectly happy to generate, oh, a hundred million followers. Great. That’s much better than 10 million followers, But they’re not even paying attention to the fact that they just created an extra 90 million that are all competing for the same resource.
Aneesh [38:55]: So even in some sense agnosticism towards the complexity prior or the diversity part of the complexity prior is liable to potentially lead you astray at least a little bit, if not entirely. idea.
Romeo [39:09]: Yeah, I mean, so I think that people’s metaphysics sort of has a slot for the sort of, there’s a god slot, right, in the ontology, and something’s going to live there. And if something has to live there, then at the very least you want to, if you have to be in an attractor, you at least want it to be the attractor that says attractors are bad and we should generate anti-attractors.
Aneesh [39:33]: Right, because presumably in order for your navigational schema to stay consistent resistant in the face of new information it has to be an attractor
Romeo [39:42]: yeah so so i mean we have to coalesce around some ideas right
Aneesh [39:45]: right
Romeo [39:46]: like these these ideas we’re talking about right now at least temporarily right but they’re non-pernicious right they don’t intentionally try to trap you in them yeah they it’s like good ideas point like what i mean by anti-attractor is a good idea points beyond itself right
Aneesh [39:59]: they do not create the incentives which make it the only option let’s put it that way yeah they maintain that openness whereas malokian competition leads to just more Molokian competition. It’s only when you get some pattern break, as in, oh, somebody discovers a new energy source or resource source or something, that you can for a while jump out of it.
Romeo [40:21]: Which is good in a certain sense, right? Incentivizing the discovery of new resources. But you don’t want it to be a blood tournament. No, no, no. Where all the, you know, what we’re going to do in the search process, but it’s going to churn through 100 billion minds and only one of them gets to win.
Aneesh [40:36]: Yeah, that sounds not good.
Romeo [40:38]: Yeah.
Aneesh [40:43]: I mean, you could if you really wanted to make the argument, say that this is the simulation where they’re trying to see how many Buddhas, like what are the, you know, conditions that produce Buddhahood. And that’s why humans are so mortal, like it’s a little blood tournament, and they don’t want to waste resources on simulating minds beyond the point that they’re They’re like, yeah, no, if you don’t get it within 30 to 50 years, out with you.
Romeo [41:08]: I mean, that’s pretty explicit within some Buddhist cosmology, right? That this realm is much more optimal for awakening than the long-lived god realms because just long-lived pleasure doesn’t create any incentive towards awakening.
Aneesh [41:20]: Yep. And it’s better than the hell realms because you’re not just distracted all the time. So how do you tie this back to the Darwin meme and the crisis of meaning that that has caused for both the is and the ought people? How does that solve that problem?
Romeo [41:33]: So again, I sort of see those memes as sort of building up from the is. And I see this as sort of a skyhook of down from the ought. So the idea is that a avoidant navigational schema doesn’t actually constrain you all that much. It doesn’t really tell you what’s valuable to go after. Whereas a positive navigation target… So if it’s a negative navigation target, all I know is stay away from these list of things. things. This is like known to be highly inefficient and also just sort of doesn’t work and is also sort of just arbitrary. So you encounter one of these things and you just flee in a random direction. Whereas if you have a positive navigation target, you can start charting around obstacles. You like know which obstacles or which pits are worth crossing. You’re like, okay, what’s going to be most efficient? Like, are we going to get there, you know, going around this way, going around that way. You like know that you’re trying to get there. And again, it’s not a concrete final navigational target. It’s an interim one. So you’re like, You’re playing sort of loose with the rules and you’re open to revising as you go. So it maintains the sort of flexibility and responsiveness to information in the environment, which are desirable design criteria.
Aneesh [42:37]: Right.
Romeo [42:39]: How does this, does this solve the Darwinian?
Aneesh [42:41]: The crisis of ought meaning from Darwin and the crisis of terrible is meaning from Darwin. I
Romeo [42:47]: think that, so I think that the, I think that the problem with the Darwin meme is that it makes it seem like the navigational target you should have is turn yourself into cancer. Like, oh, well just spreading maximally is the good thing.
Aneesh [42:59]: Whereas… It devalues all other values fundamentally.
Romeo [43:01]: Yes. Yeah. Yeah. It’s, it’s, it’s Milwaukee as all attractors ultimately must be in the end if they’re, if you’re to fall into the very center of them and maximize the one thing. It gives you something to do besides turn yourself into cancer. So as soon as you have cooperation, so we didn’t talk about it that much, but as soon as you have cooperation as your, as your target and like the, the idea of the complexity, the non-tiling, like the non-cancer, then it’s like what would it look like to generate a super intelligent civilization that is able to coordinate across the broadest possible space of mind architectures? This is like an extremely different target than just spread yourself maximally. It’s like a reason.
Aneesh [43:46]: Got it. And so this means that the Darwin meme goes from something that was extremely afflictive as in this is what you ought to optimize for to something you have to operate within the envelope of the same way that a company has to be profitable but many companies are not started with profit as literally the only goal like nobody gives money to a guy saying what does your company do it’ll be profitable like nobody does that that’s just insane right and we can speak rightly consider that insane so
Romeo [44:13]: what’s the point of survival well to survive even more
Aneesh [44:17]: yeah yeah so nobody as I said even if there is a very highly skilled startup founder, right? There has to be some concrete thing that he’s doing beyond just profit. Similarly, it takes it from the optimization target to merely the envelope which constrains what actions you can do and presumably if you push homeostasis further, then you can do even wilder, further out there things.
Romeo [44:42]: Right, you can build even more exotic, potentially valuable mind architectures that can experience things that you couldn’t have have imagined before, right? Which then may in turn open up additional avenues of even more exploration of…
Aneesh [44:54]: So somehow these have to be bought within what you would call the selection envelope. Like instead of aiming for maximizing yourself, you are now aiming to maximize the selection envelope because the selection envelope is what constrains how complex or how complex your organisms can be and what ranges of homeostasis you can…
Romeo [45:12]: Right. At least so far in our universe, complexity is associated with fragility, right?
Aneesh [45:17]: Well interdependence between these agents. That’s why modularity and hierarchy are emergent.
Romeo [45:23]: And propagation speed and things like that.
Aneesh [45:27]: So that is how it shifts Darwin from the target to the envelope. And now you can push against that envelope in greater and greater directions.
Romeo [45:39]: Yeah.
Aneesh [45:40]: Can we bring more things within the selective window so that they don’t in some sense die out? how can we do that and so on. In one very real sense what I think Carol Quigley might call the open society, the one that is open to possibilities, seems he had some basic intuitions about this stuff.
Romeo [45:59]: I think many people have intuitions about this. So like David Pierce’s Triple S Civilization, you know super intelligence, super longevity, super happiness. I think if you take some of those ideas for logical conclusion, which which he does on his various websites, you see hints of a similar idea. And like I said, when I was relating it to Buddha nature, I’m positing that that’s not just a metaphor and that there are important overlaps in a lot of the thinking that various traditions have done. Which, yeah, getting into all of that would be a whole other conversation. The fact that I talked about it in terms of attractors is what I think the Buddhists talk about in terms of realms. realms, because the realms also seem to be, these are possible mind architectures, right?