S2E4 - The Information Hazard (Roko's Basilisk & LessWrong)

0:40 Kayla: Okay. Chris: We're gonna catch all of our beautiful edits that you don't hear because we actually do edit the podcast. Kayla: We're recording. And you need to just make sure that you're gonna talk into the mic. Chris: I know that you're recording. Cause we always do this. You always start recording and then we. Kayla: Just like, say, I'm not recording it. Chris: Yes, you are. Kayla: I'm not. I was recording. Chris: Yeah. Oh, no shit. Kayla: Go do your thing. Do the podcast. Chris: Do the podcast. Okay. We don't care about the quality of the content. We're just trying to get it out. Kayla: And get it just fucking done. Chris: Anyway, welcome back to culture. Just weird. Kayla: Season two. Chris: Season two. Well, we're on now. Don't forget we're on like, episode three still. Kayla: Season two. Chris: Season two. We made it to season two, which.

1:22 Kayla: Is so should we not act like we're like, excited about those? Chris: Cuz we're, ya know, like we should. Kayla: Go back and pretend like this wasn't the first episode we recorded. I'll lie as much as I want. Chris: In any case, this will be episode three of season two. S two. E three. Kayla: Sweet. Chris: Unless we break up one of the other episodes into multiples, in which case that is also a lie, and then it'll be episode four or five or something. Kayla: I'm probably gonna do that. Chris: Oh, boy. So I've already lied. It's been like 10 seconds. Kayla: What do you have for us today? Chris: I have a cult. Or maybe just a weird. Kayla: Do you have a gut feeling about it? Do you want to prime us with it or do you want to wait? Chris: Prime you with my gut, actually. Well, are we going to do banter at all or just go? Just ready? Go.

2:07 Kayla: We did banter. We bantered. Chris: We bantered about banter, which is, I guess, the only thing we ever banter about. Kayla: Well, I hit myself in the tooth with a fork just before going on here. Chris: Cool story, bro. Kayla: And my tooth hurts. Chris: Cool story. We're recording. Oh, I know. We can talk about Perry. We're recording with the mascot cat, who is currently asleep. Kayla: The correct term that mascot was originated on. My other podcast is podcast got. Chris: Podcast cot. Excuse me. Kayla: So he is our podcast. Chris: Podcast got. Thank you, Perry. He is sound asleep right now. Okay, so I guess we can get into it. But first, I just wanted to talk to you about something real quick. You are about to lose the game. Oh, you just lost it. Sorry.

2:49 Kayla: You're doing the game. Oh, everyone listens to me. So mad. Everyone's gonna be so mad. Chris: Yeah. Cause everybody listening just lost it, too. Kayla: Yep. Chris: Can you explain for our listeners what the game is? Kayla: I don't think I can. I don't know. I just know that it exists. The game is just the game. And if you think of. If you think of the game, you lose the game, right? Chris: Yes. Kayla: Yeah. It's a game. Chris: So the game, or game. Kayla: If you think of the game, the. Chris: Objective is to not think of the game. Kayla: Yeah. And if you do, you lose the. Chris: Game of the game, which we are all now doing. Kayla: Yeah. We've lost. Chris: You lose. Kayla: Yeah. Chris: So as long as you're not thinking of it, you win, which you were all winning until I said something. Kayla: Yeah. Thanks. Chris: Sorry. Kayla: It's kind of like getting iced, but, like, psychologically.

3:31 Chris: Psychologically iced, yeah. Oh, man, I miss that. Kayla: This can be still a thing. Chris: But you can't counter game someone, though, right? Like, you can. Kayla: I don't know. You tell me. Chris: I mean, you can't. It's just the way that it works. Kayla: I feel like, technically, if you say you just lost the game, yourself are also losing the game, because I am. Chris: Yeah. Kayla: You have to think of the game in order to say you just lost the game. Chris: Right. But if I. And you have an ice in your pocket. Kayla: Yes. I know how that works. Chris: I know. I'm just saying. Kayla: I'm just saying it does not behoove anyone to say, no, it's lose. Chris: Yeah. The game is just bad for everyone. Kayla: Yeah. Chris: But it's funny and weird. I don't know if it's culty. Definitely weird and interesting, but. Yeah. The crux is that if you're thinking about it at all, you've already lost just by thinking about it. Also interesting is that the game isn't the only thing that has that property. Icing and the game are not the only things that have that property.

4:31 Kayla: Should you explain what ice is? Chris: I guess we should probably explain that, too. Yeah. So if you've never been iced or even know what ice is, there's. There's a. A beverage? Kayla: A delectable. A delectable alcoholic treat. Chris: Alcoholic treat. Very high class, very classy. Called. Wait, it's not natural ice. Kayla: That's the beer. Chris: Smirnoff ice. All I can think of is natty ice. Kayla: No, it's natty light. Anyway, there's. Chris: Oh, I'm thinking of ice house. I'm combining ice house. Natty. Wow. Okay. Kayla: Get out. Chris: I cannot get my low class alcoholic beverages straight. Kayla: Smear, not ice. First of all. Second of all, how dare you? Chris: Sorry. I smirnoff ice and there is just a, I don't know where it came from, how it started, or if people even still do it. But there's this game called being iced, where if you bring someone an ice and hand it to them and say, you got ice, then they have to drink it.

5:25 Kayla: No matter where they are, what they're doing. They could be giving birth, they could. Chris: Be flying landing, 747. Kayla: If you ice them, they gotta drink it. Unless, except they have their own ice, and then they can counter ice you, and then you have to drink both, right? Chris: So if you have an ice on you and somebody ices you, then you pull out your ice and you counter ice them, and then they have to drink. Kayla: It's like when you're playing crazy eights, or it's like when you're playing crazy eights and you play like a two card, and you think you're gonna make the person pick up two, but no, they have a two card, and then. Chris: You got, do we have to explain crazy eights now? Or it can be in this chain. Kayla: Of explain, or it's like uno with the reverse card. Chris: It's like, yeah, it's like reversing an Uno. In any case, icing is similar to the game, but the whole point here is that it's regardless of any physical object or beverage that is involved. The idea is just that if you think about it, you've lost. Yep, it's the thought that's dangerous.

6:23 Kayla: I swear to God, there's so many directions you could go with this. Chris: I know some. We will get to whatever that is. I just wanted to say the catchphrase. We'll get to that. So before we do, it's really. Kayla: I'm very interested to see where this goes, because I might get mad at you. Depending on what this is about. Chris: You're gonna get mad at me. Kayla: I don't know if I can do this. Chris: Okay, so I'm gonna do a quick inline content warning. Speaking of getting mad at me, for. Kayla: This episode, I'm so mad right now. Chris: The content warning itself is kind of interesting, because it's not that we're going to be talking about murders or sexual assault or anything involving bodily harm. Rather, this is a subject that a small subset of folks have in the past found somewhat mentally disturbing in a sort of unique way.

7:11 Kayla: I'm so mad. Chris: So if you're the type of person that has a tendency to overthink things and or you get a lot of anxiety about the future, or generally have a hard time with excess essential dread, this episode may not be for you. Kayla: But except we are both of those things. Chris: I will also say these things, which sounded really weird after you just said these things in a different context. No, that's okay. I'm just, like, blindly reading my script because it's been so long. No, but I. So I'll say these things. First of all, the anxiety that some people feel about this topic is part of the story itself. Maybe it is the story. And secondly, if you decide to stick with us, and no pressure, but I promise that there is a whole portion of the episode where I will talk to you about why the anxiety and disturbance around this topic is almost entirely baseless. Not to invalidate people's feelings that do feel anxiety about it, especially because the folks that get disturbed by this tend to be actually extremely intelligent, hyper rational, smart people.

8:14 Chris: But the premises that lead to these anxious feelings that people have are tenuous at best. Or honestly, I'd even call some of these premises, like silly if you really look at them. Kayla: Well, I call you silly. Chris: And if I didn't think that they were silly, we wouldn't be doing this episode. You have my word. Kayla: Mm. Just cause you think they're silly, don't. Chris: Get mad at me. Kayla: Doesn't mean I'm gonna think that they're silly until, see, you're just unilaterally deciding that it's silly until we get to. Chris: The end of the episode. Don't get too mad. Kayla: I'm Madden. Chris: And then for everybody else, content warning. There may be some anxiety inducing topics that we will talk about as we continue. Okay, so if you're still with us after the content warning, which I'm actually guessing is most of you, because the thing is, content warnings, as good as they are, can tend to have a bit of a Streisand effect. Do you want to explain the Streisand effect? Since we're, like, having to explain, this is going to be an explanation heavy episode. Do you want to explain that one?

9:14 Kayla: I am not really feeling like being cooperative right now because I'm very mad at you. But sure, I'll go ahead and explain the Streisand effect because I definitely wanted to do work today. The Streisand effect is the name that has been given to an effect that is related to Barbra Streisand. The end. No. Chris: Wow. That's a very good explanation. Thank you. Kayla: So Barbra Streisand, the story goes, really didn't want there to be any pictures of her luxurious, I believe, Malibu home anywhere on the Internet or, like, anywhere. And so she was like, no pictures of my house. Which naturally made people interested in knowing what the fuck Barbra Streisand's house looked like. Chris: I think that she even got her lawyers involved. Kayla: Oh, yeah. No, it was like a thing. It was like she kept doing more and more things to try and get pictures off cease and desist, whatever, all of those things to try and get pictures of her house removed from collective conscience.

10:05 Chris: Yeah. Kayla: And the streisand effect is when you try to prevent people from seeing a thing that makes them want to see the thing more than if you had just not tried to prevent them in the first place. So what you're saying with content warning is that, like, if you say, hey, there's anxiety inducing stuff in here, it's gonna make people want to listen more, right. Chris: It's like every time we watch an HBO show and, like, at the beginning, it says, violence, strong sexual content, nudity, we're like, yes, awesome. Kayla: It's gotta have all of those. Otherwise I'm not watching. Chris: But, yeah, I mean, basically, this dry sand effect is what the Internet calls censorship, right. When something gets censored and then everybody reads it with the catcher on the rye. Right. Being censored only had the effect of making it much more widely read than it probably would have otherwise. So anyway, it's kind of like, sorry.

10:54 Kayla: I'm interrupting again, but it's kind of like how every time I go, oh, no, what if somebody, we do an episode about sues us? You're like, hell, yeah, I hope they sue us. Chris: Yeah, yeah, exactly. Which may happen this time, too. I don't know. But I'm not just saying the Streisand effect because of the content warning, it's also something that's going to come up later in the show. So if you're keeping score at home, today's topic overlaps with the game, which is simply the thing that you lose, if you think about it. So you actually just lost it again. Kayla: Yep. Chris: The streisand effect, which we just described in a weird content warning about, like, overthinking stuff. So not sure if it's a cult yet or not, but very clearly weird. Kayla: Yeah, sounds like it. Chris: Very weird. So let's mentally journey back into the distant past, a much simpler era. The year is 2010, and the month is July. President Obama is halfway into his first term, and the biggest thing we're all worried about right now is the BP oil spill. Like, that's it. That's the whole thing. That's all of 2010. That's the biggest thing we had to worry about. Much simpler time. But someone on the Internet had something bigger and much stranger to worry about. In July 2010, a regular contributor to an online forum called lessrong.com made a post. What is lesswrong.com, you ask? Here is the quickie blurb from Wikipedia. Less wrong is a community blog and forum focused on discussion of cognitive biases, philosophy, psychology, economics, rationality, and artificial intelligence, among other topics.

12:28 Chris: I would also add, crucially, decision theory, which is like this weird esoteric discipline that discusses and tries to formulate, like, why people make certain decisions under certain circumstances. Kayla: This sounds so heavy. Chris: Right? Kayla: The Internet is for looking at cute pictures of cats. Chris: I know, but, oh, man, I spent a little bit of time on less wrong for this episode. Like someone like me, like, I do find it fascinating, sure, but it is super heavy. And as we'll talk about later, man, is there a lot of inside baseball like jargon on that site? Like they're throwing acronyms around and like, yeah, it's pretty nuts, but basically they're just like a bunch of hyper nerd, rationalist, AI loving people. Like, they're a bunch of singletarians. Okay, basically, don't mess.

13:24 Kayla: Oh no, you just disturbed him way more than I did. Chris: We disturbed the podcast cat. What was it? Podcast cot. Yes, podcat cot. Kayla: Podcast caught. Chris: Anyways, so that's what less wrong is. So it's just an online forum. The site was founded by an AI researcher by the name of Eliezer Yudkowski, who I think I'm pronouncing that name right, but like, such an awesome name. Kayla: That is a really good name. Chris: Eliezer Yudkowski. Anyway, not Eliezer, but the other person I referenced above, the community member who made this strange and worrisome post on lesswrong.com. The name he goes by on this community forum is Roko.

14:07 Kayla: No, I don't want to talk about this. I really, I. Chris: And he has really don't want to talk about this to some people. Terrifying ideas about a super intelligent AI that doesn't exist yet. Kayla: Some people. Smart people. Chris: Yeah. So anyway, yes, my friends, this is why we had to give a content warning at the top of the show today. We are talking about none other than Roko's basilisk. Kayla: Fuck yeah. Is it roko or rocko? Chris: I don't know. Kayla: We'll say Roko. Chris: I'm gonna say roko. And if you say rocko, then I'll yell at you because that's what you're supposed to do when somebody pronounces something totally validly a different way? Gif. Gif. We'll do that episode sometime.

14:51 Kayla: That is, there is definitely cult behavior there for sure. Chris: So if you know even a little bit about Rokos basilisk, you know that this shit gets insanely esoteric. I mean, you've already kind of gathered that. By the way, I described Lesswrong.com. So it gets, yeah, really esoteric really quickly. And we're going to try to swim into that deep end of the esoteric pool as much as we can get away with. So before we do though, I want to say, are you with me so far? Like, do you have any questions about what we've already talked about less wrong, Mister Yudkowski or Roko or just sort of like the table we've set so far? Kayla: Do you want me to have questions? I don't have questions, but I can make one up.

15:32 Chris: This is just me trying to make sure that everybody is still on board the understanding train. Kayla: So far I'm good. Chris: Okay, listeners at home, any questions? No? Okay, sounds good. So let's start with the basics. What even is Roko's basque? What the hell am I talking about? Alright, because I talked about this guy. He made a forum post and online. What does that have to do with anything? How is it a call? Why is it, why am I talking about AI? So before I answer, and this goes back to what were talking about just a second ago, I want to reiterate something also passingly mentioned in the content. Warning. If you start feeling freaked out at any point during the show here, just remember one thing. The theme of this whole episode is about how dangerous mere information can be. And also that we ought to practice responsibility with information that is potentially dangerous.

16:24 Chris: I did a lot of considering and thinking about whether this episode itself was a safe enough parcel of information to disseminate across our platform and our audience. And I decided pretty conclusively that it was. So if you stick with me till the end, I think you'll see why and you should have any possible anxiety of yours assuaged. Kayla: Is that how you pronounce that word? Chris: I don't know. Kayla: I always avoid saying it. Chris: Well, now we're on a podcast so I have to say it. I guess I didn't have to type it. This is my own damn fault. Kayla: Yeah, that is your own damn fault. Anyway, past me. I trust you. Chris: Okay? I'm glad that you trust me. I'm glad that I'm part of your trust network. Maybe by now. Okay, so again, what is Rokos basilisk. Basilisk. Oh, this is not a good one.

17:10 Kayla: For my perceived list that you don't have. Chris: So the simplest way I can put it is Rokos Basilisk is the name given to a thought experiment about a hyper intelligent future artificial intelligence. This future AI then captures or whatever any humans that knew about the potential for it to exist, such as you and me right now because we're talking about it. And every person listening to this and then tortures them for the rest of their lives. Kayla: For the rest of their lives or the rest of ever. Chris: I think there's differing opinions on that. Kayla: Okay. Chris: It gets into some weird stuff about, like, the AI may not actually torture you. It may torture some simulation of you.

17:52 Kayla: What? Chris: But then, like, the people on less wrong. And this is sort of getting ahead of ourselves here, but they have, some of them feel like a simulation of you is indistinguishable from you. So it basically is you. Kayla: Right? Chris: Or like maybe that's already happening. Like some weird stuff like that. Kayla: I mean, that I believe. Chris: Yeah, I know. Yeah, that's after 2016, I believe that bit of it. But anyway, there's. Yeah, so there's kind of different. Different ways that it executes the whole torture scenario bit. But anyway, the point is, if it comes to be and realizes that you knew it could have come to have been but didn't help it come to. Kayla: Be, it will know, because it's an AI and all, and because it's a.

18:34 Chris: Super intelligent AI, it will know, then it will find you and it will torture you. And so why does it do this? And by the way, let me know if any point, you're not following me or any of the logic here, as weird as it may be. So the Roko basilisk AI is not evil per se. Actually, this is one of the things I didn't realize and I thought was quite interesting. Kayla: How is it not evil? Chris: Almost freakier this way. Kayla: I don't like it. Chris: In fact, it may be friendly or probably will be and is quite good. In fact, it may even be so good, so beneficial for mankind that it runs some simple calculations. Once it comes online and those calculations go like this, the AI's existence is so massively beneficial to all humans that every single day, hour, even minute that it took for it to be developed, that it wasn't developed yet, may have literally cost real human lives, maybe thousands, maybe hundreds of thousands. And by the way, if you're wondering, this is worse, if you're wondering how exactly an AI could be so beneficial. Let's just say for the sake of understanding the thought experiment that this AI, among other things, can instantly come up with, like, cheap cures for all known cancers. Right? Something like that.

19:48 Kayla: Right. Chris: Because the idea among about super intelligent AI's is a key assumption that we're working with and that they work with on this website is that those super intelligent AI's will be so far above us in terms of their intelligence that it feels like they're going to wield powers that we might view as godlike. I mean, yeah, they'll just be so powerful, so intelligent that they'll seem like godlike beings and essentially, and we'll come back to that too, by the way, that idea. Kayla: But also, yeah, it's way worse for him for Rocko to not be evil. To not be evil. Chris: Yeah. So actually that was, I'd never heard that before. That was, it's, that was how it was originally posited. Like the whole evil thing is sort of just an artifact of like, well, feel torturing AI. Oh my God, that's an evil thing to do. But actually, the way that it was posited is that you've got this godlike AI bringing about this massive benefit wealth discovery, maybe ushering in a golden age, the likes of which we've never seen. So much so that the AI, in its infinite mathematical wisdom, knowing that if it were now to punish people who knew of its potential future existence but did not help bring it about, that would motivate and hasten its development, which is actually a good thing because of how much benefit it's bringing.

21:06 Kayla: I hate this. Chris: And in fact, Rocko says, rocco, we're going back and forth. Rocco even says in his post something about, like, it's something, I forget the word. It's like he says, like brutal in the way only utilitarianism can be. So this dovetails, which again, we'll get to that quite nicely with utilitarian morality. And utilitarian morality is something that is very prominent on lesswrong.com dot. Okay, one of the things you should maybe keep in mind is that utilitarianism is not the only framework we have for morality. Kayla: I've never heard that before in my life. Chris: Utilitarianism? Kayla: No. Chris: Oh, okay. Well, I actually have a section where we're going to talk about it, but we can talk about it now just real quick, just to kind of level set what it is, and then we can keep going. Utilitarianism is basically if morality was math. So the, for example, so think about. Well, no, but everybody's heard the utilitarian experiment, thought experiment of the trolley.

22:08 Kayla: Oh, yeah. Okay. Chris: Right. So the trolley is. Do you like there's a train or trolley train and it's coming down the tracks and it's totally going to hit five people. But you can pull the lever and cause it to go on the other track and hit only one person, but then you're murdering someone. Kayla: Yeah. Chris: So would you murder one person to save five? And if the idea behind that thought experiment, which actually a little tangent here because I wound up reading a little bit more about that. You know how these things are, man. Kayla: You're measuring the coastline. Chris: You're measuring the coastline, you start zooming in and it just gets more complex the more you go.

22:51 Kayla: Yep. Chris: So side tangent about this is that's supposed to tell whether you're utilitarian or not. I think there's actually a name for. If you say, I don't do anything because I'm not going to kill someone, that is the intent behind that question. Because the utilitarian is supposed to say, well, I would definitely kill the one person to save the five. Kayla: Right. Chris: But actually they find when they put people into brain scans and ask them that question, people tend to whatever the section is of their brain that lights up when they're doing that. It's like, it's more of a measure of like, sociopathy than it is of utilitarianism. Their brain lights up more in the like. Yeah, I don't care if people have to die.

23:35 Kayla: Wait, so if you kill the one to save the five, you're sociopath, or if you don't do anything, you're sociopath. Chris: It's if you kill the one to save the five. But let me check on that. Da da da. Philosophically, the sacrificial dilemma has a narrow purpose. Your choice supposedly illuminates. I'm reading this from one of the sources that I'll post in the show notes. It's from a really interesting website, actually called lastword on nothing.com. We can talk about what that is. But anyway, reading from here, philosophically, the sacrificial dilemma has a narrow purpose. Your choice supposedly illuminates whether you fall into one of two camps on moral reasoning, choose to hypothetically end a life and save a few more. And yours is described as utilitarian judgment rejected. And you are said to be making a non utilitarian, which they call deontological judgment.

24:20 Kayla: Deontological. Chris: Yeah. Kayla: I love that. Chris: So roughly translated, the utilitarian is concerned primarily with outcomes, while the deontologist has a morally absolute point of view that holds that you couldn't tell a lie to save someone's life because it's wrong to tell a lie. And they even mentioned that Kant is the most extreme member of this camp. Kayla: Wait, so if you wouldn't tell a lie to save someone's life, you're deontological? Chris: Correct means you have morally absolute positions like it's wrong to kill, period. I don't care that there's five people on the other side of the track. Kayla: That's just wrong. Chris: Well, maybe you're a utilitarian, then. Maybe you should go spend some time on lesswrong.com in any case. So they go on to say that doing some of these experiments reveals that not only does a utilitarian response not actually reflect a utilitarian outlook, it may actually be driven by broad antisocial tendencies, such as lowered empathy and a reduced aversion to causing someone harm. Which kind of makes sense in the real world, given a choice between two kinds of harm, most people wouldn't be able to cost it up quite so coldly.

25:25 Kayla: That makes sense, right? Chris: Which is part of the thing with a trolley problem, too, is that it's like, I think most people that think about the trolley problem, or at least when it's posed to them for the first time, go like, isn't there any way that we could, like, just put on the brakes, right? Or could we derail the trolley? Could we just untie the guy? Like, it's hard to. When you completely strip away that real world context, then you find yourself in the situation where the answer that you give says more maybe about, like, one aspect of your brain than the other. You know, if you have to force you to answer that, then it's like, sure, yeah, five is better than one, I guess. But most people, I think, have, like, maybe an initial aversion to it.

26:08 Chris: And if you don't, I think that's what they're saying. Like, five is greater than one. Yeah. Do that. Kill the one person for sure, right? Versus saying, like, can I save all of them? I don't know. Kayla: Right, right. Or like, because for me, I like, when I'm posed like that question, I know I would want to say if the only options are kill five or kill one. I know want to say I'd throw the switch and kill the one person. But then I'm really, like, honest with myself. I don't know if I would actually. It's hard be able to do it. Chris: Right. Kayla: Like, even if morally, I'm like, that's the thing you should do. But I don't actually know if I could do it. I think I would freeze up and panic and not even be able to, like. Chris: Right. And there's a lot of other things, too, that play into it that, like, you have to control for. You have to ignore in the trolley problem, which is, like, probably what you would do is whoever you were, like, closest to. Like, if the person that you were killing was, like, in the room with you saying, please don't kill me. And the other five people were, like, half a globe away, then you would probably let the other five people die.

27:14 Kayla: Right? Chris: Like, there's a whole thing. But two about how proximity makes it a more. Proximity makes it much easier to kill, which is, like, part of a problem with our modern weapon systems. It's really hard to kill someone with a knife. It's a little bit easier to kill them with the gun. It's easier to kill them with, like, a sniper rifle or a mortar, or it's even easier to kill them with a drone system from half a continent away. Kayla: Right. Chris: So anyway, back to utilitarianism. That's one of the things that Rocco mentions in his post, is that there's this harsh utilitarian nature to. To the thought experiment that this robot is actually maybe doing good. And because it's doing so good, so much good, that it's actually good for it torture these people that knew about it and didn't bring it into existence.

28:02 Kayla: Yep. Chris: Right. Now, again, if you're getting freaked out about this, trust me, there's plenty to undermine this, least of which being that utilitarianism is not the be all and end all for morality, which means that's not definitely the thing that a robot would do. Kayla: I also don't know if you've quite illustrated as to why people might be getting freaked out about this at this point. Chris: Yeah. Which is actually kind of a good thing. Kayla: Like, yeah, so I was gonna say the why, but. Chris: Well, so, yeah, I mean, we'll kind of get to that. But it's also, like, people that like our listeners, unless they're also less wrong forum members, are probably gonna have to make some mental leaps to get to the point where they're like, oh, I see why. It freaks people out, which is actually good.

28:46 Kayla: Right. Chris: And normal, I would almost say. And that's kind of what makes. That's also kind of what makes this a good topic for culture. Just weird, is that there's, like, a certain subset of people with their own rituals and vocabulary and everything that. That make this, like, an effective sort of harm that is, that happened to them. That's kind of what makes it a good topic for the show. Let me go back to the point in the script where I was. So were talking about Roko's basilisk and how it has this utilitarian imperative to say, I'm gonna torture people if they knew about it and didn't help me bring into existence. If you're still not kind of getting it, try this analogy. Let's say you're trapped in a.

29:30 Chris: In a house, and there's only two rooms in this house, and you're trapped in one room with a bunch of people, but the other room next to you is an abusive monster, and the monster is locked up as the door is locked. So if even one person starts trying to unlock the door, you might reason out that, oh, boy, you better also help unlock the door so you don't get punished for effectively increasing the time he's been locked up in that room. So the monster doesn't actually even have to say anything to you or your fellow captives or even know that you started picking the lock to have his future decisions and your current motivations interact in this weird way that makes the locke gets picked faster. Kayla: Right. Chris: And that may make you worry now that, oh, shit, I'm gonna get tortured by this future thing that may or may not happen.

30:15 Kayla: Right. Chris: That's basically Roko's basilisk in a nutshell. If you know about it and then it becomes real and you didn't help it exist, it will torture you forever. Kayla: Yep. Chris: So let's just say this has freaked some people out, mostly on the less wrong forum and me and Kayla, which we will get more into later. Once again, if you're feeling that, I promise you, this entire thought experiment sits on a house of cards, and a lot of the cards are missing, so try not to worry too much. But you were just asking me a little bit about, do people understand why it freaks people out. So I think I've kind of illustrated that because it's. It's the idea that, you know, oh, no, I'm worried about this future artificial intelligence that's just because I know about it is going torture me. And actually, unfortunately, that's actually sort of the most interesting part to me about Roko's basilisk is that the danger isn't really the AI, it's not actually the basilisk.

31:09 Chris: And, in fact, the basilisk isn't even really referring to the AI itself. Kayla: Oh, really? Chris: It refers to the thought experiment. It refers to the piece of information. So this goes back to the top of the show when I was talking about, you know, the information being dangerous. That's the actual danger. Knowing about Roko's basilisk, if you believe in it, is the thing that makes it dangerous to you. And in fact, that's where the badass name comes from. Obviously, the roco part is named after the less wrong forum poster that posted about it. The basilisk part references the mythological creature known as the Basilisk, whose power as a creature is that anyone that even looks at it dies instantly just from having seen it. Something I learned in doing the research for this episode, actually, is that Rokos basilisk is already a second degree reference. It's not actually referencing the mythology.

32:00 Chris: I mean, it is, but really it's referencing something else that's referencing the mythology. It's from a short science fiction story called Blit. It sounds terrifying. Actually, Blit doesn't sound terrifying. I read about the story and the story sounds terrifying. In this story, humanity has developed certain images that they call basilisks that have the property of having some specific patterning in them that if a human looks at them, it triggers the brain to undergo a lethal shutdown. Sort of like introducing a virus into a computer that causes it to crash. Kayla: Jesus Christ. Chris: So Roko's Basilisk gets its surname, actually from that short story which references the whole, if you look at it's already too late. You're fucked. I was short aside here. You know the movie the ring?

32:48 Kayla: Yes. Chris: So when the ring was first advertised in theaters, I thought, like, it's. So the movie is actually about is like if you watch a video, then a ghost comes out of the tv and like, you know, kills you in seven days. In seven days. I actually, when I originally saw the trailer for it, didn't know it was a ghost thing. I thought it was more of like, if you watch this video, it. I actually thought that it was like this basilisk sort of thing that like, if you watch the video that it just like, fucks up your brain mechanics and then, like, you die from it. Which to me was always a much cooler, scarier thing than just what I learned. It was just about another ghost. I was just like, eh, ghost, whatever. But this idea, did you ever see it? Chris: Yeah, it was decent. In any case, just decide there about. I thought the ring was actually this.

33:33 Kayla: Aren't you just so smart? Chris: Oh, man. Yeah, I am. Super smart. Somebody please give me a movie to make, because I'm obviously way ahead of the curve on this stuff. Kayla: There you go. Chris: So now you also know why we referenced the game at the top of the show, which, by the way, you just lost again. Kayla: So rude. Chris: But a bunch of people have compared this whole phenomenon to the game really predates Rokos basilisk by a number of years. A few years, yeah. Which you. You can kind of see why, right? Because the whole thing is. Again, it's not about the AI. It's the thought experiments. That's. That is the dangerous thing, because the thought experiment has caused people to stress, and it caused people distressed by this weird sort of. Oh, my God. As soon as I know about it, I'm screwed.

34:15 Kayla: Right? Chris: The same way that the game does. Like, as soon as I know about it, I've lost. It's sort of a high stakes, anxiety ridden version of the game. Perfect for millennials. Kayla: We need that. Chris: All right, quick breather time. Any questions about what Roco's basilisk is or about its weirdo basilisk nature? What a basilisk is? Kayla: What does Roko's basilisk look like in your mind's eye? Chris: It looks like a little lizard. Kayla: Mine looks like. Chris: Actually, you know what it looks like? Kayla: What? Chris: So, I mean, it looks like a lizard, but it looks like a very specific lizard. Once again, here's a reference. Magic the gathering.

34:56 Kayla: Oh, God. Chris: There was a card in the first set of magic called a basilisk, and it had the property that if it engaged in combat with any other creature, the other creature died of. Kayla: Mm. Chris: Get it? Anyway, that's just the image I have in my head of a basilisk now. And it just kind of looks like a fucking iguana. Like, it's nothing special. Kayla: Well, mine looks like. Let me see if I can find, like, a picture of it. Okay. In my head, you remember Trogdor the burninator? Chris: Vaguely I remember the name. I don't remember what he looks like. Kayla: Wait, wait. Trogdor the burninator from strong bad, which deep ass cuts right now? He looked like this. Chris: Yeah, yeah. Kayla: So he looks like a terrible drawing of a dragon with a single buff arm.

35:40 Chris: Yeah, he definitely skips leg day. Kayla: Like, in my head, he looks like. Chris: That's pretty good, actually. Kayla: Really realistic. Like a more, like, scary realistic version. And I literally just googled Trogdor the burninator realistic. And I got, like, fan art. Please show me of a magic car. That's what Rocco's Vascules looks like to me. Chris: That is a very good depiction. It's not what I was thinking. Kayla: That's what it looks like to me. Chris: But it's much better. Kayla: Yeah, that's what Roko's Basilisk looks like to me. Chris: That's a good one. Kayla: Yeah. Chris: So now that you know what it is, let's go back to our little historical timeline. Kayla: Okay. Chris: So Roko makes his posts July of 2010, and it garners some thread comments on the site that vary from things like, this is an interesting post to, this is super entertaining to, I don't know, man. This is, this kind of, this is a little bit freaky. There's places that have preserved this original post. I mean, I didn't read every single comment on the post, but I didn't really get the impression from the comments that it was like, that people were really losing their minds over it.

36:44 Kayla: Interesting. Chris: But there's one comment that I can almost guarantee that if this comment never happened, no, we wouldn't even know about Rokos Basilisk to this day. And that was the comment reply from Eliezer Yudkowski himself. Kayla: Oh, Jesus Christ. Chris: He is very active on the less wrong forums that he created. I've read some of his stuff. It's very interesting, very full of all of the jargon town. Kayla: Right. Chris: He seems to generally be, by the way, we'll get into him more later because, you know, criteria. But he seems like a generally a good dude, smart dude. He actually came up with the whole concept of like a friendly super intelligence.

37:25 Kayla: Right? Chris: A super artificial soup. A friendly artificial super intelligence. Kayla: A super friendlier, super friendly. Chris: Not very super smart, like friendly, but dumb. No, he came up with the idea of that. So. Kayla: Okay, so you're saying he's our charismatic leader? Chris: Maybe I'm saying that. Kayla: Oh my God. Chris: I mean, he started less wrong. Kayla: That's true. Oh, he started. Chris: He was the creator of less wrong. Kayla: Oh, I don't think I missed it. Chris: Did I not say that? Kayla: No, you might have. I think I just missed it. Chris: Yes, he is the creator of the less wrong forum, but he's also very active, posting and commenting on. Kayla: Okay, here's my theory. He is, he is the super friendly AI. And this. Oh my. Chris: You're gonna change your idea about super friendly here in a second.

38:07 Kayla: No, but that's, he's the product. Chris: He's the thing. Kayla: This is. Oh my God. Chris: Is your mind blown? Kayla: Yeah, that's, he's the, I can't say he's the basilisk anymore. Cause the basilisk is the thought experiment. He's the AI. He's the AI. Chris: Okay, maybe he's trying to warn us. I think he's a person. But you're right. Maybe he's. Kayla: I think he's the AI. Chris: Maybe he's the friendly AI. That's. I mean, we're all. Here's the thing. We're all artificial intelligences anyway, right? We're all man made. Kayla: I know I am. Chris: Hey. Oh. I don't know if I would say intelligence in your case, though. Kayla: Oh, my God, you made a joke. That was basically the same joke that I made. You're so funny and smart. Chris: Mic drop. Yeah. Anyway, let me quote you just. He has a pretty long comment reply on this post, so I'm just gonna quote you part of it. One might think that the possibility of punishing people couldn't possibly be taken seriously enough by anyone to actually motivate them. But in fact, one person at SiAi, which we will talk about that later, which is the singularity institute for Artificial Intelligence.

39:08 Kayla: Oh, geez. Chris: One person at Siai was severely worried about this, to the point of having terrible nightmares, though they wish to remain anonymous. I don't usually talk like this, but I'm going to make an exception for this case. Listen to me very closely, you idiot. All caps. You do not think in sufficient detail about super intelligences considering whether or not to blackmail you. And by the way, blackmail is the whole process of saying, if you don't help me come into existence, I'm going to punish you. That's the blackmail they're referring to. Kayla: I don't think I understood that sentence at all. Chris: The one I said or the one he said? Kayla: The one he said, yeah, me either. Chris: But here, let me. Let me reread it regardless, just because I like reading in all caps, you do not think. But you understand the blackmail bit, right?

39:54 Kayla: No, I didn't understand the sentence. Chris: Okay, pause in the sentence. They. When they talk about. Kayla: Who's they? Chris: People on less wrong. Kayla: Okay. Chris: When they talk about the future. Rocos basilisk intelligence punishing you if you didn't help it come into existence. They call it blackmail. The same way that I would be. Like, if you don't, I have dirt on you, so you better give me money, or else I'm gonna release the dirt. It's that. Kayla: Okay, so Trogdor is blackmailing you. Chris: Roko is. But not Roko. Roko is the guy that made the post. Kayla: Yes. Chris: I'm saying yes, the AI is blackmailing you. Okay, you may wonder, how does the AI blackmail you when it doesn't exist yet? That's actually a key part of their whole thing, which we'll get to, but that's their shorthand for that whole thing we described is blackmail. Okay, so let me reread his sentences. And so he's replying to Roko here. You do not think.

40:44 Kayla: Wait, wait. Okay, okay, okay. Sorry. I got really confused. I got confused. I got really confused. I thought that the guy who created less wrong was also the guy who posted about Roko's bachelor's first. But no, it's two different guys. Chris: Two different guys. Kayla: That's why I was very confused. I was like, he's talking to himself. I don't understand. Okay, I get it. Chris: Okay, I understand now. No, so Roko posts this thing. Kayla: Yes. Chris: On less wrong. Eliezer Yudkowski, the creator of less wrong, comes to this post and replies to it in a comment. Kayla: Yes. Chris: And then in the reply, he says the things I already said about somebody at Siai having nightmares about this. And then he says, I don't usually talk like this, but I'm going to make an exception for this case. Listen to me very closely, you idiot. Talking to Roko, okay, you do not think in sufficient detail about superintelligences, considering whether or not to blackmail you. That is the only possible thing which gives them a motive to follow through on the blackmail. That's where I'm gonna end the quote. But he goes on to continue with his comment in equal measure of let's call it urgency instead of hysteria. Okay, there's actually a bunch more of that quote. And I was saying, let's call it urgency instead of hysteria.

41:54 Chris: I just don't want to call it hysteria because I think he actually had good reason for being so upset. And he's since talked about the fallout from Roko's basilisk. And he says that his upsetness, which I believe him, was not because he thought the idea itself had real merit, but rather that it was reasoned out powerfully enough by Roko. Like, he presented this case powerfully enough that it was causing people that had these thoughts and these ideas in their brain from already being less wrong members and thinking in certain ways and having this utilitarian mindset and all blah, blah, that it was causing people distress in his community. Kayla: Right. Chris: That makes sense, and that's super valid. Just remember when were talking about how the most interesting part of the basilisk isn't the thing itself, but the fact that the idea was hurting people.

42:43 Kayla: Right? Chris: So that's what this is. And he'd probably be pretty mad at me then for making a podcast about this, actually. But that's why I'm taking so much precaution to content, warn everyone, and also to get to the part where we clean up this mess we're making. A. And actually, we mentioned this before, but because this podcast is directed at a general audience and not at members of less wrong who have all those mental preconditions for something like Roko's Basilisk to bother them, it's possible that he may not even have a problem with this. I mean, there's been a ton of articles come out, which that's what we're about to talk about here. We're finally at the part of the episode where we said we'd get back to the Streisand effect. Remember were talking about the Streisand effect? Kayla: I do remember we talked about the Streisand effect. Chris: The Streisand effect being the whole censorship. If you sensor catcher on the ride becomes more popular.

43:26 Kayla: Right. Chris: If you strike me down, I shall become more powerful than you can possibly imagine. Kayla: So bummed you said that, because I was gonna say it. Chris: Oh, I should, man. I'm sorry. I should let you react more. Kayla: No, you. You're the bigger nerds. That's fine. Chris: Mmm. That's fair. Yeah, you're just the pretty face. So that's what ends up happening here, because what Yadkowski did next after he made that very long, angry, all caps, idiot calling comment, was to delete the post about Roko's basilisk. Kayla: Oh, jeez. Chris: And ban any discussion of Roko's basilisk on less wrong. Kayla: Oh, my God. Chris: Even, like, deleted stuff that got discussed after that. Kayla: Holy shit. Chris: He was very concerned about it, and he's also, I think, since said, like, that may have been the wrong thing to do.

44:07 Kayla: Oh. Oh, my God. Oh, my God. He did it. It's his fault. He's the basilisk. Chris: Yeah. Kayla: Oh, my God. Chris: Well, I mean, I don't know about that, but, like, remember when I said if it weren't for this comment, we wouldn't be sitting here talking about it? That's 100% true. Kayla: But even though he tried to suppress the discussion about it, if the AI became a lie, if it became real, he would not be up for torture because he actually did the work. Chris: That's true. The more people know about it. Maybe that's why he did the Streisand effect thing was because he was worried about Roko's basilisk and. And reasoned out that if he spread the information by streisanding it, then they'd be like, yo, good job, buddy. You helped people know about me, and now I'm real.

44:57 Kayla: That's exactly what happened. We cracked this case. Chris: We cracked it. It's funny that we just went through that because that type of. Oh, if he knows, then I know. And then that thing. Kayla: Yeah. Chris: Is their whole site. That's all. Less wrong is. It's like they talked about, like, remember when I said most of what they do is decision theory? Kayla: Right. Chris: It's all that. It's all, well, if the AI knows that I know this, then it's gonna torture me, because XYZ, that's the whole thing. You already knew. Even if we didn't say Streisand effect earlier, you know what that effect would have if he. If I just said, yeah, he banned it, and then, like, deleted stuff, the idea basically took off. Kayla: Right. Chris: And pretty soon, you have articles on vice, on Slate. Even business insider wrote about the damn thing.

45:44 Kayla: Slate's where I found out about it. Chris: Yeah. Slate, I think, is where most people found out about it. Kayla: Slate. Do you remember when I sent the article to you? Do you remember? Yeah. Chris: Because it's called the most terrifying thought experiment of all time, is the name of the article. It's very click baity. I believe it took four years for it to get to the mainstream. It probably was starting to get into some Internet spheres before that, but I think the Slate article is the one that kind of took off. But, yeah, it really exploded into the general consciousness to the point where a lot of our listeners probably already know about it. Hell, I knew about it. And I don't think before doing this research that I had even heard of less wrong, which is where it originally came from. Kayla: Just be clear that had you. You knew about it because of me.

46:27 Chris: The important thing here is that, yes, my own wife polluted my brain with a dangerous idea. And this was, like, way before I did all this research that made me, like, not worried about it at all. Kayla: Yeah. Chris: You were just like, hey, here's this thing that me sharing it with you fucks you already. Kayla: Yeah. Chris: Here's a look at this thing that could kill you just by looking at it. Kayla: Yeah. Chris: Yeah. Thanks a lot. You're welcome. I appreciate it. But had you heard of less wrong? Kayla: No. Chris: Yeah, I hadn't either, which is kind of crazy, by the way. You know that guy. What's his name? He's famous for being famous. He shitposts on Twitter. Oh, yeah. Elon Musk. He got together with Grimes because she made a tweet about Rococo's basilisk.

47:10 Kayla: Yep. Chris: Which is just super clever. Kayla: Oh, my God. So funny. Chris: Do you want to explain? Should we explain Rococo or. We've already explained too much. It's just an art style that's, like, very ornate. Ornate. Kayla: Overly ornate. Chris: Overly ornate. But the point is that it sounds like Roco. So she said, rococo's basilisk, and then I guess Elon, he thought it fought. He would just slide into her DM's, yada. Then they became an item. Kayla: Now they're having a baby. Chris: Now they're having a baby. Kayla: So let's be clear. Chris: Thanks. Less wrong. And Roko. Kayla: Elon Musk, inventor, science entrepreneur, man Grimes, alternative canadian pop singer. Chris: Yeah, I prefer my description of famous for being famous and Twitter shitposter, but I guess you could sort of say he's an entrepreneur as well.

47:55 Kayla: Oh, and then she went from being hardcore lefty to be, like, union busting because she got with him. Chris: Well, you know. Or is it because she. The Rococo's basilisk? Kayla: Maybe it was Rococo's basilisk. Either way, she's a union buster. Chris: Now, Elon, don't hate us, just in case we ever need to have you interview on the show, because we could totally do that. I know he does interviews on podcasts because I've seen him smoke. So, anyway, that's how Elon got together with Grimes, I guess. Bizarre twitter references is like Tinder for the hyper wealthy, I guess. Yeah, I don't know. Anyway, so here we are, ten years later, and you and I are now part of the pop culture explosion that brought this whole weird post to the public consciousness. Distressing to some, but ultimately more, I think, interesting and more cultural. Just weird. I. So, yeah, I feel good to be part of that whole thing. It's pretty cool question, though. Yeah, we've talked about this in, like, four or five different places now. How worried are you still about Roko's basilisk?

48:57 Kayla: I mean, clearly I'm not so worried that I won't talk about it, and clearly I'm not so worried. I've done nothing in order to bring about its existence. Although I guess one could argue that. Chris: You haven't done shit. Although now we are. Kayla: Yeah, that's what I'm saying. Chris: This is our ticket out. Kayla: One could argue that by me bringing the article to you, I did my part in that. Now you have spread the word to our listeners. So. Roko's basilisk. I did my job. This is what I'm supposed to do. Chris: So, you know, there are some followers of this. Whatever thought experiment, followers of the basilisk that think that it's not, you did something. It's, did you do as much as you possibly could? Sure, we put it on our podcast, but did you liquidate your entire life savings and, well, I feel like those people are wrong. Well, you're welcome to disagree, but there's some people that think that if you don't do as much as you possibly can, then you're gonna get tortured.

49:57 Kayla: Here's the thing. The super friendly, intelligent AI has to be rational enough to understand that human beings brains do nothing, work in such a way that the mere possibility of something is going to be sufficient enough motivation in order to get them to do the thing. Chris: Oh, you're starting to pierce the veil. We'll get to that. Kayla: Okay. Chris: We'll get to that. Kayla: Okay, good. Chris: So, actually, that's sort of what this transition is that I was about to do. If you are still worried, it's time to start administering the cure. Kayla: Thank you. Chris: I need a lead into that. I'd like to. I'd like to read to you just an amazing and insanely relevant quote from one Philip K. Dick. It goes, for each person, there is a sentence, a series of words, which has the power to destroy him. Another sentence exists, another series of words which will heal the person. If you're lucky, you will get the second, but you can be certain of getting the first. That is the way it works. On their own, without training, individuals know how to deal out the lethal sentence, but training is required to deal out the second. End quote. I really like this Philip K. Dick quote.

51:07 Kayla: That's a good quote. Chris: It's literally this. Right, right. And it's actually, I don't want to. We'll get to that. But we'll get to that. There's a whole interesting paper that I read for this episode, and it's about something called information hazardous. Kayla: That's my band name. Chris: And information hazard is essentially theme of this episode. It's information itself can be harmful. Kayla: Right. That's what the government tries to tell us. Chris: It's super interesting. This paper is really interesting because you kind of go like, oh, I wonder what that's about. And he really outlined just more ways than you think that information by itself can be harmful. And kind of, when you get to the end of it, you're like, oh, yeah. Like, sort of. The pen is mightier. Right. It's that kind of thing. Like, information is actually extremely potent. The penis mighty.

52:04 Kayla: Sorry. Oh, God. Chris: Oh, my God. Kayla: Oh. Welcome back. Chris: So we talked about that first series of words that can destroy you already. So let's start with the next series of words that can heal. To really understand not just what Roko's basilisk is, but why it is and how it came to be, which is part of this process requires some background and actually a few different areas. And this is where we start talking about utilitarianism a little bit, which we kind of already talked about in the episode. So I won't have to say as much here, but the basic idea behind utilitarianism is that you are concerned with outcomes and you take a very mathematical approach to it.

52:49 Kayla: Do you feel like that's you? Chris: Maybe? Yes. I don't know, like here. I. So I think. Is there like a word for, like, flexitarianism? Because I think that there are some cases where you have to make utilitarian choices because it's. So let me do this quick tangent for me to answer that. I kind of have to talk about the Kobayashi maru. Kayla: See, I told you were the bigger nerd. Chris: Yep, I definitely am. So for those of you listeners who don't know Kabayashi Maru, good for you. It is a good for you. That means you're not a nerd. It is a fictional Star Trek reference. Kayla: It's actually pretty cool, though.

53:29 Chris: It's very cool. So the idea behind the Kobayashi maru is that as a test that they give to Starfleet cadets in their training. And it's a. It's a simulation test. So basically you go into this room, it's like a. It's. They act. I mean, they do simulations in the navy and the air force and whatever, but it's similar to that. And so you're a pretend you're a captain for a bit and you have your teachers or like all the officers around you, and then you do this training thing. And the idea is that there's some distress signal that's coming from over the border of an alien race that the Federation is at war with. I think it's the Romulan. Chris: You have to do the simulation of going to rescue those people, but you have to make these hard decisions of like, if we go over there and try to rescue this ship, then it may cause even more harm because then we're violating this treaty. And bottom line is the whole thing is like a glorified sort of trolley problem that they give to cadets. And it's, you know, it's supposed to basically test to see how a captain in a command position will respond to situations that sometimes call for harsh decision making. Most people don't have to pull the lever to kill the one in favor of the five, but if you're the captain of a starship, you might have to. So that's why they give them these tests. And famously, Captain Kirk, and it's an unwinnable test, right?

54:49 Chris: So, like you, it is a guaranteed you will lose the test. Nobody has ever won. It's designed to be lost, just. It's how you lose. I it, that is instructive. But famously, Captain Kirk is the only person to have ever won the Kobayashi maru, because. Because he cheated. Kayla: It's so hot. Chris: Yeah, because he was like, fuck this, and went in and reprogrammed the test ahead of time, so that it was like, and I think he actually didn't even reprogram it to be like, winnable. I think he reprogrammed it to be like, easy mode, so he could just. Kayla: I don't know if that's what happens in the new Star Trek, is it not? Chris: I don't know. Kayla: I think he just kind of goes like, fuck it, I'm gonna do it. Maybe I'm wrong. I think in the Chris Pine one, he doesn't do that, but I could be totally wrong.

55:30 Chris: So the reason I bring that up is because I think, I really like to use that as sort of a framework for how I think about morality sometimes, because I think that there are some cases when what Kirk does is not correct, and it's like, that's not gonna be the real world, man, the real world. You're gonna have to make these hard choices, and sometimes you are going to have to, as the starship captain, or as the captain of a aircraft carrier, or as a leader of a company, whatever. As a leader, you might have to make a decision that you need to be utilitarian, and it's going to be hard, and you have to live with stuff like that and be able to do that as a leader. Chris: But then I think there are some other cases where it's better to be Captain Kirk, and it's better to say, wait, you said a or b, is there a circumental, like, what is the third path? Right? Like, is there a way to stop the trolley? Is there a way to untie the person? Right. Kind of going back to that thing. So to me, it kind of comes down to the Kobayashi maru. Like, sometimes you have to be Captain Kirk, and sometimes you just have to do the damn test, right. It's not always clear, though, which one it is. I don't know. That's my answer to whatever the hell you asked.

56:37 Kayla: It's a good answer to the question that I don't remember that I asked. Chris: Yeah. All right, I. Cool. What did you ask? Kayla: No idea which form? Chris: Oh, you said, like, am I a utilitarian? Yeah, that's my answer to, am I a utilitarian or not? Kayla: It's let's go watch Star Trek. Chris: So here's where utilitarianism comes into our story. The less wrong forum, where Roco's basilisk was birthed, is chock full of utilitarian thought. So we talked about this earlier on the show a bit, but less wrong. Articles and discussions frequently resemble that scene in the Princess bride where the. Does that whole, like, oh, God, I know, you know. I know. But then if I know, then you know, then therefore, typological gymnastics. And that's all based on decision theory, right? Which we've talked about before, and we'll talk about in a minute again. And the decisions made in those. I know, you know. I know. Chains tend to be based on some utilitarian calculation, right? So it's like, I know, you know, I know. And this is how I'm sure that the AI is going to behave, because it's going to do this utilitarian calculation.

57:40 Chris: For example, I might predict that you will decide to purchase a certain bag of chips if you go to the grocery store because you like those chips. Kayla: Okay? Chris: That's the basic thing. Like, if they provide you some utility. So I'm gonna say I predict that is the decision that you, Kayla, will make in the future when you go to the grocery store. That's the type of things they're doing with their decision theories. Kayla: I can poke a hole right in that right now. Chris: Okay. Kayla: Do you want me to. Chris: Sure, go for it. Poke a hole in those chips. Kayla: I don't think I have ever gone to the grocery store to purchase myself a bag of chips. Chris: Okay. That's not what I meant with the example. Just saying the example is. Okay, what's something? Fine. What's something? You have gone to the grocery store to purchase tofu. All right, so then I might predict that you will buy a certain thing of tofu because you like that brand. You find utility from that brand. So I can predict your actions in the future. That's what they're doing on lesswrong.com dot. Or maybe I put you in a trolley problem simulator and predict which way you will choose, and I will use math to make that prediction because obviously, everyone, including you, makes your decisions using utilitarian framework, right?

58:43 Kayla: Absolutely. Chris: Everyone obviously must be using utilitarianism. So that's what they're doing on these forums. They're assuming a universal utilitarian framework, everyone, including inscrutable future hyper intelligent AI's, and then using that to predict what decisions people and AI's will make. AI is like the one that Roko thought up right now, I'm going to say this again later because it's important, but utilitarianism is a useful tool to deploy to understand how decisions can be made. But it's just a tool, right? It's not something that can be applied to all things, even all decisions at all times. And it certainly isn't a good enough reason to invent a vengeful AI deity. But, you know, as they say, when all you have is a hammer, everything looks like a nail, I guess, right?

59:26 Kayla: True. I tested that out once. It really changes the way you see things. Chris: You tested out only owning a hammer? Kayla: Mm. Chris: Mm. Yeah. Kayla: That must have been very difficult time in my life. Chris: How did you drive to work using the hammer? Kayla: I didn't because it looked like a nail, so I smashed my car. Chris: Yeah, yeah. That's definitely what the analogy is here. Yeah, yeah. No, the analogy here is that the hammer is utilitarianism. Kayla: Right. Chris: If that's all you're using, then everything kind of seems like it must be. You can break it down into some sort of utilitarian calculation. Kayla: Right. Chris: I'll add on to this that utilitarianism isn't just something that deals in whole numbers either. And this is especially true when it comes to discussions had on lesswrong.com dot. Utilitarianism can also deal in probabilities.

1:00:12 Kayla: Ooh, probabilities. Chris: Probabilities. So these are the types of thought experiments that go, if a magical genie says you can kill one person, and then I'll flip a coin, and if it comes up heads, I'll cure all cancers, would you do it? Kayla: Wait, say that again. Chris: So you find a magic lamp and you rub it, and the magic genie comes out and says, if you kill one person, one random person on the world, you have to decide that they will die. But then if you do that, I will flip a coin, and if it comes up heads, I will cure all cancers that have ever existed. Kayla: Ugh. Chris: Would you do it? Kayla: Probably. Chris: Yeah, probably. Right. Kayla: But if I had to kill the. Chris: Person, then probably not. Kayla: I don't know.

1:00:52 Chris: It's tough to say. Kayla: Yeah. Chris: But I use that example because it's like, sort of an obvious. You take that bet and it uses like, a really big probability. It uses. Kayla: Right. Chris: It uses big probabilities in small numbers. Right. It uses a coin flip. It's 50%. Right, right. And it uses relatively small number of, oh, you only have to kill one person. So it's like an obvious. It's as obvious as you can get with like, yeah, sure, you take that bet as long as you're the type of person willing to sacrifice the needs of the one for the betterment of the many. Kayla: Right? Chris: But we're talking about things like, what would a super intelligent AI look like that we won't even invent for another 100 years? And probabilities with something like that start getting astronomically low, astronomically fast. Right. They're very what if? And they're very, like, tiny chance of things like this happening. And the less wrong folks know that, but the problem is, they still think the math is worthwhile. And I'm here to tell you it's not. Or maybe not that it's not worthwhile. It's just that it's not meaningful. The thing is, human beings are very bad at understanding and processing the consequences of extremely low probability events. Our brains just kind of go like, it's either high low or 50. And that's, like, about as much as we can really conceive of in terms of, like, making decisions and thinking about this stuff.

1:02:10 Chris: When you start saying, well, this outcome is so disastrous that even a tiny probability is worth thinking about, you are actually not correct. It's still not worth thinking about. By way of example, you have a non zero chance of going outside and getting hit by an asteroid. But that doesn't mean you should spend any time during your daily routine of looking through a telescope and making trajectory calculations. Kayla: You sure about that? Chris: 100%, yeah. Kayla: You seen Armageddon? Chris: I've seen Armageddon and deep impact. Kayla: I don't know. Chris: So, yes, I am sure about that. And just to illustrate, so one of the, it's, and this is semi well known now, but one of the examples that Eliezer Yudkowski has posted on less wrong is the question, is it better for some absurdly astronomical number of people to get a speck of dust in their eye or for one single person to be tortured for 50 years with no reprieve? The dust, right, maybe, but, like, they have literal, like, they debate about this, right? And his. He's positing that it's. It's better to have the single person get tortured. You're not. So here's the thing you're not hearing or reading, the types of numbers he's talking about. He's talking about numbers so astronomically high that our brains can't even conceive of them. Like, beyond Googleplex numbers. There's numbers?

1:03:28 Kayla: Well, there's not that many people. So what the fuck are you even talking about? Chris: Right, but that, well, don't ask me. Ask him. Kayla: Also. No. If Googleplex people got a speck of dust in their eye, it's still better than one person being tortured nonstop for 50 years. That's wrong. Chris: But according to a utilitarian, you can assign some utility to that. And even if it's the tiniest, tiniest inconvenience, if you multiply that by enough people, then it's worth the inconvenience of a single person times the inconvenience of being tortured. That's sociopathy, what utilitarianism is. And that's the danger of maybe only applying that in any context. Kayla: These people actually think that it's better for one person, not all of them. Chris: It's a debate on the site. Kayla: But if you're a utilitarian.

1:04:10 Chris: If you're utilitarian, then, yes, there is some number of people that you can say that would be true for. It's not 100. It's not a googolplex. But maybe for some near infinitum number of people, that might be true for. It's just math. Kayla: It's not mathematic. Chris: Okay, think about it. And some. Maybe this is easier. Is it worth one person being tortured for 50 years if every single human on earth had to have. Have to be tortured for one day? Kayla: I feel like that's a completely different thing. Like, a speck of dust in the eye is like. Chris: I mean, either way, you're talking about some detrimental effect happening to a person, right? Yes, but a speck of either pain and suffering or. It's, like, normal. Kayla: Yeah, but that's, like, normal day to day shit.

1:04:52 Chris: Right? And, see, what I think is happening right now is your brain is having a hard time making the leap from normal numbers to, like, meaningless numbers. Kayla: Maybe, but I don't feel like this is happening to me. I feel like that there's no number of people. I'm sorry to harp on this. I'm just saying I feel like there's no number of people getting a speck of dust in their eye. Chris: That's right. Kayla: That outweighs. Chris: Well, yes. Kayla: One person getting towards. Chris: So here's what I'm gonna say. This is how I think of it as there's no meaningful number of people that could have a dust speck of dust in their eye. To make that worth it. I think that you can talk on a website about the math of the utility of a dust speck versus the utility, or I guess I should say anti utility, of a dust spec versus the anti utility of torture, and you can do those calculations. But, and this is actually what the rest of this sort of, like, little section of the podcast is about whether that's meaningful or not.

1:05:50 Kayla: Gotcha. Chris: So let me finish these, a couple sentences here, and then come back to that. Basically, what I'm saying is, I'm not advocating here that I'm either, like, pro torture or pro spec, but rather, when the numbers get that absurdly large, or when probabilities get that absurdly small, human concerns like torture and specs and eyes are completely meaningless. Kayla: I'm pro spec. Chris: So here's. I know. So here's analogy. You could talk about using a tiny fraction of the energy emitted by a black hole to power your subaru. But by the time we've reached the scale of black holes, concerns about how you get your subaru to take you to work have long since disappeared in the proverbial rear view mirror. Proverbial rear view mirror. That is a tongue twister. Kayla: I am still pro spec.

1:06:30 Chris: That's fair. But whether you're pro spec or pro torture is, to me, is sort of beside the point, right? You can say you're pro spec. You can say you're pro torture. What I'm saying is, when the numbers get to the scale that they are, the meaningfulness of those numbers sort of goes out the window. And one of the ways I like to think about the scale problem is this, is that there is a difference between what is conceivable and what is meaningful, right? So human brains can conceive of numbers that are just beyond understanding. Chris: Like, actually, the number, the notation that Eliezer is using on his site when he talks about the dust spec, is using this, like, crazy mathematical notation that results in just numbers that are, that make Googleplex look like nothing, that make Googleplex look like just not even a dust spec, so to speak. And, I mean, I could even sit here right now and like, in the next 2 seconds, I could write a number that is bigger than the number of atoms in the universe or larger than the amount of seconds that have transpired. Like, that's, you know, it's easy for humans to conceive, to come up with these numbers and I could. Likewise, I could easily write a number right now that was smaller than the Planck length. Right, right.

1:07:44 Chris: But there's a difference between what I can conceive of and then what is in fact, meaningful for the Planck length. For example, I've heard scientists say that it's the smallest meaningful length I can conceive of. I can conceive of lengths smaller than that, but they don't have any physical meaning. It's the. The fabric of reality just doesn't support that level of fidelity. Kayla: Right. Chris: And then a similar thing, I think happens when you talk about numbers that are so large, such as in the dust spec example, is I can certainly conceive of numbers that large, and I can even do math on them and talk about utility. But I just don't think that they're meaningful because by the time you've gotten that large, things like the age of the universe or a googolplex or whatever are so far beneath that, you've already left meaning behind way before you even get to those numbers.

1:08:34 Kayla: Right. Chris: So that's kind of what I'm saying here, if that all makes sense. Now, if you take all that and talk about it on a website and play some mental math with a. You know, like, I think that post, to me, is perfectly fine. The problem is when you go from like, a fun conversation you have, you know, when you're high with your friends to being scared about an actual future artificial intelligence that may be applying some of these low probability, high impact sort of things, that's when it starts to become a problem. Because now you're worried about something that is essentially meaningless. Kayla: Right? Chris: Does that all make sense? Kayla: I think so. I'm still pro spec. Chris: But I know this whole time. This whole time I was saying that in your head you were just going prospek.

1:09:18 Kayla: Yeah, I definitely wasn't listening. I'm saying you haven't swayed me. Makes sense. Chris: I'm not trying to sway prospect. Kayla: I'm saying that neither prospect all the way, baby. Chris: I'm saying that neither pro spec nor pro torture have really any meaning because the question is meaningless to begin. Kayla: Well, we're going to settle this with a Twitter poll, so it's fine. Chris: Great. Actually, we definitely should. Any questions about utilitarianism? And even, like, probabilistic utilitarianism? Kayla: No. Chris: Okay. Kayla: Too big. Too much, too big. Chris: Monkey brain can't think. Yeah. So utilitarian theory taken to absurd scale is cool and all, but that's just one part of how folks on less wrong talk about decision theory. So let me talk here about one of the jargony bits, and trust me, I'm only going to talk about one of them. We can't talk about all of them because we just don't have the time. But this is something called an acausal trade. That's one of the words they like to throw, phrases. They like to throw around. Another one, by the way, the biggest one that gets thrown around with things like rokos, basilisk, and on other parts of the site. It's called timeless decision theory. And acausal trade is a concept within that theoretical framework.

1:10:28 Chris: I won't talk about that because that's even longer, but I will go into what a causal trait is because I think it's worth at least talk about one of their bits of jargon. Kayla: I do have a question. Chris: Yes? Kayla: Do you think that everyone on this site has watched the good place at this point? Chris: Yeah, probably. Kayla: Did they make it? Chris: Maybe. Kayla: Did the good place happen? I wouldn't be surprised Michael Schur found this website. Chris: I was literally just about to say. Kayla: Turned into the good place. Chris: I wouldn't be surprised if Michael Shearer was reading some posts on less wrong and was like, I'm gonna make a tv show and it's gonna be a primetime television show where all the characters are dead. So a causal trade, in other words, a trade that lacks a direct cause, therefore a causal. Think about your standard, everyday, normal person trade. You give me an apple, I give you an orange, or you give me money, and I give you a garage full of lularoe leggings or whatever, ten grand. That's a causal trade, right? That's like a direct, like, you give me x, I give you y. An A causal trade is a trade in which you give me something in an implicit exchange for a promised behavior or even an expected behavior or other consideration. It's essentially a promise or an expectation.

1:11:43 Chris: And the less wrong folks are very big on the use of promises or expectations as motivators in their decision theory theorizing. It's all very game theory esque. And in fact, here I'm going to use a board game example to illustrate the point a little better. I don't know how many of our listeners have, like, played a game of risk, but if you played any board game, no, I don't think my mom's played risk. But if you've ever played risk, or if you've ever played a game similar to it, where there's like a table full of people and they're all sort of like fighting each other to be the one on top, right? You probably know of the crazy revenge guy at the table. Chris: Everybody's playing, trying to win, and then there's the one guy where he's like, if you fucking attack me, I am gonna destroy you the whole rest of the game.

1:12:24 Kayla: Are you talking about me? Chris: So that's you at our table? Yeah. Kayla: That's literally what I did the one time we played Riz. Chris: Yeah, but there's always at least one revenge guy where they say, like, I swear to God, if you attack me, I'm just. I don't care if I lose at that point, my goal will be to take you down. Kayla: I think it's the best way to play the game. Chris: That is an acausal trade. So when I decide to not attack Kayla because I know she's a crazy pants and she's gonna fucking revenge me the whole rest of the game, you've made an acausal trade with me? Kayla: Hell, yeah. Chris: You have influenced my behavior based on me knowing you're going to behave in a certain way if I do it. I haven't said, hey, Kayla, let's. You know, if I help you attack Bob and then you help me attack Alice, then we're allies. It's not that. It's. I know that if I attack you're gonna go all crazy, and even to the detriment of your own ability to win the game. That's what you're gonna do. So it's like this weird thing, right? It's rational but irrational at the same time, depending on how you look at it.

1:13:22 Kayla: Yeah, that's one of the best. Chris: It's irrational because. Well, of course she wouldn't do that because then she's gonna lose the game. That's stupid. Kayla: Don't care. Chris: But by making that promise, you have essentially insured yourself against attacks from some people who are worried about that kind of thing. Kayla: But you gotta be willing to follow through, right? Chris: That's the thing. You have to be willing to follow through. Kayla: Otherwise it's empty threats. Chris: And it's. Otherwise it's empty threats. So some people aren't. It depends. But that's exactly what they're using to conceive of Roko's basilisk. Kayla: Okay? Chris: It's an acausal trade in that exact same manner, except it's in reverse. Right. The robot, the future Roko basilisk. AI is saying, I promise you that you will get tortured unless you help me become existing. So it's the same type of thing. It just happens in reverse times instead of forward in time, but it's the same concept.

1:14:12 Kayla: So don't play risk with Rocco, if that makes sense. Chris: Well, don't play risk with Roko's Basilisk. But we also know that when we play risk with the crazy revenge guy, sometimes the crazy revenge guys all talk, and sometimes they actually decide they want to win the game instead of just take revenge on somebody from, you know, what they promised they were going to do. Kayla: Those people are disgraces. Chris: You better hope that Roko's basilisk is. Kayla: Like that because part of Buddy's a disgrace, if that's what you. Chris: Part of the thing that debunks what potentially could happen from an AI like that is the fact that they will probably have much better things to do than torture somebody from the past. They're actually what they're really worried about, unless wrong isn't even torturing you, it's torturing a simulation of you that they think of as you.

1:14:55 Kayla: Right. Chris: So basically what we're saying is this future hyperintelligence that is essentially godlike and inscrutable, but we can totally screwed it, which is a word, right? Kayla: Yeah. Chris: If it's inscrutable and we screwed it. Yeah. Kayla: C r U T, screwed. Chris: We're saying it's inscrutable. Except I totally know how it's going to behave. Kayla: Right. Chris: And it's going to behave in this way, and it's going to take valuable resources that a God will totally care about doing and create a simulation of little old me and torture. That simulation of little ol me. Why would it do that? Kayla: I mean, my argument, it has a. Chris: Risk game to win. Kayla: My argument would be that if Rocco's basilisk is as godlike and powerful as we say, then it doesn't expend any resources or energy torturing you.

1:15:45 Chris: Well, I mean, it's just omnipotent, and. Kayla: It can and all powerful and like torturing you. But making good on that promise is not taking away from its precious time to go solve risk, global galactic risk. That's my fear. Chris: Okay. But omnipotent isn't actually a thing. Like, at some point, it does actually have to spend something akin to resources or at least time or cycles or something. Kayla: Does it? Chris: I don't know. But one of the things, one of the analogies that was given in rational wiki, which I'll cite in my sources later, was what we're doing right now is akin to two ants trying to figure out how a human brain is gonna work.

1:16:26 Kayla: True. Chris: I don't know. Like maybe it's gonna smush us ants or I probably won't care. Like right now, I don't really care whether ants that existed a million years ago were working towards evolving humans. Kayla: True. Chris: I'm not going around looking, where are those ants? I gotta torture me some ants. Cause I. They didn't try to make me be alive, so. Kayla: Well, that's also putting the ants in a position that they're not in. The ants are not sentient, and the ants don't have any capability of speeding up evolution. Chris: How do you know that we have any capability of creating something like Roko's basilisk? Kayla: Because Roko's basilisk has come back in time and fucking told us by posting this fucking thing. Just. Chris: You are a true believer. Kayla: I am in the cult.

1:17:08 Chris: So going back, do you understand a causal trade now? Kayla: Yeah. Chris: Okay. Kayla: Mostly because I'm the risk vengeance. Chris: Because you are the vengeance person that uses an acausal trade to get people to not attack you. Kayla: Yeah. Chris: So as you pointed out, though, acausal trades are not binding. And any future basilisk style AI would know this. And here's the thing. It doesn't matter if they even have. Like you said, even if they did have infinite resources torture you anyway, the fact that they know a causal trades aren't binding means that suddenly the blackmail is not effective, and therefore it breaks the acausal trade, and therefore there's no reason torture you.

1:17:50 Kayla: So the basilisk goes. They know that this is not binding, so therefore this whole thing is illogical. Put a line through it. It doesn't work, doesn't make sense. So I might as well. Nothing do this right? Chris: I told you, it's a bunch of. They know I know you. Kayla: I'm just saying I need to think on that for a little bit before I feel comfortable. Chris: All right, keep thinking on that with your aunt brain. Compared to the Rococo's human brain, that's insulting. It's true, though. So we've talked about the limitations of utilitarianism as being a tool and not a universal framework for looking at the world. Kayla: Did you just call Roko's basilisk a tool? Chris: Talked about how absurdly scaled numbers and probabilities quickly get you into the realm of meaninglessness. And we've talked about a causal trade being non binding. Although, to be honest, I mostly told you the acausal trade story to illustrate some of the ritual that is involved with this group with all of their jargons. Now, I want to reference another episode of ours.

1:18:46 Kayla: Oh, shit. Chris: In fact, our fourth episode of season one in which we talked about. Do you remember fourth episode, the game. Pop quiz, hotshot. No. Oh, wait. Yeah, the game. Sure. Yes. Sorry. That was the name of the episode. Star Citizen. Kayla: Yeah. Oh, oh, okay. I understand why you got confused for a second. Chris: Yes. You were talking about the game that we've already referenced in this episode. Kayla: Oh, my God. That episode. Do you realize what that episode is? That episodes called the game. Chris: Yeah, I know. Kayla: If you. Chris: Oh, no. Every time you listen that episode, you lose the game. Yeah, sorry, you guys. Kayla: Sorry, guys. Chris: So we talked about Star Citizen in that episode, and we also invoked good old Pascal's wager as a lens to look at what was going on. Can you remind our dear listeners, absolutely not. About what Pascal's wager is?

1:19:33 Kayla: I have no idea what you're talking about. Chris: You don't remember Pascal's wager? It was, like a big part of that episode. Kayla: I have absolutely no idea what you're. I don't think we talked about this thing that you're. Chris: This is a terrible. A causal trade. Kayla: I'm sorry. I don't know what Pascal's wager. Chris: I should have promised torture you in the future. Okay, so Pascal's wager is the thing that goes like this. I might as well believe in God, because if I decide not to believe in God, right, and there's no heaven and no God, then doesn't matter. I wasn't going to heaven whether I believe in him or not, right? If I decide to believe in God and there's no heaven, then I don't go to heaven. But if I decide to believe in God and there is a heaven, then I do go there. When I die, I'm so believing in God strictly. And this is actually also a game theory, sort of like matrix, right?

1:20:22 Kayla: Right. Chris: The believe in God choice dominates, right. Because if you. If there's no God, no heaven, then it doesn't matter either way. But if there is, then you better believe in it, because otherwise you're going to hell or not going to heaven or whatever. Kayla: That's Pascal's wager, and it is faulty. Chris: Right? So with Star citizen, the version of Pascal's wager was, I better believe in the thing. The thing being Star Citizen becoming a real game. Because if the game doesn't happen, then no big deal. But if it does, then I get to play this awesome game that I've always wanted to play. Right? That was the version for Star Citizen. Kayla: Right. Chris: But Star Citizen, viewed as this future entity that you might as well believe in, took it a step further, fans were and still are attempting to bring the entity into existence by donating money and resources, and a lot of it. Belief in a future entity because of the potential consequences coupled with a desire to bring it into existence. Does that sound familiar?

1:21:19 Kayla: Are you saying Star Citizen is Roko's basilisk? Chris: Sort of. It's Roko's basilisk with a carrot instead of a stick. Kayla: Okay, so basically what you're telling me is that we should give money to Star Citizens so that we avoid being tortured for eternity. Chris: Well, no, but that's the thing, right? Is that Star Citizen is carrot versus stick. Kayla: So I don't know that it is. Chris: One of the things with. So Star Citizen is like, I might as well donate because there's a nice little juicy reward for me at the end if I win. Same thing. Sort of with Pascal's wager, right? There's a juicy heaven reward. The sort of shitty thing about Roko's basilisk is that it's only conceived of with stick and not carrot. It's only conceived of being tortured. There's no version of it where if I donate to this, then the basilisk will have known that I was going to do that and therefore reward me because I donated to it. There's no version of it where that happens.

1:22:12 Kayla: Okay. Chris: Which there's no reason for that not to exist, because carrots and sticks are both motivators. Kayla: What happens? Chris: Kind of weird. Kayla: If you reach the end of your life and as you're dying and you die, and then you find out I that if you are somebody who helped star citizen happen, you get eternal salvation. Chris: Wow. Kayla: You're somebody who didn't. Eternal damnation. Chris: Then I would definitely donate to Star Citizen. Kayla: So that's what I'm saying. Just because it looks like it's carrot doesn't mean there ain't no stick. Chris: I guess I'm not really seeing the tie in from making a game exist to eternal life in heaven.

1:22:53 Kayla: Cause it's Roko's basilisk. Chris: Yeah, but only in the sense that you are trying to make a game exist. Yeah, but not in the sense that you are going to heaven. Kayla: It's under the guise of a game. It's not actually a game. Chris: I see. So you're, like, just way extrapolating. Kayla: I'm just saying Roko's basilisk disguised itself as a video game to get people to make it happen. Chris: Got it. Kayla: And now if you don't, so you're unsure, contribute to Star Citizen eternal damnation. Chris: Yeah, I think you got exactly what were trying. Yeah, that was. That is the correct plot line of this whole thing. Kayla: Yeah. Chris: Right. So actually, we are not the only ones to draw this parallel. Kayla: Oh, shit.

1:23:34 Chris: It's been. Well, okay, so hold on. Let me be clear. The parallel between Pascal's wager and Roco's basilisk, not between Star Citizen. Kayla: I bet you were not the first to do that either. Chris: Probably. So it's actually been discussed quite a bit and over much more than just Roko's basilisk. Apparently, there's enough fretting over future super intelligent AI that Pascal's wager has come up a lot. In fact, here's a quote that I'm citing from rational wiki, again, that they in turn are citing from a writer by the name of Greg Egan. Quote. You know what they say. The modern version of Pascal's wager is sucking up to as many transhumanists as possible, just in case one of them turns into goddess. Kayla: Oh, okay. So I need to start doing that. Okay.

1:24:16 Chris: I love that's your reaction and not like, oh, yeah, snarky. That's funny. Kayla: Oh, no, that's a good advice. It's solid advice. Chris: Yeah. So you should be tweeting, I love youlanmusk. Kayla: I think, really, this is just about who has certain kinds of anxiety and who doesn't. Chris: Yes, that's literally what this is. Kayla: It's just about, like, what's wrong with your brain? Like, if this pokes your anxiety, then you're a certain. Chris: You experience a certain type of anxiety. Yes. Kayla: All right, well, aren't you just so. Chris: Lucky, dude, I get super anxious about. Kayla: Not about this. Chris: Yeah, because I read all these things, which I am now sharing with you. So just in case it isn't obvious as to how the pascal's wager fallacy applies to this sleek, modern robo God, this AI that we have never seen but a subset of humans thinks exists. Well, the exact same way it applied to the crusty, old school, Judeo Christian God that we also have never seen but a subset of humans thinks exists. And that is. Yeah, sure, maybe the future basilisk AI wants current you to spend your time and money helping it exist. But what about the other future AI, whose only goal was to make sure the basilisk AI didn't exist?

1:25:26 Kayla: Stop. No. Chris: Oh, shit. Now you've pissed that one off. Oops. Kayla: Well, now I have to worry about that now, too. How am I gonna please both of them? Chris: So this is the same thing that shoots down the original Pascal's wager, which is that there's a prerequisite that the God that you're wagering on is both alone and omnipotent. It's the classical logical fallacy of assuming the thing you're trying to prove and the nature of these Pascal's wagers and how they motivate people to spend money on them. They've been referred to by one economist as Pascal's scams. And. And there's even a trope on lessrong.com itself that calls things like Roko's basilisk Pascal's mugging. Kayla: Jeez. Chris: Which I just love that term so much, I can't even tell you. But what that really demonstrates, by the way, is that not everyone on less wrong is afraid of the basilisk. Like, there's people that are like, yo, this is just making people upset and, like, goading them into donating money to something that they really shouldn't have to do.

1:26:22 Kayla: Wait, are you telling me that people donate? Chris: Oh, yeah, we'll get to that. Kayla: Wait, no, hold on. Hold on. Off podcast? Are you serious right now? Chris: Why off podcast? Kayla: Because I need to know otherwise, my kayla. Chris: That's the whole thing. The whole thing is, if you don't donate to the basket, if you don't help. Kayla: I didn't know they're donating money. Chris: Well, how else do you think it was? Kayla: I was about to cry. This is really upsetting. Chris: How else did you think you are supposed to help? This anime exist? Kayla: No. Chris: Who don't need to. Kayla: Who? Chris: We'll get to that. Kayla: I don't like this. Chris: What? Cause people are actually, like, taking it to heart and acting on it. Yeah, it's shitty, so we'll get to that. Kayla: Oh, my God.

1:27:03 Chris: But, yes, there's a trope on lesswrong.com called Pascal's mugging, which I just love. But anyway, that's the whole thing with Pascal's wager is the many gods argument. Sort of shoots it down. It's like, okay, like, sure, if there was only one God, then that would be a true wager. But what if, you know, you were betting on the wrong God and the right God was like, oh, if you believe in that one, you'll actually go to hell. Kayla: This is stressing me out. That's just tell me where to give the money. Chris: But you knew that. You knew that. That's. That's the thing that's stressing you out. Kayla: Is I just need. Chris: You could have been giving money. Kayla: Yes. Chris: Actually, I'm not gonna tell you now because I really. Kayla: Okay, well, it's too. It's too late. Chris: Holy shit. All right, so on that it does seem to be an issue. For example, there is an institute now called Machine Intelligence Research institute, formerly called SIAI, which we have mentioned just an hour or two ago on the show, which stands for singularity Institute for Artificial intelligence that takes donations and takes them for the cause of reducing existential risk from the development of AI. Now, this is all well and good.

1:28:07 Kayla: Like, that's not specifically Roko's basilisk, but. Chris: I can see where this starts to sound like a pascal's mugging. Kayla: Right? Chris: If you're positing that there's an existential humanity wide risk from something that may or may not happen in the future and in fact has a terribly low probability, you're back in this territory of infinitesimal small probabilities times disastrously large, consequences equaling, oh, I don't know, about, like, $20, give or take. But again, with probabilities that small and consequences that large, the math doesn't quite work out that way. Kayla: Right? Chris: So this is why I brought up the thing about the speck of dust times the whatever, right? Because it's a fun thing to think about. It's a fun thought experiment. But if you actually do that math and say, okay, yeah, it's about $8, that's when it starts being like, no, you took something that was, like, conceivable but meaningless and tried to assign meaning to it, and that's where you broke.

1:28:58 Kayla: I'm still pro spec. Chris: I understand that. We'll just call this episode pro spec. Kayla: Because that's no matter the number. Chris: I know. Kayla: Also, tell me where to put the money. Is it to this people? Chris: Anyway, that's why I brought that up. Right. And yet, not only did Siai say that type of math was possible, but they actually did the math, and they would tell you exactly how many lives you were saving per dollar donated. And I think it came out to eight lives per dollar donated. They actually did that math and said, every dollar you donate to SiAi saves eight lives. Now, again, I'm very upset right now. Remember, this isn't a calculation based on how many malaria nets they can ship overseas, which is like, okay, this times this many equals this many lives saved. Right? This is based on. Well, there's a 0.001% chance that a world destroying AI will be developed. So if we can reduce that chance by an even tinier 0.001%, then let's see. Carry the one. Yeah. About eight lives per dollar.

1:29:57 Kayla: This is the most upsetting thing. Chris: And they took people's money. And like all good cults, I don't actually think they were scamming. I think they really believe this math. For example, here's a commenter on less wrong talking about how he's doing his part, and this is part of a much larger comment that he made, but just quoting a little bit of it. What am I doing? Working at a regular job as a c programmer and donating as much as possible to Siaihdeene and sometimes doing other useful things in my spare time. Blah blah, he continues on. End quote. By the way, Sia's founder was none other than less wrong's founder, Elie Iser Yudkowski.

1:30:38 Kayla: Now no, I'm done. I can't do test anymore. Chris: I realize I'm dropping that knowledge here in a way that makes it seem really dramatic, and that's intentional because I aim to entertain. But I will say I honestly do not think that Eliezer is a scam artist. I don't. If he were, I don't think he would have tried to shut down the roco's basilisk post so forcefully. Conspiracy theory about Streisand effect aside, I do think Mister Yudkowski, from what I've read, is genuinely concerned about mitigating existential risk that comes from AI research. Kayla: Okay, but what is the money doing? Chris: Here's what Wikipedia has to say about what Siai, now known as Miri, M I r I. Miri's approach to identifying and managing the risks of AI, led by Yudkowski, primarily addresses how to design friendly AI, covering both the initial design of AI systems and the creation of mechanisms to ensure that evolving AI systems remain friendly, Miri researchers advocate early safety work as a precautionary measure. However, Miri researchers have expressed skepticism about the views of singularity advocates like Ray Kurzweil, who we both like that superintelligence is just around the corner. Miri has funded forecasting work through an initiative called AI impacts, which studies historical instances of discontinuous technological change. So that means, like, big jumps in technology and has developed new measures of the relative computational power of humans and computer hardware.

1:32:04 Chris: Miri aligns itself with the principles and objectives of the effective altruism movement that. Kayla: Helps me a lot that like, just really contextualizes it and like, makes me go like, okay, so this isn't just people taking money. Chris: No, no, no. And driving away. That's what I'm saying here. Specifically, I'm saying, like, I dropped the whole oh, his name to be dramatic, but he's not. He's not an MLM leader. He's not. Right doing. He's not stealing people's money. Kayla: He is doing, like. That's actual good work. Chris: Yes. Kayla: That is important. Chris: They do good work. It's just that there are some people who fell prey to the anxiety thing that you were talking about because of this is essentially a home for the type of people that would have these kind of fears. Otherwise they wouldn't be talking about it and working on it.

1:32:52 Kayla: Right? Chris: And then they kind of fall into that trap and then feel like they have to donate money. Because if I don't donate everything I can, then I may end up being tortured by a future AI. Kayla: Right? Chris: So maybe it was a little irresponsible of me, actually, to, like, drop his name like that, because I genuinely think he's doing good stuff. I just think that there's. Yeah, that it's. It's just. It's just the situation, right? It's a situation in the community. And I also don't think that the stray sand effect thing that he did was. Was super useful. But we all make mistakes. Kayla: Like you right now. Yeah, I'm kidding. I don't think you make a mistake. Chris: So I want to go back to Pascal's wager real quick. Specifically the God part. Maybe this goes without saying, but we here at cult are just weird, are also far from alone and speculating on the singularity and future godlike Super AI's as being sort of a new techno religion. In fact, I think singulitarians are even on our like to do list if maybe we might do this on the show one day, right? Which I guess is like, actually kind of what this episode is.

1:33:51 Kayla: Right? Chris: But anyway, for our listeners who are unsure or unaware, technological singularity is just a fancy word to talk about how pretty soon we may have super intelligent AI's that will transform, revolutionize our very society and our very existence, and may involve things like living a really long time, living forever, uploading our brains, our consciousness, into computers, that sort of thing. And it also has its own gods and its own gods. Are these super intelligent AI that. And so the reason that this is, like, if you're wondering, like, why are super intelligent AI's, like, going from just my shitty, stupid computer to God like beings? The primary way that's discussed is that once computer intelligence, once machine intelligence reaches a certain threshold of both, like, complexity, like general complexity, and also power, that it may itself want to design an additional super intelligence.

1:34:51 Chris: And being that machine intelligence is smarter than us. It will do a better job than us of creating a new intelligence, which then itself will do a better job than its robot creator, of creating a new intelligence, and so on and so on. And it will just spiral until before we even notice. Because of that cyclical effect, we'll have some super intelligent AI that is just so far beyond understanding that we're like ants to it. Kayla: Just go watch the movie. Her. Chris: Yeah, actually. Yeah, yeah, go watch her. It's an excellent movie. Kayla: Best depiction of the singularity. Chris: Yes, actually, so far I've seen. Yeah, and we, you and I are both, like, huge fans, and we both, like Ray Kurzweil, and Ray Kurzweil is a futurist. Future futurist. Futurist. Yeah, I think that's the name. And we've read some of his stuff, and he's, like, a very positive oriented optimist about the singularity. He actually works at Google now. I think his title is futurist there, I think. Yeah, yeah, he does. So that's sort of explaining, like, how the singularity of super intelligence would allegedly come to be. And it's been playfully referred to as the rapture for nerds. And it's very true, which is very true because it's sort of, if you think about it involves these, like, godlike beings. It involves eternal life. It involves a lot of these things that we sort of maybe want to be true or make us feel.

1:36:13 Chris: Feel safe and or require some article of faith to believe in. And it's just using technology instead of, like, old school religion stuff. So the Roko's basilisk shenanigans kind of does a good job of illustrating that whole point about some angry, inscrutable God. Kayla: Are you gonna send this to Elon Musk? Chris: What? This. This podcast? Yeah, I'll definitely tweet it at him. Man, we'll get so many f cking listeners. Holy shit. Shit. One of my favorite articles I read while researching this episode was entitled sinners in the hands of angry artificial intelligence. And I highly recommend it. It was such a good article. It's on a blog called orbitermag.com and orbiter mag, which I had never heard of. Apparently, it's like a blog that talks about the meaningfulness of technological type things, which is, I think we sorely need things like that. But anyway, the title of that article is clearly referencing the classic historical Jonathan Edwards Sermon, sinners in the hands of angry God. And the article itself, of course, draws parallels to the Rocos Basilis thing to that famous sermon.

1:37:23 Kayla: Right? Chris: The article goes on to talk about how religious awakenings of the past have typically followed some pivotal technological change, such as the Protestant Reformation coming in the wake of the printing press, or Jonathan Edwards own era of religious awakening in America coming on the heels of things like the train and the Telegraph. Which then begs the question, why should the recent advances of computer Internet personal devices be any different? Why shouldn't we also be experiencing some sort of religious awakening, which maybe this is sort of a part of? Kayla: Mmm, I like that. Chris: So here's a great quote from the end of the article, which again, you should definitely go read the whole thing on orbiter mag because it's really good. And I really like orbiter mag as a blog. Now here's the quote. Religion is implicit in such technological change. It behooves us not to argue for its elimination, but to render more sophisticated forms of understanding it. As much as Roco's basilisk as a concept deserves disdain, the reality is that self aware artificial intelligence, whether malevolent, benevolent, or something else, will surely be developed sooner rather than later. Such a development requires a religious response. End quote.

1:38:35 Kayla: Damn. Chris: And I think they're absolutely right. Maybe not a catholic or jewish or muslim response. That's not what they're talking about. But they're talking about a response that involves things like metaphysics and morality and ethics and that sort of stuff. Kayla: And reverence. Chris: And reverence. Absolutely. So it's fascinating. Right? And I'd also like to add here that I'm not even sure that Roko's basilisk deserves disdain. The folks on less wrong are very smart people. They include AI researchers, coders, developers that are ten times smarter than I am. And I think they actually came up with a really interesting thought experiment. It's just an unfortunate side effect that sometimes digging around for truth isn't always safe for the miners. There can be hazards in mining each other's thoughts for insights and information, just as surely as there can be hazards in mining the earth for diamonds. I just hope that most of them are okay now.

1:39:26 Kayla: Me too. Chris: And speaking of hazard, I said we'd get to that about something called information hazard quite a long time ago on the show now. And here we finally are. Although actually we've been talking about an information hazard this whole time. That's what Roko's basilisk is. Remember we mentioned it's not so much the AI that is the problem, it's the thought itself. Kayla: Right? Chris: And don't forget that's something that Eliezer was saying about why he was so upset about it too. So he understands, I think, like, what the issue is. Kayla: A single thought. Chris: Sorry, a single thought, and you can be paralyzed into donating all of your money to research artificial intelligence. So we've been talking about information hazard this whole time. But I really wanted to mention a fascinating paper I read for this episode by one Nick Bostrom, who you may have heard of. He's a relatively famous philosopher at Oxford. Here's his Wikipedia blurb. Nick Bostrom is a swedish philosopher at the University of Oxford known for his work on existential risk, the anthropic principle, human enhancement ethics, superintelligence, risks, and the reversal test. I don't know what that last one is.

1:40:32 Kayla: Fuck is the reversal test. Chris: I don't know. Mister. Bostrom's paper defines a and information hazard as, quote, information hazards are risks that arise from the dissemination or the potential dissemination of true information that may cause harm or enable some agent to cause harm. Such hazards are often subtler than direct physical threats and as a consequence, are easily overlooked, end quote. This is like the most interesting topic to me because one of the key things about information hazard that he makes a point that he kind of mentions there and really makes a point of in the paper, is that information can be free and true and yet still hazardous, still harmful.

1:41:13 Kayla: Do you remember the whole point of the first book of the Expanse? Chris: Yeah, that's actually. That's totally information hazard. It's about like the push and pull between. For those of you who haven't read the Expanse, there's this guy who has this sort of. This virtue, which actually I was about to write about here, we're about to talk about here, but we assume that free and open information is purely virtuous and always good. And there's a character in the expanse that has the same ethic, and we sort of identify with that. But then the book and actually the rest of the series even sort of like, calls that into question. Like, is it always good to have all information be as free and as open as possible?

1:41:56 Kayla: Fucked up? Chris: Yeah, it's super interesting, and it's very counterintuitive, which I'll talk about some examples that maybe make it less counterintuitive here. I'll definitely link the paper in the show notes because I cannot recommend it enough. It's so interesting. And he actually categorizes and puts into sort of like a taxonomy all sorts of different types of information hazards. So here's a short list. This is like half of them data hazard, idea hazard, attention hazardous signaling hazard, evocation hazard, enemy hazard. Kayla: What? Chris: Knowing too much hazard. Kayla: That's me. Chris: Information asymmetry hazard, unveiling hazard, ideological hazard, distraction and temptation hazard. That's me. And again, that's only like half. So if you read the paper, you'll see it's also one of those interesting things where you kind of go like, oh, yeah, I think I knew about that. That actually does kind of make sense. So, for example, I mentioned enemy hazard above, which is basically a thing where like, we as a country have military secrets. Because if we allowed, for example, the design of the f 22 raptor to become common knowledge, it would put our national security at risk, our military enemies. And then it's not quite enemy hazard, but like, while we're talking about military stuff, you could, and I think he maybe did in the paper, posit that knowledge of how to build a nuclear device is information hazard.

1:43:14 Chris: Because again, if it were free and common knowledge, we would all be at more risk. And that's why, for example, we can't just make a YouTube video about how to refine uranium. You will rightly get in huge trouble with the us government. And I know this because a youtuber that I like mentioned it on his channel that why he didn't make a video about that thing ever. It's like this chemistry dude, Nile Red. Nile Red. Yeah. He recently had a video where he worked with some uranium compound and he was like, I thought about doing a uranium refining thing, but then I realized I would have gotten in huge trouble with the government, so I didn't. But again, the reason for that is because the government is trying to protect against this information Hazard. Chris: Anyway, this information hazard paper is now referenced on Les Wrong's very own article about Roko's basilisk. That's actually where I found the paper. So Les wrong for a long time had censored all whatever, but now they have their own article on it because obviously it's like so famous now. It's just for a long time, Eliezer would only write about it on a Reddit forum about this type of stuff, but now they have their own article on it.

1:44:17 Kayla: Good. Chris: But the reason they have it there is this paper is because ultimately that's what the basilisk is. It's a piece of information, a little memetic virus that a particular portion of the population is vulnerable to and it is hazardous to their mental health. Me and information hazards are at large. What less wrong is kind of all about they and the siAi. Now, Miri have the goal of reducing existential risk from information hazards involving AI. I think it's generally a good mission, even though it may kick off some mental trauma and poor life choices from time to time amongst their community. Kayla: Right? Chris: So real quick, before we get to the criteria and judgment, just like I did in my first episode of the season, I'm doing this thing where I'm throwing sources in at the end. I'm not sure if I'll keep doing it this way. I guess it depends on like our listener feedback and yada, but it just, it feels a little bit easier for me somehow to squeeze them in at the end versus the beginning. But like I said, feedback pending. So here, I think, all of the places I went for research on today's topic. First of all, lessrong.com quote s article on Rokos Basilisk. I also reread that old Slate article, Roko's Basilisk, the most terrifying thought experiment of all time.

1:45:31 Kayla: I still read Slate every day. Chris: Wikipedia's articles on less wrong on Eliezerz Yudkowski, Business Insider's article, so aptly titled for their magazine, what is Roko's Basilisk? Kayla: Oh, I like that. Chris: And then there was a vice article with maybe even better title explaining Roko's basilisk, the thought experiment that brought Elon Musk and Grimes together. Because that's to vice. That's the, that's the headline for Roko's best. Kayla: Yeah, literally, a new life has been created. Chris: Oh shit. Kayla: Because of this. Chris: It begins. I bet it's more than one. I bet people are having like, scary, oh, sure, anxiety sex.

1:46:15 Kayla: Specifically, Grimes and Elon Musk are having a child. Yes, because they methadore on Twitter. Because of this. Chris: I know. And there's also an article out there called this horrifying AI thought experiment. Got Elon Musk a date. That's on livescience.com, and that seems to be a common theme. Knowyourmeme.com has a whole article on Roko's basilisk because it's a very memetic thing. Obviously. There's a couple videos I watched on YouTube, Orbiter mag, which we mentioned. That article centers in the hands of angry artificial intelligence. And then there's another really good blog out there called last word on nothing, with an article entitled who's afraid of Rocco's Basilisk? Me, of course. I also used that. I read that whole paper by Nick Bostrom because it was so fascinating. It was one of the first things I read, because the very first thing I read was actually the less wrong article about their own thing about Rokovascalisk.

1:47:08 Chris: And then the second thing I read was the Nick Bostrom paper. And that's when I was like, oh, we have to do an episode on this. It was that interesting. And then finally, I made heavy use of rationalwiki.org article on Roko's Basilisk. And I highly recommend, it was a great article. They have a great article on Roko's Basilisk. It really details out in a much better way than we did even on the show here, where it came from, the nature of less wrong. They even go into a little more detail about, like, timeless decision theory and a causal trade and some of the, like, really esoteric stuff that sort of generated this thing. Chris: And then they also have this whole section called so you're worrying about the basilisk, and it's actually all about, like, here are the different reasons that you shouldn't worry about the basilisk. It's actually really extensive. It's almost like half the article. And I, when I was talking about the reasons you shouldn't be afraid of the basilisk, like, a good chunk of it sort of was inspired or framework or came from that part of their article. And then finally, at the very end of that section, there's a little header that says something like, I know it's rubbish, but I'm still worried. And they're basically like, well, you should probably talk to a therapist then, because it's, you know, they're like, exist. Well, no, but they're like, they're not being like, that's the thing is like rationalism.

1:48:28 Kayla: I'm going to my therapist tomorrow and I'm like, should I talk about Roku's basilisk? I don't know. Chris: Rational wiki is typically very cheeky in their articles, but this section was part of what made me kind of go like, all right, good job, you guys. The fact that they made that call to mental health, and they even said, if you are having trouble with existential sort of fears, like, that's something that maybe, like, a university campus mental health professional may have more experience with. So anyway, rational wikis article on Roku's basilisk. Very good. All right, it's finally time for. Kayla: The. Chris: Criteria and the dodgmandhead. So let's do this. Okay, we're gonna do that. We're gonna do this for like ten minutes.

1:49:18 Kayla: Number one, Jesus Christ expected harm towards the individual, socially, physically, financially, etc. Chris: Mm. Kayla: Number two, population of cult. That's not what it is anymore. What do we call it now, is it niche? Chris: Is it niche? And we are never gonna rewrite that apparently. Kayla: And, well, because it's whatever. Three, anti factuality, closed logical system. Four. Percentage of life consumed. Not empirically, but how much of your time is devoted to the thing? Ritual is number five and number six, charismatic leader. And for some reason I don't think that this is accurate anymore. When we originally conceived of this, we put alive leaders cult of personality. That's definitely not how we've addressed it.

1:50:11 Chris: Yeah, I don't know why we wrote that. That's literally years ago now. So those are our criterias. Kayla: Well, I don't. What's the cult? Is it the. Chris: I think it's the less community. I think it's the less wrong. Less wrong slash, siai or miri. Kayla: Now expected harm. I feel like it's high. Chris: Yeah. I mean, so there was a. Kayla: The information hazard is high. Chris: Yeah, the information hazard is high. There was a poll, I think some internal poll that they ran recently ish, a few years ago, at least not. Not back in 2010 or whenever when this was happening. But then it was something like, I think between ten and 15% of people were like still worried about it on the site, which is like pretty high.

1:50:54 Kayla: Pretty high. Chris: When you think about how like, freaking weird it is, it's like one or. Kayla: More out of every hundred. Chris: It's one out of every ten. Kayla: No. It's one out of every hundred. Chris: No. See, this is what I was talking about earlier in the episode when I. Kayla: Was saying Kayla is bad. Chris: People can't conceive of probabilities. Kayla: Fuck off. I said the wrong thing. Chris: It's one and a half out of ten. Kayla: That's a lot. Chris: Yeah. Kayla: So I'm saying it's fairly high. Chris: Yeah. I would say since we're saying expected harm, good job passed using the word expected, I would say it's fairly. Kayla: High population of cult. Is it niche? Chris: Oh. The other thing about the harm thing too is that it's not just mentally anguishing. It's also people are like spending their real hard financial draining.

1:51:38 Kayla: Yeah. Is it niche? I mean, it seems fairly. This particular community seems fairly niche. I would say given the extensive writing that's been done about Roco's basilisk on the Internet, that concept is not niche, but this community and the like, really hardcore, like philosophical, ethical, whatever kind of thinking that they do is a niche population. Chris: Yeah, totally agree. I think that since we're talking about the community as the cult, I would say it's definitely niche. Even though this one thing happened to go, like, mega viral. Kayla: Right. This one, I. This one is tough. Chris: What's this one? Kayla: Anti factuality. Chris: Mmm. Kayla: Because it's like, yes and no. First of all, anti factuality. They want you to believe that it is better for a bunch of people to not get a speck of dust in their eye and just torture.

1:52:31 Chris: It's all about the spec. It's all about the spec. Kayla: Just saying feels antifactual to me. But I think in terms of. That was a joke. In terms of, like, having a closed logical system. I don't think that exists here, because it seems like these people almost, like, take the logical to the opposite extreme, where it's almost hazardous, with following the logic. So to the letter. Chris: So that's the exact line of reasoning that. Now I forget the article because I read so many of them. But there was one article that follows that exact line of reasoning that was basically, like, being irrational can be harmful, but actually, like, being way too rational can also be harmful. Yeah. Basically to that extreme.

1:53:15 Kayla: Which kind of makes me go, like, is it kind of anti factual? Because it's like, when you get. No, I'm serious. I know this. It rolled over pedantic. But when you get this literal. Chris: Yeah. Kayla: You are kind of not existing in real world logic anymore. You're kind of not existing in, like, experience. You're not existing in, like, real. It's like, okay, the television show, Nathan, for you, the way that those writers craft the, like, business proposals is they get. And this. I know this actually, because you worked there, they get into a. An extremely logical frame of mind, and they try to be so logical that it's insane and silly.

1:54:01 Chris: Right. Yeah. And that's a part of what makes that show so genius. Kayla: Yeah. Chris: And that this definitely feels like a similar sort of thing. It's, like, so logical that it's. Kayla: It's a different kind of antifactuality. Chris: Yeah, it's. Yeah, it's weird, right. I think it goes back a little bit to what were saying about, like, conceivable versus meaningful. Kayla: Right. Chris: The logic is conceived so perfectly and so detailed and esoterically that the meaningfulness just, like, totally goes bunk. The same way it does with something like, Nathan, for you. Except it's funny instead of terrifying. Kayla: But I also don't want to insinuate that the actual thought that these people are doing on a day to day, even in the community, is anti factual. I think that we talked about some extremes where it is. So I'm gonna say it's yes and no. And, like, maybe more of an emphasis on the no with, like, a high probability of yes.

1:54:52 Chris: So it's like a Schrodinger's criteria for yes. It's both anti factual and factual at the same time? Kayla: I think so. Chris: Cool. Whoa. Gotta bring it back to quantum mechanics, too. We gotta talk. Every episode, we have to reference at least, like, three or four other episodes a single thought. It's all about the throwback, man. Kayla: Percentage of life consumed. I'm gonna say it feels not that high to me. Like, I know you talked about that one guy who, like. Well, basically all he does is work and give money. Chris: Yeah. Kayla: But outside of that one example that was given, it's people sitting on a forum. And unless I'm prepared to say that I'm in a cult of Twitter, I don't think I am. But I do. You know, I spend some time on there, but I don't know that if you're a user of this forum, you're having a percentage of your life consumed. That's high enough to be considered cult like.

1:55:49 Chris: I agree. I think that if everybody, unless wrong, were I doing what that one guy was doing and 100% of them believed in Roko's basilisk and were donating every last possible cent to the research institute, then I would say hi. But I don't think it's many people doing that. Kayla: Right. Chris: So, yeah, I would say that's percentage of life consumed fairly low. Kayla: Yeah. I'm gonna ask you to answer this one. Ritual. Chris: Oh, ritual is high, dude. Kayla: Hi. Chris: Yeah. Because, like, if you. I definitely recommend checking out less wrong and reading some of their stuff. Kayla: Oh, you said all the acronyms. Chris: There's acronyms, jargon, and so much like in jargon and tropiness that, you know, references other stuff that's been talked about.

1:56:31 Kayla: It feels like you're talking about tv tropes. Chris: It is. Kayla: You ever been to tv tropes? Chris: Tv tropes is almost. Kayla: You can't parse it. It's like, you can't. Chris: Right. Kayla: It's impossible. Chris: The vocabulary they use is been generated from other parts. Kayla: Yeah. Chris: Of their own site. Kayla: It is turtles all the way down there. Chris: Yeah. Tv tropes is completely out of control. It's not even about tv anymore. No, it's about itself. Kayla: Yeah. Chris: I love tv tropes, but it's similar to that where it's like, when you first read an article there, it's kind of hard to understand because of how referential it is within their own community. Now, that doesn't mean they're going to a temple and performing blood sacrifices or anything. They're not doing that, but I would say it's fairly ritualistic.

1:57:13 Kayla: I'm gonna ask a question here, because that behavior in a forum that doesn't feel outlandish, that doesn't feel atypical. Like, I'm just thinking about some of the, like, I'm just thinking about Reddit and some of the subreddits that I frequent on there and how with many of them, especially, like, very specialized ones, that kind of thing happens. You've heard, am I the asshole? You've heard that where it's like, people be like, am I the asshole? There's a lot of acronyms going on there. Aiti n a yta, various jargon that's used. Or, like, just know mother in law, which is a form about shitty mothers and mother in laws. Go read it. It is the greatest team of all time. These people are insane.

1:58:05 Kayla: But, like, if when you start trying to read these posts, it's like, until you get a handle on the jargon and the nicknames and the acronyms on any forum, you're gonna experience that. So does that mean, like, all of these forums are ritualistic? Chris: I think they have an element of ritual, for sure. I think anything, to me, any language that helps designate insiders versus outsiders, to me, feels ritualistic. Kayla: Okay. Chris: It's. It's a, it's a signaling tool, right? Any language that is a signaling tool as much as it is a communication tool to me, says ritual. Kayla: Okay, then ritual here is high. Chris: Yeah. Kayla: And charismatic leader. Yes. Chris: Yeah, for sure. Kayla: Mister, sir man, whatever his name is.

1:58:46 Chris: Mister Yudkowski. Kayla: Yes. Chris: Yeah. He seems pretty charismatic. Generally, he wants to help develop a future that has friendly artificial intelligence instead of malevolent artificial intelligence, so. And also, he's very active on his own forum. I mean, he's posting tons of articles and replying to other people's comments and things like that. So he's pretty active. It's definitely. He's definitely there. Is he charismatic? Yeah, I guess. I don't know. I've never seen him, like, talk in person. I didn't watch a video of him, like, I mean, speaking. Kayla: One of the only things you, like, kind of said about him was that he wrote this post where he called that guy an idiot, which doesn't seem super charismatic, but also charismatic doesn't have to mean, like, Jay Z knights a dick.

1:59:28 Chris: Right. But she's also charismatic, but she's charismatic. Kayla: Or, like, a lot of these people are kind of jerky. Chris: All right, so I'm gonna go with exists. Kayla: I'm gonna say there is a presence of a charismatic leader. So. So we got expected harm. We got charismatic leader, we've got niche. We've got semi antifactual. I think it's a cult. Chris: I think so. I was. I was leaning cult. Kayla: It's cult mandae. Got that ritual. The only thing that's kind of like, eh. Is percentage of life consumed. Chris: Yeah, no, I'm calling cult on this one. Kayla: Sorry, guys. Sorry, sorry, guys. Chris: If anybody is listening to this from. From less wrong, that doesn't mean that, you know, it's nothing bad. Kayla: If it makes you feel any better, this one said this man right here.

2:00:13 Chris: I like you're pointing to me because we're recording the video. Kayla: Yeah. But also this. They know who I'm talking about. Said that I'm in a cult because of my involvement with Cicada 3301. Chris: Yes. Kayla: And you called it a benevolent cult. Chris: Benevolent cult, yeah. I see this for the most part, aside from occasionally throwing off ideas that make people's brains hurt, I think for the most part is a benevolent cult. Kayla: It's neutral to benevolent. Yeah. Chris: Yeah. Kayla: But probably net positive. Chris: Yeah, I think so. I thought that the stuff that I read on less wrong was really interesting. I think Eliezer's mostly a good guy trying to do the right thing. Kayla: I think dedicating that level of thinking is generally a plus, a positive, a good thing.

2:00:57 Chris: Right. I think the thing that helps me think about it a lot is kind of going back to that analogy with the mining. They are mining for truths. They are mining for information and insights in this very novel field that involves some really weird ways of having to think about the world and decisions and whatever. And eventually you're gonna. By doing that, you're gonna run into something that could potentially be hazardous. Kayla: You know what, do you want me to draw analogy here? Chris: Oh my God. Kayla: You know what it's exactly like what? You remember that little flash game, mother load? Chris: Yeah. Kayla: Literally that. Chris: It's like mother load. Yeah. Kayla: I don't. Chris: Spoilers. Kayla: I don't want to spoil. Chris: Go play mother lode. Kayla: But go play mother lode and play. Please, please believe me when I say.

2:01:40 Chris: You play to the end. Kayla: You have to play to the end. It will seem like nothing's going on, and then all of a sudden you will achieve salvation. It is a fantastic gaming experience and relevant to what we just said, which I don't remember what it was because. Chris: We were mining and discover a hazard. Kayla: Yes. Chris: So I'll also say that if somehow, some miracle we get somebody that listens to this show from less wrong. I would love to interview you. Please contact us@cultorjustweirdmail.com or on twitter. I would love to do an interview with one of you guys to talk about the experience from the inside. I usually like trying to get primary sources for this kind of stuff. I wasn't able to this time. I actually tried to find Roko's contact information, but strangely, it's totally not out there. But yeah, I would love to talk to someone that was sort of like experiencing this on the inside. So on the off chance, please contact us.

2:02:36 Kayla: Question for you. Am I in this call? Chris: Both are. Kayla: I'm not part of the community. Chris: Oh, I think we both are because we're both like total singletarians. I was saying I'm reading Scared of Roko's basilisk and I thought it was, like, really interesting. Kayla: Right. Chris: I'm kind of disappointed that you're still scared of it after all we saw. Kayla: I'm not that scared of it, but it. You can't just, you know, you just snap out of it. Well, aren't you just so great? Chris: I'll tell you what. I tell you what. When we're done here, you didn't snap. Kayla: Out of it either. You read a shit ton of stuff that I've only heard secondhand. My brain needs time. Chris: I'm saying when we're done here, definitely read the rational wiki article because it has some additional things that we didn't mention on the show.

2:03:18 Kayla: I will do that. Chris: Actually, one of the things that I didn't mention I probably should have was one of the reasons that they give is that AI isn't just going to be like, oh, I invented Rokobasilisk now. Like, it's not going to be like off and then on, right? It's going to be, if it exists at all, a gradual process where, you know, one thing affects, the next thing affects, the next thing creates the next thing. It's going to be extremely chaotic, okay? And there's going to be no way to trace it back to, like, whether you, Kayla, did or didn't donate to. Kayla: But what about punctuated equilibrium, which for some reason I have talked about on. Chris: This show multiple times, that's, even if there is punctuated equilibrium, it doesn't mean that the process won't be chaotic, okay? The process absolutely will be chaotic and you will not be able to trace it back to which butterfly flapped its wings to make the AI happen. Okay, good to know, mathematically speaking, you will not be able to do that. Even a super intelligent AI will not be able to do that because. Math.

2:04:16 Kayla: Hash. Math. Chris: Boom. Kayla: You're a cult. Chris: You're a cult. So, yeah, that was cool. Yeah. Kayla: But I'm still mad at you. I'm sorry. Chris: Still mad. Kayla: Still mad. Chris: All right, before we sign off, I would like to say to you, our listeners and fans, that whatever set of words that you may have heard that have had the power to destroy you, I promise you'll find the words with the power to heal, too, and I hope you find them soon. Kayla: That's very nice. Chris: I'm Chris. Kayla: And I'm Kayla. Chris: And this has been co. Or just weird.