Polyworld: Using Evolution to Design Artificial Intelligence

>> STRICKLAND: Hey there. I'm Henry Strickland, our speaker is Virgil Griffith he's talking about Polyworld using evolution to design artificial intelligence and having had to take artificial intelligence classes in--in college, I'd be very happy to let evolution do it instead of me debugging all those list programs they gave me. So, Virgil, as a young lad read a little too much of Douglas Hofstadter and he therefore dedicated his life to cognitive science and causing trouble. After some under graduate at University of Alabama, he went to Indiana, where he teamed up with Larry Jaeger. Some of the older Googlers might know Larry Jaeger form Apple Computer. He had a project called Polyworld long time ago and it still leaves on and Virgil's been working on it and adding features and things to it. Virgil has done internships at the Santa Fe Institute and at the Keck Institute and now is his first year as a grad student at Caltech. All right. >> GRIFFITH: Thank you Stu. Hi. I'm Virgil, I'm a--I'm first year grad student at Caltech. You can reach me, that's my--that's my web site for those of you wondering, the .gr stands for Griffith, people get confused about that. I'm not Greek. And that's my email address. So--so, in short, yes, I'll be talking to you about basically trying to use evolutionary algorithms as a shortcut to creating artificial intelligence. Simply because artificial intelligence is well, hard and--and evolution is fairly easy--well this was easy to set up. And the hope is that--that us--we can take advantage having lots of the CPU cycles and we let evolution to do a lot of the designing for us. So--so, that's the--that's the general gist and well--well, let's move on with it. So, there we go. So, what I interested in--feel like asking, what is artificial life anyway? They just go, you know, this is ill-defined. Well, in short, artificial life is--is--artificial life is like a super set of biology. So, all biology is artificial life but to be more precise, artificial life is all as it is today. So, as it says in the circle, and also what like potentially could be. So, all these, all their possible evolutionary paths that are--that evolution could have taken will also ultimately create artificial life and will be [INDISTINCT] with these areas because we'll be hoping to explore AI. Please I'll say it once. So, I was--so, just to begin, let's show real quick. So, this is a brief intro to evolution. Evolution is an algorithm. It's really straight forward actually. Here's how it goes. You had a population and you have--and some things stick around more than others. So, and--but some, yeah, that must--must be the case. So, that's for selection. And then, you had these things--there are some heredity. And then, you rinse-repeat. And regardless of substrate, you always get evolution with this. Very straight forward. You have a population of things, you--you only have only one that you have hill climbing and that--that's crap, you got to have a bunch and some reproduce more than others straight forward and then there's heredity. And with--with occasional errors. Done. It's all you got to do. So, no matter--just--yeah, it's great. So, okay, get that on the table. So moving on, I'm showing you a nice--nice--good example of using evolution to design body plans so, this is--before we get to AI. And this was not my work, this is by Carl Sims in 1994, it's very [INDISTINCT] so I'm showing it to you. So, basically in this case--so, he's doing--using evolution to design bodies--design body--body morphologies to do a different task in the world. In this case, the population is a--do we have a laser pointer or anything like that? I can just point. Well, anyway, okay so, the population is a whole bunch of these nodes and connections joining them. And you can mix and match nodes so, it's like they say, "Hey. I'm going to put this [INDISTINCT] over here and vice versa." And you kind of see, can--how they make these morphologies you know, about how--you know, how this makes a tree and vice versa. It's actually kind of cute when you look at it. So, they're actually worth understanding so. All right, sweet. Okay. So, in this case the--yeah, usually there's joints between parts. So, yeah. So, this population is a graph of nodes and edges and the--and the selection is to go different with certain tasks so, walking, jumping, something like that. And the--and the mutation is grafting nodes here and there. And we're just going to let it go and see what happens and here we go. So--no, okay. >> This demonstration shows. >> GRIFFITH: Trying to... >> Virtual creatures that were evolved to perform specific tasks in simulated physical. >> GRIFFITH: And that one. All right, start it again. >> This demonstration shows virtual creatures that were evolved to perform specific tasks in simulated physical environments. Swimming speed was used to determine survival. Most of the creatures are results from independent evolutions. Some developed strategies--is their evolved. Multiple--these creatures in simulated together--friction. Some simple solutions was just two parts were found. Some seemed like they could use some assistance while others were fairly efficient such as this rowing like behavior. Here is an odd cousin of the previous. A mutation caused him to tumble. Some creatures evolve to incorporate contact sensors in their control systems. Here is another inch worm like creature that tends to go in circles. This was actually a creature first evolved for its ability to swim in water then later put on land and evolved further. A successful side winding ability resulted. Here is one with a hopping style. The protrusions on its arms seem to help prevent it from tipping over. This was the fastest with a successful galloping like stride. This group was evolved for their jumping ability. This group was evolved for their ability to adaptively follow a red light source. The resulting creatures are now being interacted with. A user is moving the light source around as the creature behaves. This one seems to flail randomly but somehow still manages to approach the light. Perhaps it is mean to move the goal away just it is arrives. Here is one that has propeller like fins which are tilted depending on the direction of the light. It can adaptively swim up or down very well. >> Just a pause. This is one is especially nice because it looks like something that a human would design. Some kind of motor thing and if it weren't for this little part just hanging off here, you'd swear it was design and this case is a case where evolution has--has toned across--they're are very good designs, extremely efficient and it looks, you know, very much something that we would build ourselves. So, like basic seeing designs like this should like--should comfort so yes, this--this can work. Sure, is there a question? >> [INDISTINCT] recently this [INDISTINCT] >> GRIFFITH: You mean the cross network? X, this work was recently redone for the Artificial Life Ten Conference. I know that I used to, to evolve catapult designs. So I don't--I don't know if they've actual recreated all of this but--but I do know at least large sections of this have been recreated and I know that for a fact because I worked in the lab. So, so that's all I got. >> I'd like to read this book. >> GRIFFITH: Okay. All right, now I still-what's the next one we got here? Oh, so I've set some before where they--where they're moving kind--kind of weirdly specially the one where this--have like the big hanging mass. The sole fitting this function in this case was to--was to move your center of mass forward or just--just move it period. So in this case like--like evolution is very--like it loves to cheat all the time to--to find some way to do this. So in this case what I was doing is was have this big long tentacle thing and it was just moving its tentacle thing around. Thus its--thus its center of mass was moving. So just another thing to keep in mind is that--is that if you ever have any--you have to--when you design your evolutionary simulations you have to always know all the weird ways it could cheat and we'll get back to that later. So here's some more. >> This final group of creatures was evolved through for their ability to compete for control of a green cube. The creature closest to the cube at the end of the simulation is the winner. Here a strategy first arose for simply tumbling towards the cube. Then one learned to block out his opponent. But then later one learned to overcome the obstacle by climbing over it. Some pinned down their opponents. Some covered the cube with protective arms. Others simply unfolded onto the cube. The success of this strategy is often highly dependent on the opponent. Here's a Hockey playing creature, which takes the cube away and wins by a large margin. Here are two similar Hockey strategies battling it out with the appropriate gestures. This crab like creature walks well but often continues past the cube and instead seems to prefer beating up on his opponent. Against the arm, the crab seems to simply walk away. A successful strategy is this two armed technique that swipes quickly in from the side and moves the cube over to his second arm. These are the final rounds of competition amongst the overall best. Finally, the seeker arm goes against the sideswiper but the cube is just out of reach. >> GRIFFITH: Okay, so this is a fun movie that I would like to show. Number one, it's pretty and the second is because, you know, designing body types--well, that's kind of hard like doing those solutions yet I kind of think about them for a little bit. Now this is not designing AI but it does show ho--how like--how, how evolution can sow across very inventive solutions. And so this is meant to be like inspiring and say, "Oh, you know, maybe you can do something else better with this." So that's what we have next. So next is using artificial life to evolve artificial intelligence. So here's a--well, hear this--this idea. So the first question is how we do a population for--like, like what, what's--what thing do we mutate and tinker with to--force it to be intelligent and there's a lot of answers to this question. So Marionettes had--the Greeks had Marionettes and so--yeah, they--they strings so they're all deeply connected in this clearly the way you think about intelligence. And then Descartes says it's Hydraulics, so the mind it's like the Sewer system, here we have little compartments here and then lots of pretty art from that time of all about it. And Pulleys and Gears such Industrial Revolution--yeah, we have done this before. Telephone switchboard--yeah, we, we--we've heard--we've even heard this analogies. But Boolean logic--yeah, that didn't go so well. But I'm pleased that we finally solved it. And the answer is, not digital computers but it's neural networks. Praise the Lord. So, so I guess--I mean given the history should partake neural networks was kind of a grain--a grain of salt. But, you know, definitely there's some reason to think--think neural networks are a reasonable way for representing intelligence. I mean, after all, we, we, we really are--like we're modeling the brain much, much closer than say digital computer or Boolean logic. So, so even though there had been many--been many attempts, it's like what is the proper frame to--to capture intelligence. You know, the hi--history is not really on our side. But I still think there's--there's a good reason for it. So just--just go with me on this one. So now, the nervous systems--now, this ends very nice is that, if you look at the neuron--see a human neuron, like an individual one versus--versus say--say some other mammalian creature--even reptiles, you often can' t tell the difference between them. It takes like a real expert to do it. So like the--like an individual neuron level, we're all pretty much the same. It's all in--it shows in the connections. And evolution and it--like from us all the way down to like sea slugs. You see--you still see nervous systems that are roughly the same. So this very nice because roughly this says, "Because hey, if we could just get our basic model right. You know--you know--was say a sea slug, it could perhaps ride this model all the way up to the top." And if evolution did it once, why couldn't it do it again? So yeah--so now, we'll talk about sort of the way--so now we have our nervous system, the important parts about it. So in this case--so this case, we, we do not--do know some behaviors are innate. There must be--must be some things that are--that are, I mean, inherited. We also have many things that are learned. So the--so the nervous systems must change with the organism's lifetime. This just--this is just sort of basic principles, seems reasonable, we're going to go with that--so not too hard. And with all this in mind--I'm [INDISTINCT] to you, Polyworld. Tad-dah! This is the simulator. Not to be confused with Polyworld, we--we--we got a thread about this. So just so you know, this is not us, we're the other one. It's with two L's, we're with one. And we do--we do pre-date them but not that really matters. Okay, so what is Polyworld? Poly--Polyworld is an attempt to--well, before--since we evolved, artificial intelligence the same way natural [INDISTINCT], which is simply put the evolution of neuro systems in--in a complex, rich ecology and they compete with one another. So and we're--yeah, so the, the hope is that, we, we view with the model to make very simple and then through competition and through making the world, world richer. It can gradually like get better and better and better. Sure. >> [INDISTINCT] what causes the natural world very rapidly if they happen with enormous perils. How are you going to beat their time schedule? >> GRIFFITH: That's hard. I mean, I--let's see. Well, how would you do that? I guess, in short, the, the answer would be number one, we can place the ideas that create, or even an intelligent designer. We, we can help it along. And, and the hope is that, you know, we can say. "Oh, that's good. We want to like really help you." And it's not something natural evolution had, had the benefit of. And furthermore, Moore's law is really nice. And so I agree with you that is a problem but, but both of those two, two factors help. But it's, it's, it's, it's, it's a legitimate concern. So yeah--and in short--but Polyworld is a new software, it's open source. I'll give you the link at the end. And, and, you know, there's a kind of a girls--but most recently, people are using it for doing behavioral ecology experiments and like--and like--we experience very simple neural networks. So if you're side is to use for that to. So now we know what Polyworld is, what Polyworld is not. So Polyworld is not fully open ended. It's currently just--just designed to be a flat world. Well, it's like--yeah, let's have your fight--where--where critic interacts. It's not an accurate model of really anything. But it could be done. There's--I mean, there's--it's a--there's, there's no real problem with it. The--the only reason we hadn't made an accurate model of especiallly anything is because it's computationally expensive. And we don't believe it's, it's specially important. So if--so like right now, we're still using a simple summing and squashing neurons. If you wanted to, you could like--you could render all the way down to actual biochemistry if you're in to that kind of thing. I'm personally not. But, you know, you could. And if you're into ecology, you can do that to. So yeah, that's what I got. So we want some more. So until we uphold more--so here's usually what evolves in Polyworld. So organisms have evolving genes and mate sexually, straight forward. They, they do have a body but the most important thing about them is the neural network brains. Now, the connections in the net--in the brains are genetic. But at birth, all the weights are random. And--and Hebbian learning, which is the learning mechanism and, and, and the primary that makes the human brain. But simply put well, and that sets all the weights. And Hale learning, it's a very simple algorithm. It works like this. If two neurons that are connected together, fire at about the same time. The connection between them gets stronger. And then--so that's step one. And then step two is, all connections decrease in strength slightly so--and that's it. It's, it's kind of surprising. It's kind of surprising that--that it's this one learning mechanism that accounts most of our intelligence. But you know, so it goes. And their--and their vision of the world is quite--it inhabits flatland so they see a little--a little strip of pixels in front of them. And so basically, it's evolving--it, it is evolving a neural network to take their one dimensional vision and turn it into behaviors that help them survive. So--and just to let you know, there's no cheating in any of this on as you often see in evolution stimulations. There's no fitness function. This is like pure natural selection. This is as raw as it gets. If something survive--like, like the only criterion is really to survive any way you can. And this includes exploiting bugs in the code. And we'll show an example of that. So okay--yeah, so--yeah, I'll show you that in a second. So too-too-doo. Okay--go back. Okay, so here's a nice, pretty picture of Polyworld. Here's how it goes. So--where is my thing? Here we go. So, these round things are barriers, they can't cross those. These moving things here are the critters. And these green things here are food. So you see when a critter dies, they become food. Now, this is kind of an early stage--stage of the stimulator. And so they aren't very smart. They like going along the edge a lot. But they get smarter. I promise. So--so--so, basically, merely existing in this world cause you to lose energy. And if you--and if you--and if your energy gets to zero, well, you--you seize to exist. So, so thus like for anything to stick around, it must go out and find food or go out and kill something and--or it must mate with other organisms as well. If it doesn't, it just not going to stick around long. It's, it's, it's pure Darwinian. So--and you can kind of see how it looks here. So here's the--there's a top down view. And in between these little--well, in between these little squares here you saw, this was the world rendered from one critter's perception. But it's a stretched out slightly for our convenience. But to know exactly, they see--they see the middle strip of pixels in that. So okay--so that's Polyworld. So now listen to the, the Genetic model because I always get asked about that. You have to pay a lot of attention--this is mostly for reference, for those of you who are into this kind of thing. So these are--so the--I think before, there are body genes, there are brain genes. And this is the body ones. So here's usually how it works. A critter can be big, but when--but when it's big, it doesn't move really fast, but it can hold more energy in it. So, you know, it's kind of a trade off. And if a critter wants to be a predator, it can be really strong, so we can do that. And it can also determine its maximum lifespan. This come--this actually--this actually form from the evolution literature. They--they said that it--it's good that we have like a hard limit that we can't that--see. It's good we have a hard limit that if you just--eventually die of age. Because even though it's extremely unlikely for something unfit to live a long time, it's so utterly bad if something unfit lives a long time and mates a lot that you--that you want a really hard limit on the--on how long you can live. So this is also kind of motivated, it's kind of [INDISTINCT] so like for example, you want to have tons of kids, but give the most no energy. So it's the parent can decide how much energy they want to give them. Or if you want to have a few kids, and gives them lots of your energy. So this is, you know, whichever you want to use. So we'll go back to the colors in a little bit, but--yeah. So the green--how green a particular critter is--is determined at birth. So you could have like the light green critter and dark green critters, and stuff like that. And also their mutationry is also specified genetically. So--yeah. No counter points of genetic grade. Okay. So this exciting part, so this is the brain genes. This is like 95 percent of the genome. So here's how it works. So the genetic models specifies which colors you want to attention to in your environment. So if you think red is really important in your environment, you can spend a lot of neurons to go see it. Yeah. Also there are internal groups and these internal neural groups which were like this, and supposed how they're connected. So the genetic model only specifies roughly how many connections are between each neural group. It does not specify at the pure neuron level. And this is motivated from biology. So you --so if you see--yeah. Like--well, it just is, and stuff we were getting into. So for those who are neural network buffs, you can be with all that. But the main thing that takes home from this is that the genes loosely specify--loosely specify the brain, and it does that in sort of the neural groups level. That's really the main thing to take from this. So, to make this clearer, so here is how a typical brain looks. So you have one neural group here, you have excitatory neurons and inhibitory neutron. We distinct--we distinct--many neural networks have been the inhibitory neurons and excitatory neurons. They can like, like a single a single neuron have both excitatory and inhibitory connections. But when you do that, some biologist puts up their hands and says that brains don't work like that. And you say, "Well, fine." So there, for you biologists in the room, they're different, be happy. All right. So you have multiple of these things and they can have different numbers of excitatory--inhibitory nodes. And they cling to each other. So straightforward. And they connect back, it's nice. And then you can have multiple neural groups. And they can all connect to each other however else they want. Now, these internal neural groups connect to some output neurons or behavior neurons. And here they are. Now, these are--these are the seven behavior neurons, and they're defined in the simulation. And in short, there are things like move forward, turn left, turn right, eat, mate, fight, blink--I'll show that in one second, and focus. So basically every critter has this little light in front of it. That it can sort of, they can--that they can blink with. The idea is they could use some primitive signaling mechanism. As far as I know, they haven't fully--they haven't taken advantage to this fore signaling. But you know, you can give room to grow. They obviously can't evolve from doing it if you don't give it to m in the the first place. So it's in there. And we also weren't sure what kind of eye they wanted. So this--so depending on the activity of this neuron, they can have sort of a fish eye lens where they can have like, you know, really straight. So, and that's just only because we weren't sure what kind of eye they might want. So, you know, evolution can decide. Sure. >> [INDISTINCT] >> GRIFFITH: Oh, no. This comes next. Oh, sorry [INDISTINCT]. Okay. So here--so here are the inputs. Okay. So genetically--so if you're going to pay attention--so this critter wants to pay attention a lot to green, a little bit to red and not so much to blue . And so these are basically the inputs. And these inputs can connect to any of these internal groups that they want. And it also has an energy level. So this tells you roughly how healthy the critter is how healthy it is. And it also has sort of a random firing. Just because, you know, might want it this is the free will of the critter. You can think of it like that. And I'm surprised that they actually use the random. You wouldn't really think so. But they like random. I'm not entirely sure why they like random. But--you know, well, regardless. We put in there because they might--they might like it, and behold they do. So... >> [INDISTINCT] networks, how does a [INDISTINCT] networks? >> GRIFFITH: So like these internal groups could connect to each other however they want. >> [INDISTINCT] convergence? >> GRIFFITH: Yes. >> Okay. >> GRIFFITH: Oh, okay. Yeah, we'll deal with this later, so this thing's the input units and processing units. Not so important. Sure. >> Have you assigned energy cost to neurons? >> GRIFFITH: Yes. And roughly, the reason we did the... >> Repeat the questions. >> GRIFFITH: Huh? >> Repeat the question. >> GRIFFITH: Oh, I'm' sorry, I was asked whether or not there's an energy penalty for--for having a large number of neurons or for neurons being activated, the answer is yes to both, that problem was you didn't do this, they grew huge brains that like 99% did nothing, so you're just like well, like computation is just silly, so if you're going to have a big brain, it better well do something. So yes they get a cost for having--for just having a size--certain sized brain, or for neurons being activated, so like doing anything, cost you something. Okay, so good question, we didn't do that initially, and that's what happened. So this is rough--roughly the same picture I showed you before, and this is made using--using dot, it's really nice. Oh sorry, graph this, so this just shows your Polyworld brain map, saying no, really, I'm not joshing you, that's how they work, and these are the inputs here, they connect to excitatory neurons and inhibitory neurons, and these are sort of the behavior neurons, up here, you know, there's fight, turn, light, blink, et cetera. So it just kind of shows you what their brains typically look like, when they are not idealized, so, that's all you get from that. So, okay, so as far as the previous concern, everything is about getting energy so they get energy, they die, and that's bad. So, here's how they get energy, they can eat food pellets or they can eat other critters, straightforward. And they lose energy by doing anything, like merely existing loses energy, so if they don't do one of these things, they're gone. These especially, like mating cost energy, and being big and strong costs energy and just for having a brain costs energy, so, mention that. Okay, so now I'm going to show you some behavioral samples, of how the output neurons, well this is what it looks like, when they turn these things on. So here's eating, this neuron right here, and you see it slurps it up, Ta-dah. So, I'm going to show you some more of these, more into the emergent stuff. So, what's going to happen here is that, one critter, so okay--oh I'm sorry, I should mention this, the color of every critter, is an Archibee triplet, so, the redder a critter is, is how aggressive it is at this moment, the bluer a critter is, is how much it wants to mate with--mate with--just mate, at this moment, and the green is genetic, as specified before. The reason we decided this is because, well you know, you want to know when someone wants to kill you, you want to know someone wants to mate with you, very straight forward. And for green, the idea is that you might want to do kin selection. It's like for example, say hey, I'm in light green now and I want to be nice to you because you're a light green. Sure. So it's--we've seen a few cases where they have done some tribalism based on the green, but usually you have to kind of like trick it into doing it, but it does happen. So, based--the important thing is here is that these are both kind of red so they're going to do battle, so let's watch this one. So here we go, he runs into it, and it gets eaten and it turned into a food pellet and this one slurped up the body. That's how eating works. Oh, in this case, so the bigness of a critter is proportional to its strength, so basically, even though this critter was stronger, it just had like a lower amount of energy, and it got eaten by the weaker one. Okay, so here's how mating works, so this goes--I'm going to come in here, and mate with this one, and a little child will pop-out. Okay, so now we see what happened here, okay so, they made a child but they were so, they expended so much energy given the child--they put so much energy into the child that they immediately died afterward and the child ate their carcasses. So here, we can see that again, for those of you with kids. What is he going to--with the loop? Let's do it, there we go, nope, okay, mating, let's see it again. Okay, so their coming on, make the child, and they--they both die, and the child doesn't really care, slurp, okay. Next we have is the lighting, this is the blinky, I'll just show you this. This is--because it's me coming here and he's going to blink at you, so here it comes in oh, I'm sorry, it turns his blinky off, so right now the blinky is on cause you see that's his normal color and that's the blinky and now it's turned it off. So, so they could shine lights at each other. Okay, so, now I'll show you some--I'll show you some of the emergent behaviors. So this is one of the--so we call these things species just because it's kind of natural, technically they can still mate with each other but behaviorally they're so different that, it's seemed trees would call them that, so these are joggers, and all they do, they just go forward all the time. This case the world is--is--is--is-is a toroidal world so you can't go off the edge. We have other worlds where you can go off the edge. And they move in circles a lot. So, by this case, usually the first thing you see in a simulation, just always go straight. It's very easy to code and the food is--is--is--is--is randomly distributed. Why not? I mean you know, you're--it's--it's quick and simple. So that works. Okay, so this is a really nice one. I talked to you before about how evolution takes advantage of absolutely anything, like including your bugs. So, this is a very nice bug. Now, what this was--this was initially done, it had not occurred to me that--that having a child cost energy. You know, because you--you just do it, it's pretty easy. So, you know, that's my male bias. But, well, I'm sure it happens. When we initially--there initially was no cost for having children. Guess what happens so you'll see them, I think they're over there and we will zoom in a little bit. So, you see, they're all in a cluster over there. And we're going to zoom. There we go, okay. So, you see that--that they have this whole *** going on here and they are--they are popping out kids, like--like looked like, and then immediately eating them. And with this is--this is because--because eating--eating the children becomes a free source of energy. So, you have two so as far as the critters are concerned, you have two choices: you go out and get food or you can mate and have a piece of food appear right next to you. The solution is clear, and--and this was like really boggling, like why are they doing that? Because, this would be immensely successful, we take over everything. And I could--took a lot to figure that out. But yeah, so we--now, we cost--so, now like it costs energy to have kids. So, now we don't eat them. It's not as--not as--not as not as prevalently. So, okay. So, just--just to let you know that evolution will take advantage of your bugs. That's a really good way to test. So, okay. So, moving on from the indolent cannibals. Okay. So, now I'm going to show you some--so now I'm going to show you some actually intelligent behavior, at least well, primitive intelligent behavior, that has emerged form this. So, this is just to show you that yes, this is actually doing something, all right. Okay. So, we're going to actually get to see them act--they actually use their visions. So, [INDISTINCT] come by and this--and the critter lurched forward. And see that--okay. Here, well's--okay, there's more of them. Yeah, see--see, it jump forward. So, really all this was saying is that hey, they actually are using their eyes for something and they're using their eyes to control their behavior. So, simple enough, not--not very big claim. But you'll see is that we're actually getting something right, like keep in mind when these critters start, they have completely random brains. And I assure you, they're crap. They don't do that. So, I'll show you examples if you'd like. Okay, so now I'll show you some more ones. Here's fleeing attack. Or in short running away from red things. So, usually like the first thing--usually the first things the critters learn is number one, move that helps to find food. Number two, move towards green things because green--because food is the only green things. Well, solely green things and that internal getaway--turn toward the blue things because they want to mate with you. And get away from red things because they want to kill you. So, here's them wanting to get away from red things. So, we see a red thing coming up here and it's going to run away form it. And, run away. So, this is very nice. So, they--they--and this came out completely naturally. No--no--no--no supervision at all. Just--just--just playing do as the creator and letting it go. So, here's some more. So here are some foraging patterns. So, usually they--they--they like to kind of act out on their own, become a lone forager. But some of them they swarm, so you'll find like a whole bunch of very weak critters and they mostly just go in--just go in circles all the time. And they--and so, like they say hey, like, say there will be dark greens, okay, I want to turn--I want to follow dark green things and I want to turn in circles a lot. And if you do that, the swarm just sort of gradually moves, because the ones that are near food, they live. And so the swarm just kind of gradually moves towards the direction where food is. And that's--and that works. Slowly but it does work. Okay, well that's--let's see, [INDISTINCT] this one. Oh, this is kind of fun, you can see--actually you can--can't see them engaging in a purposeful behavior. Like you saw at the very begin of the stimulation, they all just kind of sat there. We just [INDISTINCT] no, they're actually moving around, actually turning towards green things, actually displaying kind of you know, pseudo purposeful behavior. So, that's a--steps in the right direction. All right, so here is what we've seen so far. First of all they make a lot of different kinds of brains. They're actually--they are using their eyes for something, that's good. And they're actually doing useful things with them, also good. So, all right. So now I'll show you some--show you some more science-y things we've tried to look it--we've tried to analyze the behavior to determine if we're actually getting anywhere and trying to quantify it. So this is a nice one from the animal foraging literature. So this is actually pretty straight forward. This is what you do, you have a world. You have a food patch on one end and a food patch on the other. And you say "Okay, well how are the critters going to allocate themselves?" So the very beginning they kind of uniformly dispersed, middle some like "Oh, like you know some hang out in here, some hang out in there, some in no man's land. And then the late they go "Oh being in no man's land is bad" I don't want to go there, so they hang out in the two food patches. So, so, so this--they're foraging that's good and they are doing it correctly. And even better if you actually look at--they actually form their optimal foraging pattern. So there's this distribution you commonly see in the foraging literature called the ideal free distribution and lo and behold, they hit it perfectly. So, all right, good for the critters in optimal foraging. So now I'll show you some Predator-Prey Cycles, these are kind of neat. So the colors don't come out that gray but it will be okay. So in this case we're looking at predator-Prey Cycles between the critters and the food. So in this case the red is the critters. This is for a particular food patch, the ones you saw before. So the red is numbers of critters in that food--is the percent of critters in that food patch. And the green is the percent of food in that food patch. So in short, what you see, let's pick I'll say this one here okay. There you see that the--that the critters lag the food. So first the food grew up high and then shortly afterward the critter said "Oh I want to go in this food patch" and then they over harvested and the food goes down. And the critters leave and go to the other food patch. And then the food was back up again and moved up and go back to the food patch. And this oscillates forever. Yes? >> [INDISTINCT] distribution there is no food growing in the middle... >> GRIFFITH: Right. >> Does the food in this graph strictly other critters? >> GRIFFITH: The food in this patch? No, no and this--this case this was two food patches close to each other and they would just go back and forth between the two food patches is what they would do. And depending on where the food--where more food was at that time. And they would oscillate always following the food. So, yeah. And this is nice because this is--this is a very similar pattern to what we see in like--in Predator-Prey cycles. You know the standard [INDISTINCT] thing so, also nice. And this is again like we didn't program any of this. Like we just simply designed a simple world with food and neural nets and said go. And we get all this--it just comes right out. So, okay, so now we look at the brains cause that's what we're really concerned about. So the main thing to keep in mind here is really kind of the connection matrix. This other stuff here being a scientist, like that. So anyway, this is a random brain from the very beginning--at the very beginning of evolution. All things are randomly wired together. And so there--there's one connection matrix. And this is one from the vision cortex of the cat. Now and it should be random slides of it. And actually one from a Polyworld critter after evolution. Ta-daa! Now let me take away from this. It's not a cat but it's certainly not random. And so they seem that evolution has gone from this to this with doing nothing but just sitting there and letting see a few cycles turn on it. So, again I'm not claiming the poly organisms are cats but I am saying that evolution is doing something very useful and it's putting tons of structure in there that you do not put in so all right. And this is kind of inspiring and you would go wow and maybe we actually could get a cat with this. So here we go. So now I'm going to show you some more quantitative plot, more than just looking at pictures. Oh sorry, so I always get this question a lot from philosophers in the room. They always say, "Oh is not alive" well okay, fortunately there's [INDISTINCT] that a really good definition of life. It's the Farmer Belin--the artificial revolution, published from the Santa Fe Institute. And basically it says it has these measured criteria to determine if something is alive. And not so coincidentally Polyworld explicitly designed to satisfy all these criterions. So in short yet kind of space-time, it does reproduce, it does have creature storage, it does eat and it has interactive environment and it does evolve. So in short, to that--well it fits the definition of life that most people used. So in your face. Okay. So then you will you say, I'm not sure if it's intelligent. Well it's a--sure? >> [INDISTINCT] it certainly has metabolism and it has functional interaction. >> GRIFFITH: Right. >> [INDISTINCT] >> GRIFFITH: Yeah. No, no, no, I'm saying here is quite satisfies all these. >> [INDISTINCT] >> GRIFFITH: It doesn't have information storage, it doesn't have that. >> Well if you have a coal left over from a fire you can initiate another fire. Would--is that information? >> GRIFFITH: I suspect--I mean, I don't really care if fire's alive or not. Fire probably can satisfy three or four of these. I mean, I'm not really attached--I'm indifferent to fire. But I suspect if you look at the kind of structure of coal or something. You probably wouldn't find--it might be I wouldn't have much information there. I'm not sure exactly how you'd like at it, I'm sure it's something you could do but even if fire is alive, okay sure why not. Okay Belin would say, well it is it really intelligent cause we just see them just moving around. Well there's no real way to quantify intelligence unfortunately. And I even [INDISTINCT] can do this. But however we see this on simulation means we have access to a lot more things that biologists don't. And sure we can use information theory and complexity theory to try and analyze the critters behaviors and their brains. And this is most of our research right now. So yes we analyze their brains over time. So, so here's a nice one so there are like three or four measures of Neural complexity out there. And so far I've implemented two of them and the critters all kind of follow this pattern. Oh sorry for this kind of complexity this is the [INDISTINCT] complexity. I'll get you the paper on it. In short this metric of Neuro complexity and Schwartz says, if all neurons fire independently that's not complex. And so yeah and if they all fire in unison, that's not complex either. So in short you want this kind of middle ground between everything behaving randomly and everything behaving uniformly that's what Neuro complexity is. But in short, if you look at any of these they encourage all on. They go up for a little bit and then they kind of plateau. And they're like "hmm" And both metrics do that. So well that's what I got. And right now we're trying to figure out how to make that go up more and try to explain why it plateaus. So I'm changing some other stuff now. So now that we know that neuro complexity does indeed go up. We want to know if evolution is actually helping this--helping the complexity go up or if it was just kind of going up accidentally. So there are two kinds of views of the evolution of complexity. The first one is this one, this is a more natural one. And it says that "Hey, you know evolution actually favors more complex from bacteria." You know just big bacteria and then eventually to us. And evolution really wants that. And the other one kind of says, you know what evolution really doesn't give a crap about complexity. Some things just kind of increase by accident on complexity and some doesn't really care. And the idea of this one is that if this is just mirrored if this diffuses outwards. You know on the spectrum of complexity you know just doesn't care about it at all, you know you will eventually get complex things and it's ready to start with this and you could get to that. And so this is evolution actually favoring complexity versus evolution not giving a rip. This is actually a debated question and we can use part one to answer this. >> [INDISTINCT] >> GRIFFITH: Right. >> [INDISTINCT] simple environment. >> GRIFFITH: Yes I do. The question is whether or not the complexity of organisms is predominantly a product of their environment. And the reason that we're not seeing a big increase in complexity is because the environment is so simple. And I think that's exactly it. So--and--so what we are looking at that now for ways we can make the environment more complicated to encourage more interactions and things like that. But that's about four or five slides from now so we'll get to it. This is the two ones this kind of experiment. Here's what you see so basically [INDISTINCT] polyworld to make all matings random. So in short even if you mate with someone, you don't actually get their gene. You get some random persons genes. It's sneaky so--and this is the dash line. This is where evolution turned off and oh sorry. This is complexity here, and this is time and the dark line here is with evolution on. Now this is very depressing, because you're like oh well with evolution turned off you get a higher complexity. You're like, well you're doing nothing. And I was very sad when I first saw this graph. But I always look at this thing here. This always appears like I've run this thing--I don't know, I believe it's ten times now. In short, there's always this hump here and I'm sorry and this is also a T test right here will get that in a second. But in short--the idea I came out with is that there's always this hump here and this--and the solution that I came was, well evolution does fairly increase in complexity but only up to a point. After you solve the world, we don't care if you're complicated anymore. In fact it actually costs you something to be complicated. And so as to the result we're going to keep you roughly right there. While the diffusive one just kind of goes up on its own. It's completely--it doesn't give out complexity at all. And it continues to go on up. Sure. >> Yeah. Isn't this [INDISTINCT] evolution just where the fitness function is how long will you survive instead of how much you made because if you randomly select a creature, creatures who live a long time are going to be around more to get selected at random? So, if you just survive a long time and you're alive when other people are mating then your genes will get passed on more? >> Let--let me think about this. >> [INDISTINCT] what did you do to select this--selection--the selection of the random genes from all the creatures who are alive at that time, that's my question. >> Yeah. I'm thinking, how was it done? I think it was--I think it was from all the critters who were alive at that time. So, the idea was--no, I'm sorry. No, actually--no, this is [INDISTINCT] that a very good--that's very good question but that was controlled for. So, in short, I'll--well, I'll give the more of the detail. Basically, this was that we ran this black line first and then, we said, okay, you know, I--and then we--then we, and then we said okay like critter--critter one lived exactly as many times of creature two, live exactly this number of time steps. So, we did random--so, we did random mating combined with enforcing that--that each creature lived exactly the same amount of time. So, but--but good question, clever. >> [INDISTINCT] for several thousand years sort of pruning out the dead code? >> Sorry, what does that mean, I don't quite understand. >> The complexity goes down because some of it is discovered to be unnecessary? >> Yes. I think--yes, correct. And that basically fits with my--with my current belief. I'm not exactly sure--sure why it plateaus and why it gets--why it stays there while the past has go up. But I think--I think it's pretty reasonable. So the idea is that, I mean, because you always see that like in the complexity is useful at the beginning but you want to be more complex than your environment makes you be so--so, the idea is that we don't make that more complicated and we'll see that if it goes up more. But yeah that's--I agree exactly. If you want to see this here, this is a T test Pleistocene to what extent based on the degree of confidence to which the dash line and the solid line are thought to come from the same population and they say, if it's above this critical here, which basically says, "Yes, we're pretty sure that humans have different populations." So, we see that--okay, right here, we're sure they came from different populations now, but actually [INDISTINCT] kind of crosses about right here. It just--it just--it just kind of--it mostly kind of sits there. So, there's a--so there's some math to make us think that as well. Okay. That's just what I got. So, now it's a Neural complexity--another one for genetic complexity and this came from my professor at Caltech, Professor Adame and it's really nice that you correlate math over quite well. So, it [INDISTINCT] complexity of the genes. [INDISTINCT] actually was it--it was 7,000 when they cross before ? Yeah, about 7,000. Okay, how about this one? 7,000 we see is roughly similar. Okay. So, the way this one works the dash lines again are the passive runs and the solid lines are the--are the--are the--with the evolution turned on. And so, in--we basically, see that on the passive runs the genetic--genetic complexity basically went down to crap while on the--on the active ronds the [INDISTINCT] did not go to crap, and in fact it stays quite high. So, roughly what this says--roughly what this--what this measures look for, it looks like the amount are not of disorder in the genome so, basically, if every gene was equally probable or--sorry, if every gene is equally present in the population then--then it goes to here. But if there some genes that are more favored than others then--then, I get this measure gets higher. I can see the equation for it but that's roughly how it goes, roughly it measures the amount of disorder in the population of genes and roughly this says, okay with evolution turned on, there's less disorder in the genes, so. That's good and nice. It's also can be that we see, the genetic complexity and the Neural complexity being roughly correlated, yes? >> [INDISTINCT] when you say evolution is off, your [INDISTINCT] turned off the sharing of genetic information for mating. >> Yeah. >> [INDISTINCT] for mating, where do you [INDISTINCT] >> Okay. When I say evolution is off, I say that the matings are random. And--yeah, I just say, the matings are random and critters are forced to live the same amount of time. So, the idea--so, there's controls and the matings are random and so. >> [INDISTINCT] made the results is one, it is in fact [INDISTINCT] Okay. Whenever evolution is off--when evolution is on, when two creatures make, there genes get match together and they make a child, so, completely normal. When evolution is off, when two creatures mate, it takes a completely--it takes a two random genes from things currently alive so, and then, it pops out that child. >> [INDISTINCT] made a copy of one of the parents or something. I don't understand the motivation for getting a random gene from some other creature. >> I'd like to think... >> [INDISTINCT] main copy [INDISTINCT] >> If it's completely random, its random--I mean I--I think--I know I have to--there you may be able to do this if you just make--make a copy of one of the--of the parents. You may be able to--I'll have think about it that's why that one would work two but-but I know if--if that--if--if very creature is equally favored, no matter what its genes are, evolution doesn't move like that--that's the rule. Like--like--like that--that has--that has selection with everything being equal--equally selected for. So that--that's what motivated it. Sure? >> Random selection on the [INDISTINCT] any have plan of population? You will have a genetic group, is that right? >> Yes. >> So. >> I think you should see here. This--this up and down genetic drift due to decline. >> I mean sometimes some ideals will be lost in the population just because of they. >> Right. >> Regular see. There would be--there will be this pair of mixing of possibilities but its slowly go to a fixed point, right? >> Um. >> Do you--do you see this? >> Well, you--you certainly all right. Like I mean because of funny population you--you will see variations in the pop--in the population. And I think its--is what you're seeing here. So in this case like this is two has completely random mating and it's moving up and down a little bit. And I--and this is--this is due to drift but as you increase population size this gets less and less and less as exactly as you'd expect. So--so yes, you're right. And--and in we're seeing it. So it's good. Okay. >> [INDISTINCT] >> Oh, okay we have to quick to them. Al l right, so it's the next time do really quick and to pass through this. So there's a real question of, so for this passive complexity it could be just be this passing complete--like why is this leveling out at all? So it could be that--that sort of--sort of upper bound in simulation because simulation cant support something--something of higher neural complexity plus we'll--so we journey with Polyworld to say, okay we will sole--we put through a fitness function mode. There's no longer natural selection of. We were working solely for having a complex brain and that that's the red one here. So in short this says, hey, you know the simulation can support much higher complexity if you--if you like really forced it do it. So, in short this phase says, hey there--there's room to grow for--for evolution. So, all right. So basically we have so the next pencil be making more complex environment and trying to move--move this curves closer up to the red. So, okay its making a draw from there. So, these are the few directions will take Polyworld into but predominantly making the world more--more complex and then come in with more measures of complexity for studying it. So in short more exercise in complexity there's--there's still like--there's still four or five that we haven't looked into yet, more complex environment. So the first thing at right now I want to add all like day and night cycles. So--so in this is really easy to do because it's all an open GL and you could just tweak the ambient lighting up and down. And the idea is that these would force them to--to have a sort of an internal clock, saying, hey it's dark now. I--I can't see anything probably shouldn't go foraging. Notice having different kind of food types. So you could have different colors of food and--and one will give you more energy that the other. So it's kind of having specialization. And the others giving them more--more senses, right now they only see and if you give like smell or touch it is they could have more interaction with the environment and that would be good. Yeah, so were done the actual forging we did that by recently. Yeah, and we held this to answer question about evolutionary theory as we did. Answer more questions of evolutionary theory like we did before. And did eventually we can skip up to casual--casual conditioning experiments. So this is kind of like--like the direction you want to go for the next few years. And I think you have ideas especially for here. Let me know or you want to get to decode. So this is mostly it. The source codes available, you can get it now. It runs on Linux and Mac via Qt, its just works. And then we can download it. Yeah--and at the very end I always get the questions, oh, you're making Frankenstein this is a terrible idea. And I--I was like this snide respond to them. So and yeah, I have no problem with that responsibility. It's a--its--its--if the polygon kill us all well--well it happens. Okay and I'm done. >> Questions. >> So just an idea about directions for--to test theories and evolution. Have you thought of a sex selection to see if their specialization have been given very little or a lot of contribution to the offspring, as if their two initials, two genders developed? >> Well, currently there's no gender. You could certainly do it. My--right now there is no gender right now it could be the one cut-cut the population like the mating pool in half. So like right now, these critters currently run with about 300 agents in a simulation, I'm sorry, the answer is yes you could do that. That'd be really cool but right now we don't do it because we are concerned about, it might be hard to find a mate. >> But I mean, I'm pretty ignorant to this, under some theories would say that the origin of the division of genders is that there was a specialization to niches, the males contribute very little, they tried to mate a lot, the females contribute a lot more and so maybe you can look for, you see if this two niches develop, even in the absence of explicit gender, I don't know, that was just an idea. >> GRIFFITH: You could certainly--it's certainly possible like if you had two different kinds of behaviors, and one was favorable one time and the other was favorable the other times, you could get that to come out naturally, but when they can always mate sort of all the time, it's going to be tricky for that two not to be enforced over the long term, but yeah, it's certainly possible and if you wanted to do gender differences, it's actually really neat, I mean if you start enforcing it see if they were starting to use each other, things like that, so that would be cool. >> This networks, at least, as I took it to mean, don't have any state on our cursive networks? >> GRIFFITH: Ah, no, they are recurrent networks so they can connect back if they want, we actually have a new kind, this recur to something like squashing neurons, we have a brand new model that has spiking neurons, I don't know much about it yet, but I haven't use it much yet but we do have more fancier models. >> And do you save state in between cycles? >> GRIFFITH: Well see, no we don't save state between cycles, but we do update their vision. >> Right, right, that seems to me to be necessary in order to maintain a mental model of where you are in the world, as opposed to just a single state, here I am, what am I going to do, it seems like that's uh... >> GRIFFITH: That's a fundamental part, yeah, let's see, I don't think we're saving--we're saving the state of the network from [INDISTINCT] to the next, like of the internal nodes. I'd have to think, well I can answer the question empirically, and like 10 minutes I went to the code, so I'll answer it a little bit. >> I think this is a really good, interesting presentation but I guess I have a little difficulty because I'm not that familiar with the area to have some context for it, could you say just a few words about sugar world and Tierra and Neuro Darwinisms so I have some sense on how this .. >> GRIFFITH: Oh, yeah, I've heard of sugar world, but I haven't--I have never, I've heard of sugar world, I know that--I'm sorry, I should back-up, so there are previous simulations, Tom Ray's Tierra was basically--was the first thing of evolving code, and it was really awesome, but there were a few problems with it is that they see things always got smaller and smaller and smaller, so that was kind of a problem in Tom Ray's Tierra, so like it always became better, if you're genome got smaller because that way you can reproduce faster because they were penalized. They only get a certain number of cycles to reproduce themselves and if you're very small you reproduce yourself a lot. I don't actually know if, as far as I know Tierra has not been extended to account for this original defects, but certainly Tierra is like really great, as far as sugar scape, I've heard of it, I don't know much about it, so, but if you send me a paper on it, I'll certainly read it, and I can go and come and tell you then. Sorry, what was the other--oh, neuro Darwinism, okay, so neuro Darwinism is a theory of neuro science. It's probably even true, in short it says that the way connections are formed in the brain, is kind of like evolution , it's not exactly, but roughly it says that neurons initially kept to a whole bunch of things and most of them suck and the ones that sucked, get pruned and they go away. So, roughly neuro Darwinism is like expand, prune, expand, prune. And it says this is how connectivity in the brain comes about, and it's probably true. >> So on your final slide, I think it was the final slide, you said that one of your goal is to make the environment more complex? >> GRIFFITH: Yes. >> And experiment with more features [INDISTINCT] and so I think it's, maybe a little bit of problem because your current system is already very complex and the thousand thing that affect the way evolution goes in your kind of system and you know how you construct their production procedure and so on and so on, so you're not afraid that if you make the environment more complex, you will be, possibly you will be able to see very fancy simulations but, it maybe more difficult to understand why actually evolution went this part, not the other way. >> GRIFFITH: Your bachelors isn't in physics by any chance? >> I'm sorry? >> GRIFFITH: Your bachelors isn't in physics by any chance? I mean, physicists always say that. So I'm wondering what your background is. >> No, no. my background is actually, I solve evolutionary computation for my... >> GRIFFITH: Oh, okay, all right. Well, yes, okay. Well--the concern is roughly, well if you make it more complex, you always have parameter help. You already have parameter help but it could be even worse. Like ninth layer parameter help. And the answer is yeas. That--that can happen. And I guess the response is, well, it seems like a lot of these things don't depend on the parameter very sensitively. So be like very, a bunch of parameters we have right now, you roughly see a lot of the same stuff. And the hope is if you choose this even remotely reasonable values, the good stuff will come out. And so, the point is valid. But we don't think--but we think like the benefit of having a more complex world far exceeds the concern of parameter help. >> I've been thinking about this for a while, I mean, you showed it to me earlier today, but I've also been thinking about this general problem, and I think that we can state without being too contentious that there are better strategies in the worlds that you're presenting. Like if we're really careful and designed one, we could probably clean the clock of a number of these evolved systems. And I think part of that's going to be not a product of the structures of the brains but the kind of input that they have available to them when they drive their behavior. Put another way, I don't think you should be adding complexity to your simulated world in terms of adding lighting effects or for or the things like that. I think there need to be more signals that have to do with kin selection and not--not just green. Like, in the natural world, even at the very cellular level, you--just is a natural by-product of the way evolution's is going to effect what kind of presentation you throw up on your cell walls. Like you can do kin selection in the environment pretty easily, like that's assumed. And so you can--a lot of the complexity we see in natural systems and how central systems are and how predation systems interact, seem to be driven by really complicated gradients that end up working out down the kin similarity. Like, I don't want to mate with someone who's exactly like me and I don't want to mate with someone who's really, really different from me either because, if I mate with someone who's exactly like me, it's not worth the energy because there's not going to be much variation. If I mate with someone who's too different, the child's not going to be viable. And like, the complexity in your environment should flow out of the behavior of the features that you're competing with. And you should see speciation resulting from preferences. And alternate patterns in like--that it doesn't seem like there's enough input for the neural networks that you're evolving which seem to be really cool to exploit that gradient. So I think maybe finding some way to allow them to sense the presence--and go ahead and cheat. You know, like look aside--do similarity scores and provide like, a sense that--similarity sense. You know, not based on light at all I mean, you're looking directly at the genes, because in any natural evolving system, you'd end up having pheromones and various other markers that you would learn to exploit. But they don't really have that. All they have is what they present directly and it would take a very, very long time for that to evolve. >> GRIFFITH: I think I get your--so your point seems to be roughly that the critter should have more complex interactions with each other rather than more complex interactions with the environment. >> Well, not even necessarily more--I mean, the actions that they can take are fine. I just don't think that they can observe the other critters well enough. >> GRIFFITH: Okay, yeah. Well, and so I guess the answer is "I agree." And if someone's to write it. If you're a writer, I would gladly put the patch in. So, now as to whether that would be--that--as to whether or not more complexity between--more complexity between critters would be more valuable than interaction with the environment, I guess you could try it and find out. I mean, I think those would be great. So, yeah, so there's no contention. Okay. >> All right, let's take one more question in Mountain View and then we'll let the video tapers go, unless there's a remote office that had a question that I wasn't fair to. >> GRIFFITH: Yes, so as a biological creature myself, I kind of hope that death is not inevitable and I was curious of what you were noticing in your simulations if you had turned off the limited lifespan of a creature. >> Oh, let's see. I guess you could just clamp it. You could do that. I don't know. The reason I did that is just because, like, I saw a paper at a conference that just had these mating populations. And it said that--that having a fixed lifespan or at least a max lifespan was a good thing. So I said, "Oh, well, just put it in a gene, done." So I have no--I've never actually clamped it and compare the differences. But you can certainly do it. I mean it's just a parameter. So... >> GRIFFITH: So what I've been thinking is that if you didn't have a limited lifespan, what would the results of your simulations be? That's what I'm curious about. >> Well, most critters don't get to their max lifespan. Most of them die of energy. So in this case, like, I think, like, the average critter lifespan is something like 300, 400 time steps and the maximum lifespan is something like 700, 800, something like that. So most, so very, very few get killed by that. So, I guess I don't think the maximum lifespan has much impact on it. And I just put it in there because I saw a paper that said this was good. So, and it was--and I was writing that piece of code at that time, so. >> GRIFFITH: Okay. We'll still be around after the talk is over if anyone wants to chat more. >> Okay. >> GRIFFITH: Thank you.