Watch Episode Here
Listen to Episode Here
Show Notes
This is a ~1 hour discussion with Kevin Mitchell (https://www.kjmitchell.com/), Nick Cheney (https://www.uvm.edu/cems/cs/profiles/nick-cheney), and Ben Hartl (https://allencenter.tufts.edu/benedikt-hartl-ph-d/) on the application of connectionist generative models in understanding the role of the genome in evolution and the control of form and function. Links to relevant materials mentioned in the discussion:
Nick and Kevin's recent paper preprint: https://arxiv.org/abs/2407.15908
Kevin's recent book: https://press.princeton.edu/books/hardcover/9780691226231/free-agents?srsltid=AfmBOorWozwAsuTtM1bnIa3xe1XVUWQShSWpkIVRMf1Z-6dNUarBhWqU
Levin lab papers on this topic: https://www.mdpi.com/1099-4300/26/6/481,https://link.springer.com/article/10.1007/s00018-023-04790-z , and http://rsif.royalsocietypublishing.org/content/13/124/20160555.abstract
Yuri Alon's lab: https://www.weizmann.ac.il/mcb/alon/
Andreas Wagner: https://sites.google.com/view/evo-wagner
Fernando Rosas: https://scholar.google.com/citations?user=OZNAs2wAAAAJ&hl=en
A few papers mentioned on the topic of evolution over agential materials: https://www.mdpi.com/1099-4300/25/1/131 , https://www.mdpi.com/1099-4300/26/7/532 , https://aeon.co/essays/how-evolution-hacked-its-way-to-intelligence-from-the-bottom-up
CHAPTERS:
(00:00) Noise, competency, robustness
(07:28) Planaria and bioelectricity
(12:11) Genome as generative model
(21:00) In silico morphogenesis
(28:16) Robustness, motifs, morphospace
(37:56) Control knobs, disentangled traits
(46:06) Development, cognition, multiscale causation
PRODUCED BY:
SOCIAL LINKS:
Podcast Website: https://thoughtforms-life.aipodcast.ing
YouTube: https://www.youtube.com/channel/UC3pVafx6EZqXVI2V_Efu2uw
Apple Podcasts: https://podcasts.apple.com/us/podcast/thoughtforms-life/id1805908099
Spotify: https://open.spotify.com/show/7JCmtoeH53neYyZeOZ6ym5
Twitter: https://x.com/drmichaellevin
Blog: https://thoughtforms.life
The Levin Lab: https://drmichaellevin.org
Transcript
This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.
[00:00] Kevin Mitchell: Ben, we haven't met, I think.
[00:02] Ben Hartl: That's right. Nice to meet you. I'm working with Mike.
[00:08] Kevin Mitchell: Okay, very good.
[00:09] Michael Levin: Ben, give 30 seconds about your background and so on.
[00:13] Ben Hartl: By training, I'm a physicist, a computational physicist, so I started working with evolutionary algorithms and how to use them to find ground state structures in molecular systems. This turned out to work but not as good as I wanted. I started looking into more sophisticated things like reinforcement learning, trying to give every molecule which tries to find its place a brain to self-organize. I think that's the overlap I have with Mike. To treat self-organization in a gentle way. We are now working on a few subjects which relate evolution and morphogenesis. We had a recent paper where we tried to answer whether evolution can go faster if it's working with competent parts or with cells which can do things — if it's faster if these functional parts of the gene are manipulated rather than a structural part of the gene, for instance, if you go for direct phenotypic encoding. The answer is yes, it can. Not always. It's dependent on noise. It's a little complicated. But depending on the system parameters, if you have realistic noise in the system, evolution can really well utilize these dysfunctional genes. To speak in machine learning language, we use neurocellular automata for morphogenesis experiments. In this case, if you have noise in the system and a program which is not super simple, then evolution can very efficiently utilize its competencies for self-organization. On top of this, it's also very efficient in transferring to different problems. For instance, if you manage to get these cells to grow a particular structure, it's very efficient to transform the goal into a similar structure if you have these functional genes available. If you go for more direct phenotypical traits, then this is much more complicated for the evolutionary process to find.
[02:25] Kevin Mitchell: Ben, by functional gene there, do you mean genes that regulate other genes?
[02:32] Ben Hartl: In this neural cellular automaton, you have a neural network in every cell, which is the update process of this cellular automaton. So that's the functional genes, the parameters of these neural networks, which regulate their own states in this case, the cell type.
[02:55] Kevin Mitchell: It's so interesting, the importance of noise. This has been a theme and something I've been in all of the developmental stuff that I've been working on, but you just see it in so many parts of neural signaling. It seems to me there are very, very fundamental principles at play in relation to noise and variability and how organisms or systems can take advantage of it. It's totally a feature, not a bug. They can tune it, release more or less of it when they need it. I was actually writing a paper with a number of people about variability and noise as part of the system, an ability that organisms have and can take advantage of. I must get back to it because there were too many authors on it; we were trying to pull it in from many different angles and everybody's schedules got in the way. I must get back to it because it's absolutely fundamental and I think not well appreciated.
[04:06] Ben Hartl: In this system we found a functional relationship between how competent, or what's the optimal level of competency of these cells, and the level of noise. There's always a trade-off in how competent the cell wants to go if we leave the competency as a parameter to the evolutionary search, as a gene. If you're very low in noise, then you can go direct, but if you're very high in noise, then you need more competency to solve the problem.
[04:37] Kevin Mitchell: Yeah.
Ben Hartl: But leaving the noise as a parameter would also be interesting in our system.
[04:43] Kevin Mitchell: The trade-off between stability and evolvability is there. It's the explore trade-off. It relates in terms of developmental robustness as itself a genetic trait, which can manifest as a tendency or possibility or risk of ending up channeling into states that we call pathological, which the genetics is reinforcing. The genetics of schizophrenia, epilepsy, and autism is pointing towards the idea of general developmental robustness as the underlying risk phenotype. The phenotypes that emerge are nothing to do with the molecules underneath; they're properties of the system. They're emergent attractor states in the system. It's a much more generic underlying risk that's all about noise and how much the system can buffer it.
[05:56] Michael Levin: Another way to think about noise, consistent with that, is that by committing to the fact that you have fundamentally an unreliable substrate, where, unlike the way we encapsulated in computer science, where the higher levels can depend on the metal being good, in this case, if you know that everything is going to change, so your low-level details are going to change, you're going to be mutated, you can't ever know how many copies of anything you have, then that force is the kind of architecture that you guys and we have been looking at, where you don't take the past literally, you compress it, but then there's the creative forward end of that funnel, that bow tie, where you don't really know exactly what happened, and it doesn't matter because you know it's going to change anyway, and it facilitates making these kinds of systems which just have to do the best they can in the current circumstances, which may mean interpreting the old stuff in different ways, because you could never guarantee. I tend to think of a spectrum, and I don't know to what extent these examples are real or we just haven't found out enough, but you can think about C. elegans being on the low end of that spectrum, where it's pretty stereotyped and it just assumes things are going to be what they are. Then you move forward from that into mammals and amphibians and then planaria on the other end, which is like they'll figure out a way to do the right thing no matter what you do to them, almost. We can think of it like that control knob of how confabulatory you are.
[07:28] Kevin Mitchell: I thought I really enjoyed the way that you had phrased that in that recent paper that you sent, not the one today, but the previous one. About the planaria, the fact that you can't make mutants of them that have any phenotypes. I didn't know that. That's interesting. The idea that they're at this far end of this spectrum, where they need their genome because they have to encode proteins. The regulatory web is just maintained on the fly, not so much by constantly referring back to the genome saying, what should I be doing? What are my settings? It doesn't save the settings in the genome in as rigid a way. I found that fascinating. It also explains why planaria may be unusual in relying on things like electric fields, more than ones that refer back to the genome more often.
[08:35] Michael Levin: We have no evidence yet that the C. elegans care about the bioelectrics really at all. In between amphibians and mammals, we have pretty good evidence for a lot of stuff with planaria. I always thought it was interesting that in planaria, there are no mutant lines except for the two-headed and the cryptic lines that we made, and ours aren't genetic. They're precisely the bioelectric ones. It's wild. So I think one thing that is interesting about bioelectricity in particular, although I'm sure it's probably true of biochemistry and biomechanics too, is that it's a really convenient mechanism for tying together the neural connectionism that you guys were talking about with actual development. That's literally the same mechanisms. I think evolution really just loved that symmetry and really made use of it. So this is something that drew me from the very beginning when I started to work on bioelectricity; that's what I wanted to get to, is to see if there was a symmetry between how it's used in brains and why it's such a convenient feature for cognition. And could we use that to explain this amazing problem-solving capacity of embryos, what you were just talking about, the robustness in that?
[09:50] Kevin Mitchell: One of the other upshots of having noisy molecular components is that relying on single mappings of 1 gene to 1 phenotypic thing becomes just a bad strategy. You can't do that. Instead, this sort of logic of fields and landscapes and collective global constraints to me is a much better picture of how developmental biology works, whether it's bioelectric stuff or molecular fields or some combination of interaction between those, which goes against the trends in developmental biology over the last few decades, which has been find the molecule that does the thing, find the signal. Where's the receptor? Where's the pathway? What's the transcription factor that turns on the things? Then we've solved it. Of course, it's a nice, tidy story and people love those until someone comes along and goes, but that same signal somewhere else with a different receptor or even the same one does something completely different. Context is key. It's interesting to me in terms of the sociology of the science of that field that there's been this absolute drive for nice, simple, linear, reductive stories. I think we're moving; people have realized we've exhausted that paradigm. We need to go back to the older way of thinking about fields and landscapes and so on.
[11:32] Michael Levin: What else is interesting is that, as far as that is true, there's also massive resistance, especially among the more organicist thinkers in developmental biology, against computational metaphors. You can get flack for bringing in any kind of connectionist cognitive thoughts, but you can also get flack for "oh man, you're saying it's a computer, it can't be a Turing machine." People really don't like some of the computational stuff, which is a shame because it has such utility in certain places.
[12:11] Kevin Mitchell: Nick, if you've gotten any flack on that front yet. I'm certainly expecting some because we're making this proposal, and I know, Mike, you've made a similar one, that the genome is basically a generative model. That's what it encodes, which is in the sense it's a compressed representation in a space of latent variables that allows the developing organism to construct a new instance of the type, which is whatever the species is. In the scheme that we have, that's analogous to this variational autoencoder, which is a machine learning thing that learns to categorize, say, pictures of horses. And it learns abstract horsiness through these compression through multiple layers. So it abstracts the invariant features of horsiness, and then it can decode them through an expansion of layers in the decoder to produce a new instance. So for our analogy, evolution is doing the encoding to get this compressed representation, and then development does the decoding. That to me feels pretty apt. And the idea of the genome as a generative model just sounds like a description of the job description of the genome. But the analogy with the variational autoencoder, I think people are going to say, oh, look, here they are. They're just reaching for the latest technology and it happens to be AI and it's just a fad. It's when we thought brains worked as clocks or hydraulics or whatever the latest thing was. But I do want to say that, for me at least, there's a distinction here with the idea that the genome is a program and it works algorithmically. Variational autoencoders and other models like that don't work algorithmically. They're trained algorithmically. But then the model is not an algorithm. The model is not a program. It's embodied as a set of weights. And so we have these new things, these large language models and other things, which are new kinds of things in the world that we don't even know what they are. We have to have a natural science of them. It's not like we just program them and you can say, here's this bit of the program does that, this bit of the program does this, and it happens in series as a nice clean algorithm. I'm hoping that people don't take it that we're making a computational, strictly algorithmic claim when we're making that comparison, because those things are not algorithmic. Nick, what do you think of that? Have we had any feedback so far?
[15:06] Nick Cheney: I received, I think, similarly very mild but poignant comments in that direction. I tend to agree that I'm even less interested in the analogy directly of the genome being generative. The lining up of these two things, I think, is fairly straightforward, and many people have made this in the past. But if we use this as a model, as a substrate, how can we poke at the computational thing and ask questions that give us insight into the biological thing?
[15:54] Kevin Mitchell: Yeah, I mean, I think that's the goal.
[15:56] Michael Levin: One thing that happens sometimes, especially on that more organicist end, is that people are pretty willing to accept that just because you know the parts of a biological system, you don't know the capabilities of the whole. But people still think that if you have something that is either literally an algorithm or something that can be simulated by an algorithm, then they know everything it does. This does not capture the magic of biology. This is one reason why I love these minimal computational systems. We recently did something with distorting algorithms. We don't know what bubble sort does, because if you look at it the right way, you can see some amazing things that it's doing that are not explicitly in the algorithm at all. If we extend the same humility that we like to extend to biology, to say that just because you know what the parts are, you don't know what it's capable of until you study it. As you said, it's a new thing in the world; you have to empirically study it. If we could extend that to computation and not assume that we know everything it does, then we could be okay.
[17:14] Kevin Mitchell: I think it lends itself equally well. You could build it as a computer system. The neural networks are built in silicon. But they effectively model collective dynamical systems that generate energy landscapes that force the configuration of the thing into sets of either local or transient minima if there's a trajectory. The perspective that you take on it is a choice in that sense. That's right. And for me, it's the same as the choice of saying, looking at your embryo and saying, because BMP7 is here and it binds to this receptor and turns this on, it's a linear reductive mechanistic view that isolates it away from the context of every other molecule and so on. That's a perspective that's not wrong. It's just an incomplete perspective because when you change the context, when you don't control the other variables, when you put it in another part of the embryo, other things will happen. That's important to know as well. But it is funny, people do have prejudices. There are some philosophers of biology out there now who are pushing against what they call gene centrism, as you know, Michael. And to me, they're overdoing it. Partly they will say genes can't specify traits in this one-to-one way that is claimed. And to me, that's a straw man because I don't know many geneticists who make that claim. But it's still, they look at the evidence for genetic effects on a one-to-one basis and say sometimes there's evidence for affecting something, but other things do as well, and it's all complicated. And sometimes they get to the point of saying, by rejecting gene centrism, they reject the function of the genome as a whole somehow, which is a weird move and unnecessary. You can accept that genes don't do everything and they don't drive things; they're not actively causing things, while also accepting that cats have kittens and dogs have puppies because they have cat DNA and dog DNA. You can accept that genetic information is a real thing, just that it doesn't have a simple relationship with the ultimate phenotypes that emerge. I'm hoping that the model that we present can marry those things and can reconcile, because it's not genetically deterministic, one-to-one linear simplistic mapping, but it's also not a free-for-all; anything can happen. That's in the middle, I hope.
[20:16] Michael Levin: One thing about that. The straw man is that I think you're right. I think explicitly most people would not at this point push that view. However, it has implications in, for example, biomedical approaches. I do think that most people think CRISPR, genomic editing, we're going to fix all this stuff. There are some single-gene diseases, but after that, what are you CRISPRing? I agree that this model and models like it have a good chance to provide that layer. Questions of causation are one thing, but then practically: how are we going to use these to actually make the changes that we want? How do we drive it?
[21:00] Kevin Mitchell: For the kind of work that Nick does, and Ben, that you do similar, the idea of in silico models that we can use as sandboxes to try and figure these things out, this kind of generative model encoding is, I like to think, two-step indirect. It's not just that it's not specifically encoding elements of the outcome. It's not even individually encoding or directly encoding parts, separate processes. It's collectively in this weird compressed way leading to those processes being constrained so that they produce this thing. And to me, that's a powerful encoding. It's really versatile. It's very evolvable. But I also think I've been thinking about how to go about actually encoding it in the kinds of systems that you guys have. And it's a challenge and partly because what you also need is just the self-organizing processes themselves. They just need to be there. Those are the things that the genome is constraining. And so in your systems, you're going to have to encode some physics of the stuff that you're making your bodies out of. I know, Nick, that your soft-bodied robots have that physics in there to a certain degree.
[22:23] Nick Cheney: To a very limited extent. This is a really nice connection with Mike's work: because there's such an indirect relationship between the things that are encoded and the final outcomes, there must be some complicated process of going from one to the other. But because this is a complex adaptive system and behavior is emergent, you can get really complex, chaotic behavior with very simple rules as long as you iterate them over time. Thinking of the layers of the decoder as steps in a developmental process — what's actually happening within each layer of a neural network, going from some set of features to another, more or less abstract set of features, is a very trivial transformation. Yet doing that repeatedly gives you these weird, crazy things that we no longer understand. Mike, the way you're pointing out the pieces of information pathways in which biological development does that too, and showing how we can manipulate very simply a limited number of pathways, and yet doing that within an actual process that's unfolding over time gives you amazingly interesting and weird phenotypes. Our current soft robot systems don't have that much complexity in unfolding and we've bought into a system that purposefully ties up a lot of that to make it simpler to compute. The idea of that process and the regularities and generative nature of them being the magic sauce here is really interesting. It's something I hope we capture a little bit in our current systems.
[24:27] Kevin Mitchell: It's all the stuff that the genome doesn't have to encode. That's part of what gets it away from the overreaching claim that the genome directs everything. It doesn't have to. We have proteins that have properties. When they're expressed in cells, they'll interact with membranes to cause physical tension, causing things like a tube to form or a vesicle to **** off, or other morphogenetic processes. The genome can constrain what happens by directing the expression of different sets of proteins with different properties in different physical domains of a developing embryo. It would be really interesting to try and set up a model system like that that's relatively simple, making a patterning of a limb. There are some morphological processes that happen; the condensation of cartilage cells is a simple Turing diffusion process. There are a few things where the physics are fairly straightforward. We've got a developing field that can develop into a whale's flipper or a bat's wing. It might be a nice test system where you could evolve the underlying genome and see if it can find constraint dynamics that produce interesting differences in the phenotype, where you could apply selective pressures — swim, grasp, or fly. That's one where the physics is constrained enough that you have enough to go on.
[26:25] Michael Levin: That's really interesting. I like the limb also because it has, there's lots of stuff that's worked out, but there's also puzzles: why you have two bones in your arm is not known at all despite the tons of molecular data—why are there two instead of one or three? Nobody knows. I like that a lot. Partly from Ben's work and other work, we have a bioelectricity simulator that is more or less biorealistic, and we do that and gene regulatory networks. The idea is exactly as you said: one thing we've been working on is the electric face in early frog development. The thing that maps out the eyes, the mouth, where all that stuff is going to be. To see what it takes to evolve something like that. Ben has some nice work on it. Another thing about GRNs, which goes to the point that Nick just made about emerging complexity and also to Kevin, what you were saying before about different perspectives and attractors. All of those are there, but there's another perspective which gives you some additional capabilities, which is that when we started looking at biological GRNs, and this is mostly real biological ones—random ones don't do this as much—it turns out that if you look at them through the lens of simple behavioral control, just training experiments, we found six different learning capacities there: habituation, sensitization, Pavlovian conditioning. You can get to that once you're willing to test the system in that way. That also has biomedical implications. I think this idea that there are multiple perspectives and each one facilitates certain things and doesn't particularly enable other things is valuable.
[28:16] Kevin Mitchell: Yeah, that's a good illustration of what you get from applying the perspective of machine learning, connectionist network ideas to gene regulatory networks. Uri Alon has really nice work on this. He wrote a great book on systems biology. I've been teaching some of this to our undergrads because when I first encountered his way of thinking about things, which is very much an engineering-physics kind of way of thinking about things—why are the circuits designed like this? You have two genes that turn each other off and they turn themselves on. Why is that? That gives you a bi-stable switch, a simple, obvious one. You can see why in a developing organism or a lambda phage that's deciding whether to be lytic or lysogenic, that's a good thing to have: nice tight control. You can look at multiple levels where those regulatory systems get more complex. They've got extra layers. One of the really interesting things is that you can get effectively multi-layer perceptrons out of those systems, which do pattern completion. Even if the signals coming in are a bit noisy and they're not the full range of all the signals you would expect, the system basically does pattern completion: it says, okay, I've got enough of these, and then the interactions at this level fill out the rest of it, and it can execute the developmental program for a limb or a heart or whatever is appropriate. It's a really powerful way of seeing the logic of what the system is designed to do, while still allowing that it's made out of messy biological components. In fact, it's because it's made out of messy biological components that it's designed this way. One of the interesting points that Uri makes really well is that when you look at the types of motifs you see in gene regulatory networks, you only see a subset of the ones that could exist. That's because those are functional and useful. But even of those, you see a smaller subset — those are the ones that do the functions robustly. That's really key. The biological systems are just dealing with noisy stuff, and therefore they must land on the motifs that work that way. Really powerful insight.
[31:06] Michael Levin: Andreas Wagner has some amazing stuff on this. Have you seen some of this? It's really good.
[31:14] Kevin Mitchell: Andreas, because these systems have to be robust to noise, they become robust to genetic variation.
[31:28] Michael Levin: Right.
Kevin Mitchell: And that allows genetic variation to accumulate, which can be cryptic in the sense that it's not affecting anything because of the robustness until it reaches a certain point or it's put together in a certain combination or environmental conditions change, and then you can get a new phenotype. It all ties together. The robustness and the evolvability are completely linked with each other.
[31:54] Michael Levin: I want to get your guys' thoughts on something. Let me share a screen for one second. I want to show you — I'm sure you've seen this before — and see what you guys think of this. Remember this? This is D'Arcy Thompson's "Deformations from Growth and Form." He puts these species on the grid, and then you do some sort of mathematical deformation to the grid, and you get something else that's a real animal. In the book there are all sorts of skulls and bones and everything else. What's this grid? I mean, this has been driving me crazy for years. What is the actual grid? It's clearly something. How does that map onto actual developmental biology? I wonder if the kind of thing that we're doing now with these encoders — do we now, because of that, have more insight into what this is?
[32:58] Kevin Mitchell: Nick, if you have any thoughts. What springs to mind is that we can think of these gene regulatory networks as shaping these energy landscapes in the sense of the landscape being patterns of gene expression that the cells could adopt. That's an internal landscape. But you can also think of those as being distributed across an actual physical landscape — the actual physical stuff of the embryo. I've been thinking about this in terms of axon guidance and the growth of nerves, how nerves are guided. There, there's a molecular landscape that they follow, but they're also following a physical landscape. They're sticking to other nerves. The rigidity of the axons is part of the force that pulls them one way or another. You can have levels where you've got energy landscapes at a molecular level within a cell that are effectively descriptors of attractors in a dynamical system space. At a higher level, as you distribute different cell types across a domain of cells, you get an effective energy landscape: gradients of protein expression, possibly gradients of ion channels that could create electrical gradients as well. What's interesting about them is that they regulate, so if you truncate it, they'll form the same gradient over a smaller distance. There are collective self-organizing capacities within there. That's really interesting. Mike, I'm sure that none of what I just said is news to you; you were very insightful. But that's the idea that you're nesting the same type of process at a molecular level within cells and then at a cellular level across fields of a developing embryo. To me, it's a helpful way of thinking about it, even if it's vague.
[35:15] Michael Levin: I wonder what you guys think. I showed this to a student years ago and she said, "That's great. Where's the knob? I want to twist the grid, I want to stretch the grid in a real embryo." Do you guys think that this computational formalism helps us to do that, to be able to use the kind of interventions that are available to us to actually deform the grid as a whole without saying, "OK, so this cell has to go here and that cell has to go there?" How do we get access to these large-scale features?
[35:49] Nick Cheney: I don't think it helps. I'm not the person to go and tell you what the knob is in biology, but I think this formalism of thinking of it as a generative model absolutely goes to your classic L-system of grow a tree. There's one little hyperparameter in that system that tells you how long each branch should be or the angle in which you see branching happening. You see exactly the same sort of thing in D'Arcy Thompson's book: the same pattern, but stretched or mutated and deformed in some way. We're not formalizing; we have the language and here's the knob within that to change, but taking that conceptual framework would suggest there is a knob somewhere. If we're trying to get the analogy of that in the particular example of a neural network, then we can do some fancy tricks to try and interpret what's happening within each neuron or synapse of that network to say, oh, here's a knob that might be good to tweak and do walks through the latent space or perturb the decoder in some specific way that would get you things that do in fact shape very much the way that happens. I think that's also an interesting question of what the ideal use case of this analogy is: focusing on the evolutionary part—how does this infrastructure come about such that we have this regularity, or to say, a lot of these creatures that we see in biology share these same regularities. Do we just have to find them once, and then we have this enormous palette to work with the way biology does, where so much of what we see is constrained according to one very good set of rule sets versus doing the search over the rule sets themselves.
[37:56] Michael Levin: The one thing that's—this is something that Daniel Lobo and I talked about some years back—is in particular for models like L-Systems, these feed-forward emergent models, what's frustrating about them from the biomedical angle is that there are parameters that will change things, but going backwards and saying, okay, I want to change this. For morphoengineering, for regenerative medicine, you want to be able to say, all right, I want this. What parameter do I change? For those kinds of emergent systems, that's really hard. I think biology does have an answer to it, and I'd like to take advantage of that. For example, here's a crazy idea. I don't know what you think of this on the computational side. Remember Vincev's deep dream stuff, where you push it backwards through and you ask, okay, what is your ideal representation? I wonder if we had these models properly parameterized, if we could do something like that and extract the archetypes, for example, of some of the things that we see in morphospace. And then you use that to say, what kind of prompt do you need to be able to shift it and basically control large-scale features? I want control over large-scale features, not molecular features.
[39:18] Kevin Mitchell: So Mike, what's interesting about this question is when you say, is there a knob that changes this thing? The question is, is it a single knob in terms of one molecule? Is there one molecule that controls whether your fish is long or wide? It wouldn't surprise me if there were developmental mutants that did affect that kind of form in a dramatic way, where there's one Wnt protein or one BMP that has a dramatic phenotype. I think a lot of evo-devo stuff is also looking for those nice, clean stories where it says the difference between this fish and this fish was that this gene got mutated, and they went from this all of a sudden to that. To me, that seems unlikely to be the way that evolution happens, that suddenly a fish is born that's twice the size of all the other members of its species; that's not the way it is. What's interesting about the encoding, the way we're talking about it, is that it's this distributed collective encoding. The question is how do you get evolvability out of that at all? It's so distributed, each of the latent variables in there affects many things, and each of the traits is affected by many variables, which is what we know from human genetics and quantitative genetics. All the traits are really polygenic and all of the variants are really pleiotropic. They affect many different traits. It seems like you should have gridlock. What's really interesting in the analogy with artificial systems is that they have the same kind of problem with the way they do the encoding in connectionist networks, but they have these orthogonal disentangled representations that emerge when there's some kind of pressure to make them that way, or when there's some connection costs as well. There are other kinds of factors that lead to this emergent modularity where it's not a unigenic modularity; it's polygenes, overlapping, but in separate dimensions of the representational space. This is an abstract representational space. What that means is that it's a great explanation for why it's possible to select for, either artificially or by natural selection, particular traits that you can get to move really far. Artificial animal and plant breeding is incredibly potent without affecting too many other traits until you get to really extreme selection. That's one of the things that got me excited by working through this model; we weren't thinking about that going into it. It popped out of the comparison with the machine learning stuff and these disentangled representations. It seems to be an apt way to think about the genetics of complex traits, where what it means is that to get at them, the best way is to select for them and then see what got selected. In an interpretability use case: do the experiment. Don't go in and say, "What can I tweak?" Say, "What am I selecting?" and then ask what got tweaked.
[42:54] Michael Levin: I completely agree. When I say control knob, I don't mean one gene or one protein. There are some convenient examples: one is there's a particular bioelectrical state that will induce regeneration of tails at a tail location, but not limbs in an adult frog. It's very context sensitive. We don't know how to micromanage either of those things. We didn't have to say which stem cells or anything. It has this feature: it's not a single gene for sure, but it does have this sort of "bow tie" node feature that all the stuff afterwards is handled, it's coherent, it makes sense, it's integrated together. I'm hopeful, just on the biomedical side, to identify some of these control knobs. Other stuff that we play with relates to cognitive analogies: if we take behavioral neuroscience reagents for modulating attention, precision, information processing, and so on, then you can do interesting things like apply SSRIs to regeneration or hallucinogens, which we're doing now, or serotonergic modulators, which we've done for years. For example, you can chase a planarian with a very particular type of head into a different species-shaped head with that species' brain shape, with that species' distribution of stem cells, and so on. No genetic change needed. It's a very simple effect. I don't know what controls the parameters of the head shape, but you can get them to do that. You can get frog tails that are shaped like zebrafish tails or the face of a different species of frog. You can change precision. One of the things in a recent paper under review talks about how if you use certain cognitive modulators that reduce anxiety and precision, you still get developmental outcomes, but they're stochastic and it's as if you've taken off the pressure to get it exactly right. They'll make something, but the error parameter that enforces tolerance is removed. I want to see this push together with other formalisms that are trying to merge neuroscience with developmental biology to see how much of that we can steal and reappropriate, because I have a feeling we're onto a very deep symmetry between the formation of cognitive systems and the formation of bodies. I bet we can reuse a lot of that stuff.
[46:06] Kevin Mitchell: The analogy, thinking of development as a cognitive problem that the organism has to solve and the cells within it have to solve makes sense to me. I know some people will object to the use of cognition in those terms, but I think it's perfectly defensible if you just define cognition in terms of using information to solve problems under potentially novel scenarios to produce adaptive outcomes. That's the type of cogitation that we do in humans is another type of cognition. We don't have to inflate one or deflate the other to draw analogies between them. I think that's perfectly valid. What's interesting, you got me thinking about what I said earlier about what the genome doesn't have to encode. It doesn't have to encode basic physics. It doesn't have to encode the fact that negative and positive charges attract each other. It can take a lot of that as read. And it's interesting, in an evolutionary sense, when we were thinking of pushing phenotypes one way or another, that in many cases, the bits that you're tweaking also don't have to encode all of the outcome, because there's other bits that will do the work. This bit that you're tweaking doesn't have to tell them how to do it. They know how to do it. It just has to stop them doing one thing and impose a different constraint regime so that the possibility space is narrowed over here instead of over there. And so what you get in that sense is this hierarchical relationship of constraints. And that's analogous to the hierarchical relationship of constraints in decision-making. Yes. I can choose. Even in prefrontal cortex, I can choose a goal that acts as a top-down constraint on parts of motor areas that pick out activities that I should be doing to achieve that goal, which themselves act as constraints on the primary motor area, which is selecting actions to do right now in the service of that activity, in the service of that longer term goal. And I think that idea of hierarchical constraints, in that case, operates over nested time scales. The prefrontal cortex goals are over a very long time scale. Within that, they're giving some top-down information that constrains stuff on shorter timescales. It seems to me you can draw similar analogies in developing systems, and you can think of the gene expression system as reconfiguring things on a fairly long time scale relative to the biochemistry that's doing signal transduction, for example. When you think about it that way, you get this segregation of what's going on at each level, because what's going on at the level above it is too slow to show up here, and what's going on below is too fast. You get these independent domains. So modularity emerges that way, because you can tweak one level and get effects at the lower levels for free, because that's the architecture of the system.
[49:26] Michael Levin: I think we have an opportunity here. We've been calling that a multi-scale competency architecture where each layer deforms the action space for the layer below, exactly as you said, so that it doesn't have to. There's autonomy, and so you don't have to micromanage. I think we have a good opportunity here. I've wanted to do this in a much more quantitative way: to actually look at the information passing through the different layers in this architecture and more rigorously tie it both to developmental processes and to decision making. I love that aspect of it, with the attention, the modularity, and the goal setting. I think we should do that. I think we should try to make a model where we can trace that happening. I can almost visualize the main figure from this, where you've got the network in the middle, and then we show the cognitive case and the developmental case below and just show exactly the mappings of what is happening layer to layer to enable the autonomy, the delegation of properties. The time scales are critical. We haven't done much on that at all.
[50:41] Kevin Mitchell: The time scales really give you a causal insulation. They allow for some causal integrity at a single level. And then top-down effects and so on. Fernando Rosas.
[50:57] Michael Levin: Yeah, I do.
[50:58] Kevin Mitchell: Fernando has a nice paper out recently with a bunch of other people looking at this kind of macroscopic causation and insulation between levels from a computational point of view. First of all, the reality of macroscopic causation: it's not the case that all the causation comes from the bottom up. This is something that we've been dealing with in discussions around free will, because there's a notion that everything is deterministic down at the low level, and it leaves no room for any other causes. We've exhausted all the causes deterministically at the level of atoms and molecules, everything else emerges bottom up. You can show that that's demonstrably not the case. It's even not the case in very simple physical systems. In more complex systems, it's not the case because they've evolved for it to not be the case. Once you add selection in there, you get macroscopic causation towards a purpose and a function. You get systems that naturally have this scale, a functional scale, where things are happening at different rates and different parts of the system have an independence from each other, not complete because they're talking to each other. They're not one causal web. They're segregated causal webs. That feels like a powerful method. I agree, I get this same sense, this intuition of something powerful there. The idea that if you could get a system where you formalize it and where you could play with it would be a powerful experimental theoretical system to have available.
[52:53] Nick Cheney: I think these questions of the different time steps and evolving towards them are an interesting way to think about the various time scales across development and about the generation of these things over longer evolutionary time scales. It's interesting how these things arise when the selection pressure for them is many generations down the road for things like robustness and adaptability. I could see cases — Wagner's work — where you have very reduced selection pressure that lets you build up enough mutations to have some complicated thing down the road, but I think the more interesting versions are when there's indirect pressure for that. Modularity in the brain is extremely helpful for a number of reasons, but presumably comes around just because of physical space limitations, connections, costs, energy constraints, where here and now gives you this longer term property too.
[53:56] Kevin Mitchell: I think that's absolutely right. A lot of the longer-term things that come out are due to very different immediate pressures, the evolvability coming out of having to buffer noise, which is just there right now. Evolution can't select for future good evolvability. Those are really good examples where the immediacies at play inform the way that the system can evolve, the possibility spaces that emerge.
[54:49] Nick Cheney: I'll hop in and say one thing quickly: the Polaria's constraints are relaxed under the hallucinogen that allows it to find this new phenotype, that it's relaxing some of them, but there's also very much decanalization and error correction still happening, that it doesn't create some totally random thing, but a well-structured phenotype of a different type of organism. Maybe I was misinterpreting that, but I think it's interesting to pull that to your question: in this computational model, how do you think of what things change what? It draws me to the deep neural networks-are-easily-fooled idea that we can go in and perturb a few pixels and this thing that looks like a dog will be extremely confident it's a cat or vice versa. Usually we're interested in those at the end of the process, in the phenotype themselves, because that's what the classifiers are looking at and will give us these interesting counterintuitive outcomes. Thinking about that at the earlier stages within your latent space — where are the opportunities where you're on some ridge in the manifold and can knock things into another part of the space. I think absolutely we could use those same tools to move around in that space in order to give you whatever frog face you want instead of the traditional stereotype one. I'd love to think more about that.
[56:44] Michael Levin: Thanks very much, guys. This was super interesting. Lots more to talk about.
[56:50] Kevin Mitchell: I remembered what I wanted to say. It was about the credit assignment thing in evolution, which is a really interesting analogy, and there's a very different solution to the credit assignment problem in machine learning with the backpropagation algorithm than there is in evolution, where it is this highly parallelized search across a population. It doesn't do it in series by training; it does it in parallel, and then it does it again and again. There's an interesting analogy there. Nick, in relation to what you were just saying, the idea of these atavistic phenotypes that we sometimes see, when you have a mutation and you get this throwback to what looks like a more primitive, evolutionary thing, where there's a memory of it, a latent memory of a phenotype that evolutionarily these things used to make, and then you can reveal that. To me, that's a super interesting thing, and it fits really well with the latent variable compressed generative model analogy, and it doesn't fit well with the blueprint program metaphors.
