Watch Episode Here
Listen to Episode Here
Show Notes
This is a ~30 minute talk by Rafael Kaufmann, Pranav Gupta, and Jacob Taylor. There is a 30 minute Q&A here: https://thoughtforms.life/wp-content/uploads/2024/08/QA-from-Gupta-Kaufman-Taylor.mp3 ; the paper referred to is: https://www.mdpi.com/1099-4300/23/7/830
Some more relevant links:
https://drive.google.com/file/d/1bX5GK3rZX8RdgmcvXyfpOGEVYbIePMq8/view?usp=sharing
CHAPTERS:
(00:01) Active inference in collectives
(16:21) Organizational collective intelligence
PRODUCED BY:
SOCIAL LINKS:
Podcast Website: https://thoughtforms-life.aipodcast.ing
YouTube: https://www.youtube.com/channel/UC3pVafx6EZqXVI2V_Efu2uw
Apple Podcasts: https://podcasts.apple.com/us/podcast/thoughtforms-life/id1805908099
Spotify: https://open.spotify.com/show/7JCmtoeH53neYyZeOZ6ym5
Twitter: https://x.com/drmichaellevin
Blog: https://thoughtforms.life
The Levin Lab: https://drmichaellevin.org
Transcript
This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.
[00:01] Rafael Kaufmann: Thanks for the context, that's spot on. A bit more on that background. This was written while I was working at Google, and by the time it got published I was no longer at Google, so that's why it shows up as independent researcher. To me, this interest in both collective intelligence and active inference came about while working in organizational effectiveness. I was trying to get the great big collective of engineers, product managers, finance people, and so on that is Alphabet to work more as an intelligent entity. Being a mathematician, I decided to approach it in a mathematical modeling fashion. That also coincided with me running into active inference. On the one hand, it fit really well with my other interests and background in Bayesian modeling. But it was also pretty early days in terms of active inference modeling and code. If you dig into the paper, you'll see that the modeling is pretty outdated. I'm doing this in a quite different way. For discussion today, I'll skip over some of the details that I would try to do in different ways. I think the results still hold and the message is still spot on. As Pranav mentioned, it has led us, the three of us, in different directions, which I think are going to be interesting to you. The main motivation in a nutshell is that formal models of collective intelligence have not been well formalized in terms of the relationship between the local-scale interactions between the agents that make up the collective and the behavior of the collective itself at that higher-level state space in which we would expect to observe intelligent behavior.
[04:04] Rafael Kaufmann: The goal of bringing active inference in is that it's a pretty natural lingua franca with respect to which we can interpret these behaviors and measure what's going on in terms of free energy reduction and their Bayesian mechanics. In order to achieve that, we had to look at augmenting our agent models with some non-standard features. The most interesting being the particular way in which we think about alterity, about modeling the other: an agent modeling the other's behavior, which is particular to our interest in modeling collectives made up of agents that are highly sophisticated, where you would expect them to have a theory of mind that refers to the other's behavior in the world, and for these things to match up and to also have self-referential beliefs. The other part of this is the concept of goal alignment, which is commonplace in collective intelligence literature, but the way in which goals show up in most active inference modeling is either exogenous or top-down. It represents avenues towards endogenous goal alignment in a specific sense. What we wanted to do was explore the effects of providing active inference agents with specific cognitive capabilities. We hypothesized separately that theory of mind and goal alignment, these two capabilities, would result in better performance of the collective, and that their combination would be even better. As you're going to see, this combination is crucial because each of these two capabilities, even in a very simplified world model, has limitations that in some cases are very hard to breach, having to do with self-preferentiality, for instance. This is an exploratory paper and we haven't really put any effort into turning it into a full-blown theory. Take it with a grain of salt; you will see a lot of heuristic results, but we do think that it's illustrative. I'll very quickly go through the model itself and the results and pass it on to Jacob and Pranav. If you were looking at active inference literature around the end of last decade, you might have run into this paper from McGregor et al. It's not from the Friston camp; the formalism is a bit different, but we thought it was nice for the formal clarity and simplicity of the framing. This is two agents that have one common target position, which is the shared target here at the bottom. They each have individual target positions. This setup lets us explore the concept of goal alignment: an agent being willing to preferentially pursue the shared goal as opposed to its own particular goal, which might be easier. In our simulation we used 60 cells instead of 20, but it's this general setup. The positions of targets are randomized between runs.
[08:08] Rafael Kaufmann: But the basic concept is that there is a chemical signal that corresponds to a color or shade here, and that is the main sensoria that our agent has. We also posit that our agent therefore doesn't know its absolute position; it only knows its relative position, so it understands the geometry of the space it's in, but it doesn't know exactly where it is. It doesn't know the position where it is, only the signals it can pick up, which are imperfect. It can also observe its peer's delta, its signed distance from itself. That setup is sufficient to create compelling behavior. What we end up having here is that, as I mentioned, we implement these capabilities of theory of mind and goal alignment in straightforward ways. The illustration A is for the baseline agent, where each agent has just one internal belief state and one goal, one target state or target distribution, and performs a very simple active inference loop without planning or forward prediction, just one-step forward action. The setup B here is with theory of mind, where our agents, in addition to the self-actualization loop from normal active inference, also have a partner-actualization loop, which assumes that the partner wants the same things as they do and has the same generative model of the world, and therefore they try to infer what the partner is going to do as well. This symmetry helps the agents extract information from the partner's actions. Here there's the assumption with goal alignment that the agent is pursuing either the shared goal or a weighted combination of the shared goal and its own goals. With this setup, combining theory of mind and goal alignment, here's one example run. The setup is that there's a strong agent that is very capable; it has strong ability to track the signals and to find its target state. There is an agent that is weak and whose behavior looks more like a random walk. It can also track their beliefs, the distribution of their beliefs about the partner. There's discussion about the potential ways in which these mechanisms interact. For instance, when we endow the weak agent with a stronger theory of mind, it's better able to infer the location of its peer, which is what happens here.
[12:11] Rafael Kaufmann: As the weak agent's beliefs about the strong agent in the bottom, they start out pretty fuzzy, but they get increasingly sharp here. As the strong agent finds its preferred position, the weak agent, even though it doesn't really know where it is, has a pretty strong ability to detect where the partner is. So the main result is that for individual performance of distance from their target position. In the end, when you average this out, you really need for the weak agents to get anything anywhere close to the strong agent's performance. You actually need a combination of theory of mind parameter that is strong enough and a goal alignment parameter that is strong enough. And there's this interesting aspect of theory of mind being too strong: it's also negative because it can lead to a blind-leading-the-blind effect, especially if the environment is ambiguous, which can be the case when you have multiple targets as in this setup. Therefore, we observe a Goldilocks effect with respect to theory of mind, especially as it relates to goal alignment. And finally, how does this connect to collective intelligence? We modeled here the collective-level system free energy; you can imagine a bunch of identical copies of this same two-agent subsystem that are asked to decode different pieces of a problem. So like a parallel inference problem. Here we're just looking at the inference, the belief inference part of active inference, not at the active part, at least from the collective perspective. The setup here is that what we define as collective intelligence is the ability of this setup of M, a large number of identical copies of this two-agent subsystem, to aggregate into an effective variational free energy minimization ensemble. And that's what we observe here as well: the system free energy for this collective, which I mentioned was 60 pairs of peers, its ability to reduce free energy is, empirically, at least for this study, okay but not great to get to solutions with just theory of mind, as well as with the baseline agents that don't have anything. When you introduce goal alignment, you get that a lot better and more intuitive. But to actually get to results that look like they're going to result in a perfect, exact solution to the minimization problem, you really need the combination of theory of mind and goal alignment. There are a bunch of potential implications of this, and I'm going to get into it because I've already been too long-winded. From my side, where I took this has been first in the direction of trying to use multi-scale active inference to model the interaction of economic agents and the climate and nature risk, which eventually led me to leave Google and start a startup focused on this. More recently, I've been looking at this from the perspective of what I call not AI safety, but everything safety. So risk minimization and collectives more generally: working right now with setups such as fleets of autonomous vehicles and their safety, the ability of collectives to minimize overharvesting in fisheries and other tragedy-of-the-commons problems, and more recently in the financial system. I'm going to pause here and hand it over to Pranav and then Jacob.
[16:21] Pranav Gupta: Alright, is my screen visible? Is my presentation? I have a whole bunch of slides, which we will not look at all of it, but if you're interested, here's the link to the whole thing. I'll take about 10 to 12 minutes, and Jacob, you can jump in. So as I was mentioning, I'm in the business school, and I care about thinking in terms of how does this come about? Parallel to this information theoretic approach and using simulations, I was pursuing my PhD at Carnegie Mellon at that time under the same topic of collective intelligence and trying to understand where it comes from. The theory that we've been building on and trying to falsify and validate through empirical data and simulation, multi-unit simulation approaches, was the big idea being that collective intelligence can be thought of in terms of attention, memory, and reasoning at the collective level, regulating itself. The better it is at regulating these systems in response to environmental threats or the environment that they are in, the more intelligent the collective will behave. Two big takeaways are thinking of a systems problem requires thinking in terms of system solutions, and collective intelligence is a system solution that might not be obvious at the individual level. One perceptual or framework shift needed is to think in terms of dynamics of emergence: the person is not in the collective in the physical sense, but the collective intelligence or the rules of engagement that lead to collective outcomes are in the person. So these are the two things I will be making a case for. In my world, I've been talking to hospitals and software developers and things like that. What you're starting to see is most hospitals are extremely fast-paced and high-uncertainty environments. There is variability in patient admissions. There are case complexities that keep changing. Even if you have benchmarked case complexities as you're diagnosing and as you're doing procedures, the complexity is evolving and unfolding as we go. Technology is central. So at this point, it may not be fully agentic algorithms going in, but technology is central in creating most interactions. There are people with different specializations. In any given hospital, there are about 25 different provider teams, such as nurses, the cleaning crew, the doctors, different specialists among doctors — all of those people. They have to be interacting and coordinating in a fast-paced environment when the future is unknown. This is not just a problem of hospitals. As the pace of technology has grown, we are entering a place where much of the work is happening in larger teams where there are many different specialists and they have to coordinate. So this interdependence, this fast-pacedness, and this dealing with uncertainty, progressively so, is the crisis in the world of management or organizing. Software developers, physicians, scientific teams — we have evidence of this growing. A question that most teams or organizations are now pushing for is they're not concerned with their specific performance right now. It is not like an assembly line. It is about sustaining performance in a changing environment. When their workload is changing in unknown ways, their knowledge interdependencies are changing according to that, and you have to maintain and retain members. The best people have to be retained by organizations or teams because if they leave, they are also losing a lot of potential. So in short, in the management world, we've been using the metaphor of teams or organizations as adaptive systems, but we've really not embraced it. We cannot run away from that anymore. We cannot have structures that are hierarchical in terms of the org charts that you see, but in terms of dynamic networks of people interacting. At the simplest, highest level of abstraction, it is a systems problem of dealing with fast-paced change as a collective. It requires systems solutions. Local optimization, like imposing hierarchies, which are more stable over time, does not work and tends to have significant negative effects. The idea of collective intelligence from that perspective is the ability of a collective to sustain performance as their environment or the different tasks are changing over time. This mathematical equation is simply illustrative. This is not how I calculate anything of those things. Now, back to the point. What is the mechanism? How does this come about? I'm restricting ourselves only to humans or human-like things in which you have attention, memory, and reasoning as three cognitive features in people. In the large setup, we have a dynamic environment, changing levels of workload, knowledge interdependencies, and different members having different goals.
[20:58] Pranav Gupta: There's some form of interaction, then in response. There are collective attention, memory, and reasoning systems which interact with each other. The second point is: how do I get my team to behave with collective intentions? This is where I lean into the idea of using flocking as an example. This is a very simplified example, and most of you know this. I'll jump over it by highlighting the key frame in which we can think about this. I often start my classes and many of my business talks with: how are these birds coordinating? Current management theory up until very recently would say that there must be a genius right up top, because that's the leader. We know that is not true. We did not know that for a long time. Even top papers were trying to figure out how this communication was happening until we pushed through the idea of emergence, where we started to think in terms of decentralized control: there is no centralized control happening. There are some rules of interaction. Every individual agent only has local perception, but they're interacting with their neighbors. Neighbors, not steady neighbors but neighbors that could be changing themselves, interact with their local neighbors with specific rules in mind. If those rules are followed and well designed, you have sustained behavior that emerges at the level of the collective. None of the individual birds is trying to coordinate centrally or holistically. Very amenable to agent-based modeling. The insight here is once you think of the rules, the bird is not in the flock. Physically it is; cognitively, or from a perspective of corrective intelligence, the flocking—the rules of the flocking—are in the bird. How do we pull this into humans? The crisis of organizing, as perceived by individuals, is really a problem of Tetris. How do I make sure all of my work gets done while being a good team? I want to thrive and be successful individually, as well as the collective has to thrive and be successful as a whole without negative externalities. What are the rules that govern such behavior? In that push, the theorization we are building is that individuals have skills, focus, and goals; this could be machine agents also. The way these coordinate—how the skills get coordinated—is your memory system. How your skills and knowledge get coordinated is your memory system. How your focus and time get coordinated is your attention system, the collective attention system. How the goals and outcomes that people are after—the motivations that they have—get coordinated is a reasoning system. In a fast-changing environment, each member has a changing set of focus, goals, and skills. When all three of these systems regulate each other at the level of the collective, you would expect collective emergence and corrective intelligence to emerge as a whole. The theory I won't dive into deeply is that all three systems—the collective memory, attention, and reasoning—are built upon metacognitive processes. That is how our meta-memory functions, how our meta-attention functions, and how our meta-reasoning functions. Transactive memory, or collective memory, has been well studied for the past 20 to 25 years. But attention and reasoning as collectives are something we are starting to build more on. Ideas have been floating around; coalescing them is the work that has been done. Just as flocking—the average size of the flock—is an indicator of all the underlying processes working well, we have emerging patterns for each of these three systems. If there's a well-coordinated memory system, you will see specialization emerge. If there is a well-coordinated attention system, you will see a bursty type of activity happening in these teams. There is temporal convergence. There is a quick exchange of emails, a quick exchange of work being done, and then there is a period of silence. Then a task comes in: quick exchange.
[25:34] Pranav Gupta: So burst teams. And if there's a good reasoning system, you see increasing levels of commitment by members towards the collective. For those who are interested in systems dynamics, this is a multi-agent model, so this is only a representation. It highlights two big things. There is an efficiency aspect: how are we using our resources, memory and attention resources, and how are we coordinating that as a collective? And there is a maintenance of the collective itself. It is the collective tapping into the right opportunities and the right people. At a higher level, what regulation means here is that memory, attention, and reasoning have to regulate themselves in response to the main threat in the environment. If the task demands resolving high levels of knowledge interdependence, you need to have a dominance of coordination that is driven by the memory system. As these things go up and down and different types of threats come in at different levels, the response of the team should be able to regulate how many decisions are being made by individuals, with coordination based on memory, attention, and reasoning, respectively. We ran this in a simulation model. At a high level, from a purely efficiency perspective (memory and attention), rules that combine both memory- and attention-based coordination outperform any other individual rule or an uncoordinated rule. Once you build in that members have motivations and can regulate how much effort they put into the work toward the team or the organization, regulating the reasoning system — memory, attention, and reasoning — all three together will give you the best outcome. So you cannot ignore any one of them. So again, the big message is that at the individual level, when each member acts in response to others, as a manager you're working with your employees, making sure they're empowered, their skills are being utilized, and their time is being utilized in the best way possible leads to the best outcomes. To test the veracity of this idea at the high aggregate level, we used open source data from open source software teams. I'm not going into the details, happy to chat about them if this is of interest. The first thing you do is assess what is driving the behavior: what does the environment look like? Over a one-and-a-half-year period, it seems that much of the performance or quality of code produced by these software teams depends on their ability to deal with workload — the number of bug reports and new features they are developing — and not so much on the complexity of the code itself, such as different programming languages or modules. We have to be able to retain members. Once we know that workload and membership are the two driving factors in this dynamic environment, from the theory perspective, we would expect the attention and reasoning systems to dominate the explanation. Teams that have good attention and memory systems will have higher collective intentions. And that is what we find. We operationalize burstiness by looking at the attention system — the level of specialization in how members contribute to different parts of the codebase. We find that, while this is just an aggregation, once we dig in and look at how attention is affecting outcomes, you can see a clear modulation. In fact, it is only in the highest 25% of the sample where a well-developed attention system starts to matter. This points to underlying nonlinearities: the effect doesn't show up until about the 80th percentile of the sample distribution. But when it is needed, it is absolutely necessary. Without it, performance tanks. So that's pretty much where I'm at. We've pushed this theorization to more theory pieces on the general architecture of collective human-machine social cognition, and thinking about developing AIs not for assistance but as coaches — artificial social intelligence. This has been a big research effort we've been doing with DARPA for the past five years on how to develop artificial social intelligence. That's where we are pushing. I'll stop here. Thank you. And I'll hand it.
