The Compatibility of Free Will and Cosmic Destiny

Is the future determined or do we create it? What if it's both? The self-organizing universe theory suggests a grand plan but leaves room for human freedom and creativity.

Is the universe perfectly deterministic, meaning that everything that will ever happen was predetermined at the moment of the Big Bang? Or, is there an element of randomness and chance to nature, leaving open the possibility for true creativity and free will? This has been one of the biggest debates in philosophy since its inception, and it does not seem to be going away any time soon — though there might be a new solution that is as curious as it is comforting.

The celestial orbits of Copernicus, the laws of motion articulated by Newton, and the mathematical logic of Laplace suggested a clockwork universe, ticking inexorably towards a predetermined future. The implications of this view are pretty depressing. Accepting determinism leads to a nihilistic picture of existence that lacks meaning. If the future is entirely predetermined then our experience of having freedom and choice would just be an illusion. This philosophical implication has devastating practical consequences. If we accept that we have no volition and quit living as if we do, our lives drastically suffer from it. There would seem to be no reason to get out of bed in the morning if none of our decisions are really ours. This has led to advocates of determinism making the shockingly paradoxical statement, “We don’t have free will, but we must live as if we do.”

But history, ever the cunning storyteller, had a twist in store. The quantum revolution of the early 20^th century, ushered in by the likes of Niels Bohr, Werner Heisenberg, and Erwin Schrodinger, painted a cosmos not of certainties, but of probabilities. In this probabilistic cosmos, there is room for surprises and creativity in nature. However, as philosophers have pointed out, randomness does not equate to free will. In the minds of some, a universe governed by chance is just as nihilistic as one that is completely determined. Cause and effect seem to give the world an intelligible logic that allows us to make sense of reality.

But what if there was an alternative picture of existence that had elements of chance and determinism with no contradiction? What if there was room for free will and a “cosmic destiny” — a grand plan that intelligent life is a crucial part of?

The view that is emerging from complexity science is that the universe is a self-organizing system that grows more complex through the emergence and spread of life, knowledge, and awareness. In this new emerging picture, rather than being in opposition, random and deterministic processes coexist and collaborate to produce ever-increasing levels of organized structure. This trajectory of growing complexity, ultimately headed toward a maximally-complex, maximally-integrated state of the universe, creates a predictable outline for cosmic evolution, but still leaves room for novelty and human agency.

The great enlightenment-era mathematician and philosopher Gottfried Leibniz argued that we live in “the best of all possible worlds.” I am not sure about that, but I think we could say that we seem to be in the most interesting of all possible worlds. That is one in which there is a compatibility between free will and cosmic destiny. To understand this new narrative of fate and freedom, let’s explore each relevant concept — determinism, randomness, self-organization, and free will — so that we may see how they are woven together to form the fabric of our reality.

The Dance of Determinism and Chance

Determinism became a formal hypothesis with a famous thought experiment from the 18th-century polymath Pierre-Simon Laplace, who pictured the cosmos as a giant deterministic machine. According to this picture, inspired by Newton’s laws of motion, every event that occurs is a strict consequence of a preceding event, which is itself a consequence of a prior event, and so on, seemingly leaving no room in the picture for chance or choice. The unfolding of reality is merely particles of matter obeying fixed laws of motion, such that their trajectories through space and time are entirely decided at the moment the universe came into existence by mindless mathematical laws. While it is impossible to prove this conjecture mathematically, Laplace proposed a thought experiment that came to be known as “Laplace’s demon,” which laid out the logic in what seemed to be an airtight way:

“An intelligence which, for one given instant, would know all the forces by which nature is animated and the respective situation of the entities which compose it, if besides it were sufficiently vast to submit all these data to mathematical analysis, would encompass in the same formula the movements of the largest bodies in the universe and those of the lightest atom; for it, nothing would be uncertain and the future, as the past, would be present to its eyes.”

In other words, an intelligent being who knew the position and velocity of every elementary particle at one moment could use Newton’s laws of motion to calculate the state of the universe at every other moment, past and future.

For many centuries, determinism seemed to be the only option for a universe that abides by the logic of cause and effect. But when quantum mechanics emerged about a century ago, there was finally an alternative to the seemingly inescapable concept of determinism that could allow for some “wiggle room” in the causal chain, reopening possibilities for a reality that allowed creativity and free will.

Theoretical and experimental work suggested that reality is fundamentally probabilistic, meaning that a specific event could lead to more than one possible outcome, and a specific outcome could be caused by more than one possible event. This revived the idea of indeterminism, which says the future is not determined in the strict sense once imagined. If reality is probabilistic, then it would mean that nature is inherently noisy, unlike the clockwork universe popularized by Laplace.

For example, the famous “uncertainty principle,” discovered by the German theoretical physicist Werner Heisenberg, revealed that we could never know a particle’s precise position and velocity at the same time. This made the dream of Laplace’s demon a physical impossibility.

While the uncertainty principle destroyed the idea that the determined nature of the world could allow a sufficiently advanced intelligence to predict the future, it didn’t completely kill determinism. Why? Because it didn’t seem to prove unequivocally whether our inability to measure the position and velocity of a particle was epistemic, meaning purely a limitation of what we can know, or ontological, meaning that such information was unknowable because it was actually not determined before measurement.

However, further experimental work suggests that uncertainty is an intrinsic property of quantum systems, and that at this level there is simply a “fuzziness” to nature. The Irish physicist John Bell formulated a theorem in 1964 that proposed a way to test whether local “hidden variables” — unobservable factors which could predetermine the outcome — could explain quantum phenomena. Bell's inequalities would be violated if no local hidden variables could explain the observed correlations between entangled particles. Experiments have repeatedly shown violations of Bell's inequalities, supporting the view that quantum mechanics cannot be explained by local hidden variables. This suggests an ontological interpretation of quantum uncertainty. This means the apparent indeterminism is not simply due to our ignorance; it is the way reality is. It would upset Einstein, but it seems that God does indeed play dice.

Additionally, the mathematical formalism of quantum mechanics does not require that the wave function (the mathematical object that encodes a quantum system's state and evolution) represents mere knowledge. Instead, it operates perfectly well under the assumption that the wave function is a real ontological entity, which also suggests that the probabilistic nature of quantum mechanics is a feature of the world itself.

Though many believe in quantum indeterminism, this degree of randomness is often assumed to get “averaged out” at the classical level of reality, which is the scale that matters for human interactions. That would mean the randomness doesn’t have any practical effects. However, new lines of argument have been presented by physicists for classical-scale indeterminism, primarily existing in phenomena described as “chaotic.” This type of indeterminism, which could possibly be connected in some way to quantum indeterminism, could even play a crucial role in the process of self-organization, and the phenomenon of free will.

Chaos and Indeterminism

Not long after some physicists had discovered that nature is quantum at bottom, others began describing a similarly curious phenomenon known as “deterministic chaos.” In the early 1900s, physicist Edward Lorenz showed that all “chaotic” systems, weather systems being his area of expertise, cannot be predicted in any perfectly precise way because of an extreme sensitivity to initial conditions. For example, if you put the measured value of a chaotic system’s position into a computer, you have to round that figure to some finite number of decimal places, and just that tiny bit of error you get is enough to completely throw off the accuracy of the prediction of the system’s future state. Why? Because with chaotic systems, which have dynamics that involve feedback loops, that small error gets quickly amplified.

This amplification inherent to chaotic systems also means that small fluctuations inside the system can get magnified, producing completely unpredictable behavior. A famous example is the so-called “butterfly effect,” where something as trivial as the flapping of a butterfly’s wings can lead to the formation of a catastrophic tornado weeks later. The resulting behavior of systems that display chaos of this variety is unable to be computed, even in theory, because a perfectly precise measurement would require an infinitely precise number. This led a Swiss physicist working in the field of information theory, Nicolas Gisin, to challenge the entire notion that chaos is deterministic.

Gisin proposes that even without quantum mechanics, classical physics inherently possesses a form of indeterminism due to what he calls “the finite information density of physical systems.” His argument is a bit complex, but once you get past the jargon, it is easy enough to understand.

In his paper, “Indeterminism in Physics, Classical Chaos, and Bohmian Mechanics,” Gisin argues that any physical system only occupies a finite volume of space, and for this reason can only contain a finite amount of information. As a consequence, a physical system’s properties, like its position in space, cannot be represented by a number that is perfectly precise, because a perfectly precise number would have an infinite number of digits after the decimal point, and hence an infinite amount of information. This means that there is a certain amount of fuzziness or imprecision to the initial state of any system, since it can’t contain an infinite amount of information.

To be clear, this imprecision is again ontological and not epistemic. That is, it is an intrinsic fuzziness, not a limitation on our ability to know the exact state. For chaotic systems, where initial imprecision gets magnified, the behavior of the system is intrinsically indeterminate — it can go this way or that. Gisin explains the history of this idea, based on a century-old field called intuitionist mathematics, in a recent article published in the journal Nature Physics.

Indeterministic chaos suggests that physical laws at the macroscopic scale might not be as different from the quantum scale as originally thought, as the underpinning reality at both levels is not entirely determined. Perhaps they are even related in some way — Gisin believes this line of thinking could help lead to a grand unified theory — though that is a question for future physicists.

This randomness in chaotic phenomena may not only be relevant to inanimate systems like tornados. In fact, it may turn out that the amplification of random fluctuations in chaotic processes can help explain the agency we see in the behavior of biological organisms, which are self-organizing. Randomness in nature does not give us free will, but it creates the possibility of true options or alternatives that an intelligent agent can choose from. In other words, indeterminacy is a prerequisite for free will.

The Role of Randomness in Self-Organization

Randomness, or “stochasticity,” plays a pivotal role in the process of self-organization, a phenomenon whereby a system of initially disordered components spontaneously organizes into a coherent structure without being guided by an external force. Self-organizing systems come in different varieties — from tornados and whirlpools to organisms and societies — and their evolution toward states of greater stability and complexity depend on a certain degree of indeterminism. In such systems, randomness is not merely a source of disorder but a catalyst for the emergence of complexity from simplicity.

The concept of "order from fluctuations" or “order from noise” posits that the inherent randomness within a system can act as the seed from which greater complexity and organization is forged. This counterintuitive idea suggests that far from being detrimental, a degree of unpredictability is essential for the functional order in self-organized physical, biological, and social systems.

A simple example of self-organization is the formation of a convection cell. You can demonstrate this phenomenon in your home by simply heating a pan of water on the stove and bringing it just below the boiling point. As the water is heated, the random molecular motion of the water molecules starts to form a structured pattern that more efficiently allows the transport of heat from the warmer region to the cooler one. This ordered structure emerges expectedly and deterministically, governed by the laws of thermodynamics, even though there are constant random fluctuations still occurring at the molecular level. It is as if randomness and determinism conspire to create functional organization that is patterned, but can also change form.

Random fluctuations are not just necessary for self-organization, but also self-amplification, or growth. Fluctuations in a self-organized system can be magnified by feedback loops that grow exponentially. In a receptive environment, this can lead to a self-reinforcing process where a specific pattern of behavior gains prominence over the surrounding disorder.

In the living world, the randomness involved in genetic mutations is what gives rise to new traits. Most of these mutations may have little effect, but occasionally, a random mutation confers a survival advantage, which is then honed and amplified through the process of evolution by natural selection. The result is the diversity of form and function we see in the biological realm — structures, behaviors, and networks intricately arranged to maintain and propagate life.

According to the British computational biologist Denis Noble, it is precisely an organism’s ability to harness and amplify random fluctuations that gives rise to the agency that is characteristic of living systems, which can be seen as the basis for free will. He explains this mechanism in his 2018 paper “Harnessing stochasticity: How do organisms makes choices?” published in the journal Chaos. In it he argues that living systems can essentially exploit the kind of chaos described by the butterfly effect through utilizing feedback loops to amplify fluctuations. For example, if an organism faces a problem that it does not have an encoded response for, it can harness randomness in mental activity to generate novel response choices. Then, the conscious mind selects from that new menu of possibilities. This selection process could be said to represent free will.

When we think of self-organizing systems, we typically think of those that exist inside the universe, and we often picture systems related to life, like organisms and societies. But what if the universe itself were a self-organizing system? That is precisely what some leading physicists and complexity theorists have proposed, like Lee Smolin, Stuart Kauffman, and Seth Lloyd.

The Self-Organizing Cosmos: A Predetermined Trajectory

If the world is moving toward a more ordered and interconnected state of existence, then that would represent a deterministic developmental trajectory, in the same way that an embryo deterministically develops into a mature organism. Where the developmental plan is encoded in the embryo’s DNA, the universe’s evolutionary trajectory would be encoded in the laws and constants of physics, which have been described as “fine-tuned” to produce structure and life.

However, unlike a simple deterministic model where every state of the universe is perfectly predetermined by a preceding state, this picture of the cosmos incorporates elements of randomness and unpredictability in quantum and chaotic phenomena. Yet, at the cosmic scale, the universe exhibits a developmental inevitability characterized by the continuous increase in complexity. This complexity growth, though it involves random processes, follows the intrinsic tendency for the universe to organize itself hierarchically, through a series of nested transitions. Nature’s simplest parts systematically organize themselves into wholes, which become the building blocks for the next level of complexity, and the process continues iteratively.

Atoms came together to form molecules, which came together to form cells, which came together to form complex organisms, which assembled into societies and ecosystems. Those ecosystems and populations have also formed an interconnected planetary-scale network of life on Earth that we call the biosphere. This means cosmic evolution is a process of multi-level self-organization which includes physical, chemical, biological, and technological evolution.

In this version of cosmic determinism, randomness and chaos serve a purpose. They allow self-organizing systems to explore a more diverse set of states and produce the novelty necessary for the trial-and-error processes of evolution. However, constricting this stochastic undercurrent are the top-down constraints — laws of physics, principles of natural selection, and eventually, the survival imperative of conscious beings — which guide the universe's relentless pursuit of its predetermined goal state. These constraints act as the boundaries within which the chaotic exploration unfolds, ensuring that, despite the element of intrinsic randomness, the overall direction is unwavering.

As stars forge elements and planets give rise to diverse chemical environments, the stage is set for the most intricate manifestation of this cosmic progression: life. Once life emerges, it becomes the vanguard of complexity. Consciousness and intelligence arise not as mere accidents but as crucial agents in the universe's evolution. They are the catalysts that accelerate the self-organization process. Living organisms learn, adapt, and manipulate their environment, perpetuating a cycle of ever-increasing complexity and order. The most intelligent of these agents have the ability to see forward in time, enabling them to consider multiple options before choosing the path they follow. In that sense they are truly “free.”

MIT professor and quantum physicist Seth Lloyd, author of Programming the Universe, writes about this type of computational determinism:

“The digital revolution under way today is merely the latest in a long line of information-processing revolutions stretching back through the development of language, the evolution of sex, and the creation of life, to the beginning of the universe itself. Each revolution has laid the groundwork for the next, and all information-processing revolutions since the Big Bang stem from the intrinsic information-processing ability of the universe. The computational universe necessarily generates complexity. Life, sex, the brain, and human civilization did not come about by mere accident.”

The emergence of technologically-savvy beings signifies a pivotal shift in the cosmic evolutionary process. With the advent of knowledge, particularly scientific understanding, cognitive agents begin to unravel the mysteries of the universe, becoming active participants in its self-organization. Through culture and information technology, life orchestrates a transition from passive matter shaped by natural forces to the goal-directed construction of cosmic organization.

The American inventor Ray Kurzweil, Google’s Director of Research, is a major proponent of this idea, writing in his popular book The Singularity is Near, “It is clear that the physical laws of our universe are precisely what they need to be to allow for the evolution of increasing levels of order and complexity…Ultimately, the entire universe will become saturated with intelligence. This is the destiny of the universe.”

In this vision (pictured above), determinism is not the antithesis of freedom but the scaffolding that makes it meaningful. The free choice of intelligent agents is constrained by the imperative to survive, and it is precisely this will that drives the continual growth of complexity and knowledge. As life spreads in an effort to persist, transforming more and more inanimate matter into its intelligent network, the universe grows more organized, increasingly conscious, and increasingly closer to its goal-state. Randomness and free will exist within the constraints of nature’s laws, enabling a diverse exploration of paths that ultimately converge towards the same inevitable zenith — a universe that is a complex, integrated whole, akin to a vast, self-aware mind.

How might this happen, exactly? While the details of this fantastical future remain fuzzy, we can already see that continual scientific progress inexorably leads to the outward expansion of life into the cosmos, since the persistence of life requires finding more usable energy. Though it might take some time to get there, our knowledge and technological prowess will eventually allow the human race to transcend our biological limitations. Post-biological beings that are capable of continuously improving their design — making every iteration more intelligent and resilient than the previous one — will result in what Kurzweil calls an “intelligence explosion.” This meta-evolution, an evolution of evolutionary mechanisms, will make spreading through the galaxy and beyond attainable for our unfathomably tech-savvy descendants, as well as any post-biological civilizations that have emerged on other planets.

Not only would a sufficiently advanced civilization rapidly proliferate and expand through space as a matter of necessity, they would also transform much of the matter in their midst into a form that increases their computational power. Integrating “dumb” matter into their computational structure will suffuse it with patterns of information, and with them, intelligence and sentience. Because the growth of complexity and knowledge both increase at an accelerating rate, continual technological progress will enable life and consciousness to spread through the cosmos at accelerating speeds. Some cosmologists, like Lee Smolin and Paul Davies, speculate that this process of technology-aided self-organization would eventually lead to the entire universe becoming a single computational entity.

If the cosmos does have a built-in tendency to self-organize, then that would seem to have big ethical implications for conscious beings, and how we should use our freedom of choice. The goal-directed nature of the evolving universe creates a cosmic context that instills life with meaning and purpose, and that purpose is to suffuse the cosmos with intelligence and experience by thriving, multiplying, and expanding. In this world, the universe evolves along a predetermined trajectory that is not only compatible with our free will, but critically depends on it. And that, to me, is a very spiritually-satisfying thought.