Knowing Sooner

Global Reset Series / by Seed /

Our world is an uncertain place where biological systems and financial markets can collapse in an instant. Powerful predictive models fueled by smarter data sets are the tools that will allow us to know sooner and adapt more quickly to the problems that define our complex age.

On July 10, 2009, Swiss econophysicist Didier Sornette published a paper in the online journal bearing the provocative title, “The Chinese Equity Bubble: Ready to Burst.” The title implied that Sornette had accomplished a seemingly impossible feat: building a model of financial markets that was able to identify bubbles and predict when they would burst.

Sornette and his team asserted that they had found a bubble in the Shanghai Composite Index, and more boldly, that this bubble would end between July 17 and 27. Few outside of Sornette’s group saw reason to believe the prediction, because constructing a model of a financial market and making accurate predictions about its future behavior have been a long-sought holy grail of economics.

Perhaps not surprisingly, the 27th came and went, and the index continued to climb. Sornette seemed to have failed. But then on August 4 the market changed course. The index dropped sharply. Over the course of the next two weeks it fell almost 20 percent. Sornette’s prediction was correct.

There is still much debate as to whether Sornette’s team actually succeeded in forecasting the decline in the Shanghai Index. Some critics claim that the prediction itself may have influenced investor behavior and caused the market to drop, and many others simply question the idea that we are capable of making such accurate predictions at all about systems as complex as the Chinese stock market. But Sornette’s team is not alone. His is one of a number of pioneering groups of multidisciplinary researchers seeking to identify reliable, generic early warning signs that can be used to forecast the behavior of a wide variety of social, planetary, and biological systems.

This wasn’t Sornette’s first experience with the vagaries of a dynamic system. Before turning his attention to finance, Sornette studied rupture points in biological systems, which led to predicting earthquake eruptions. “But natural systems don’t fight back,” Sornette says of his decision to attempt predictive models of more complex systems. “In social systems such as markets, the theories become the engines that modify the structure of the system.” To date, his model stands out as one of the only potentially successful attempts to predict the behavior of a real-world complex system. But more may soon be on the way. Researchers have spent the past few decades laying the theoretical groundwork necessary to build powerful predictive models. Now smarter technologies capable of collecting and parsing more robust data sets than ever before may begin fueling these models with the information they need to address some of the world’s most pressing problems.

A Critical Theory
By definition, complex systems—be they financial markets or weather patterns—contain too many moving parts to be reduced to any simple mathematical formula. It’s not just that we haven’t discovered an equation to express the behavior of the stock market; it’s that such an equation does not exist. Instead, researchers like Sornette construct and run computer models in order to gain insight into the potential behavior of these systems.

In developing these models, they have discovered that systems all share some surprisingly simple underlying properties. For instance, systems have the potential to change drastically in very short periods of time and often exhibit early warning signs that indicate when and how these changes will occur. These changes could be stock market crashes, tsunamis, heart attacks, or colony collapses, and in general are known as critical points. The theoretical properties of critical points have some profound—and often alarming—implications for real-world complex systems. In the case of climate change, when a critical level of greenhouse gas emissions is reached, it has been suggested that Earth’s climate may undergo rapid and irreversible changes. Identifying points like this one, and devising smart solutions to avoid the catastrophes they may bring, is critical.

To understand how phenomena like critical points come about in a system, consider a rock concert: A band has just finished its final encore, as a 60,000-plus crowd reacts in rapturous applause. The clapping begins as a cacophonous patter, eventually growing to a loud, chaotic roar. And then something interesting happens. Amid the noise, seemingly without effort or conscious guiding by the audience members, the applause evolves into a synchronized, steady rhythm; the claps become a single beat, with thousands of fans clapping in unison. Finally, it slows to a once again out-of-sync denouement, before abruptly ceasing altogether. The synchronized clapping emerges spontaneously in the crowd and is analogous to what is called a self-organizing property of a complex system, of which critical points are one example.

Sornette borrows this metaphor, originally articulated by Phillip Ball, to explain stock-market behavior. “A financial crash is not chaos. It’s when everyone agrees; it’s when everyone is clapping together,” he says. “So you have a synchronization of actions in the same way that clapping becomes synced.” The bubble in the Chinese stock market burst at just such a critical point. This is what Sornette claims to have predicted with his market model. According to him, what indicated that the financial bubble was going to burst was that the behavior of investors began oscillating, more and more wildly, between widespread buying and selling.
The human body, itself a complex system, can exhibit similar early warning signs before the onset of an epileptic seizure. It experiences a series of minor seizures as it oscillates between normal and catastrophic states. What soon follows is a full-blown attack. Heart and asthma attacks are often preceded by similar reactions.

Oscillations are just one of the many different types of behaviors in complex systems that indicate the approach of a critical change. And as scientists continue to study the theoretical properties of these systems, they become more adept at identifying these behaviors. The importance of being able to predict the onset of catastrophic changes in our bodies, our planet, or our markets is obvious: The sooner we can anticipate a heart attack, earthquake, or tsunami, the better we can prepare to manage the consequences.

Complex Solutions for Complex Problems
Today researchers hope to gain a sophisticated enough understanding of the behavior of complex systems to devise solutions that avoid certain catastrophic system changes altogether. But we are discovering that in order to successfully navigate the murky world of critical points and complex dynamics, we must reconsider the way we interpret systems and construct dynamic solutions.

Once the various principles that drive a real-world complex system are understood, potential problems like stock-market crashes or ecological catastrophes can be identified. But the question of how to devise meaningful and actionable solutions to these problems remains. “With a complex system like a social system, you are never going to get to the level of knowing exactly what is going to happen in response to something you do,” says Tom Fiddaman. In collaboration with MIT and the Sustainability Institute, Fiddaman examines the policy implications of dynamic complexity in climate and economic models at Ventana Systems, a firm that specializes in practical modeling tools for high-level decision makers. “You are in a sort of dance with this complicated mess,” he says, explaining that it is impossible to determine the individual steps of this “dance”—and this is in some sense the error of current thinking. Instead, we need to be able to construct robust solutions that provide general guidelines for what style of dance we should be doing. They need to be flexible and capable of withstanding the inevitable unpredictable behaviors of complex systems. In short, we need to begin developing complex solutions for complex problems.

In order to begin converting a theoretical understanding of complex dynamics into actionable solutions, more policymakers and scientists need to be aware of the importance of managing multiple aspects of a system. We don’t currently recognize that fact when we make predictions. Fiddaman states the problem in terms of the way climate models are used to devise solutions for environmental problems: “The debate is basically framed as ‘What are we going to spend now in order to avoid a disaster at some distant point in the future?’ And if you think that there are actually lots of interesting dynamics in the global system—critical points and thresholds—then that is the wrong framing.”

What exactly these solutions should look like is still unclear, but Fiddaman has developed a climate model called C-ROADS, which takes emissions data from various countries and projects how it will impact the planet in the next century. The purpose of C-ROADS is in large part to help scientists and policymakers develop a more sophisticated sense of how emissions affect the planet. “Playing with C-ROADS helps people to understand how the dynamics of the system actually work,” he says. “The old approach has been, ‘If you do x then y should move in step.’ But that isn’t adequate to understand the complexity of these systems.” He cites the example of some global-warming critics: They look at emissions data and claim that because emissions rose between 1940 and 1970, but temperature did not, there is not a correlation between emissions and temperature. But, he explains, these critics are making the assumption that emissions and temperature must move in step with one another in order for them to be related. “That is pattern matching, and unfortunately it has nothing to do with the way the system actually works,” he says.

Over the past half-century, researchers have come to appreciate many of the dynamics of complex systems. They can, at least theoretically, identify points at which a system will undergo rapid catastrophic change and detect early warning signs indicating when such changes will occur. And their ability to interact with these systems, and circumvent potential problems within them, is beginning to catch up as well. But no matter how robust the theoretical understanding becomes, models are only as strong as the data they are fed. Fortunately, we’re entering an age of information abundance, with ever increasing means of acquiring more.

Data Is the Language
CERN’s Large Hadron Collider alone is projected to generate an astonishing amount of data once it begins physics research–level proton collisions: on the order of 15 petabytes per year (an amount equal to 1,500 times the entire US Library of Congress’s collection of printed material). Processing and making sense of such an unfathomably dense amount of data presents many logistical hurdles.

What is known as the “grid” began as a data-storage solution to address these challenges at the Large Hadron Collider and has now expanded into a global experimental tool for collaborating scientists. Grid computing is a technology and network concept that connects thousands of computers across the globe. But unlike the World Wide Web, which allows the sharing of only information, grid computing—sometimes referred to as “the parallel internet”—enables the integration of data-storage capacity, processing power, sensors, and visualization tools among far-flung research groups across the world. Grid technology is a standard protocol that transforms thousands of different computers into a single, massively powerful computing resource.

For scientists trying to solve extraordinarily complex problems in systems where multidimensional data from several sources must be processed, grid computing provides the processing power to run highly complex mathematical models. Grid computing also represents a leap of processing speed by several orders of magnitude: A single model simulation that would require weeks of processing on a personal computer would complete in a matter of hours on the grid. The stated goal of the European organizing body that leads the grid initiative, Enabling Grids for e-Science (EGEE), is to “enable the next leap forward in research infrastructures to support collaborative scientific discoveries.” The current infrastructure includes ongoing projects in high-energy physics, biomedicine, Earth sciences, astronomy, and finance.

Complex data sets require smarter ways to analyze, share, and combine information. Data is becoming the vernacular in which a global community of scientists converses. Without more robust data, parsed in innovative ways, the utility of complex models is limited. New ways to share and compile data across disciplines and methodologies within the global scientific community may provide the necessary infrastructure to fill the gaps of knowledge present in many real-world applications of systems thinking.

The Earth System Atlas, part of the larger research imitative known as the Global Earth Observation System of Systems (GEOSS), is a new graphical tool that currently synthesizes more than 55 data sets from research on the Earth’s physiography, atmospheric constituents, biochemical cycles, ecosystems, and human dimensions. Combining both real-time and historic data, the Atlas acts as single portal for monitoring the planet’s complicated interactions. For instance, an oceanographer, an ecologist, and a policymaker can collaborate to identify and solve factors contributing to the collapse of an overharvested marine population, each working off the same repository of multidimensional data sets. Data becomes the common language, spoken across diverse fields and interests with a broader scientific dialogue focused on anticipating elements of a larger, interconnected system. In this way, the data itself defines a new set of global change researchers.

According to the project’s director, Stephen Reid, most of these data sets have not previously been compared. His goal is to make it possible to combine these sets in a way that allows researchers to recognize large-scale patterns in order to anticipate future trends. The project also has a policy-centric dimension. The idea is that predefined parameters—such as those outlined by the Intergovernmental Panel on Climate Change on emissions or land use, for instance—can be plugged in to a sophisticated batch of models that take into account as yet unforeseen interactions in the Earth system.

It’s becoming increasingly clear that systems research needs to combine sophisticated solutions with robust, multidimensional data sets such as the Earth System Atlas in order to make meaningful predictions.

A New Toolbox
It’s not just how we model, use, and collect data that’s affecting the way we understand complicated phenomena. What we consider data is changing as well.

Certain phenomena, particularly those influenced by human behavior, are difficult to measure directly. The clever workaround is to measure different, more reliable data points that are closely linked to the variables or behavior of interest. A stand-in data point, so to speak. This proxy data can come from unexpected, faraway places and at a global scale, through a technique called remote sensing. Space-based sensors continuously scan the Earth’s surface, sending back real-time images with increasingly high spatial, spectral, and temporal resolution. The widespread saturation of mobile and wireless devices has produced an unprecedented record of human exchange. This information is pregnant with potentially useful signatures that tell the story of humanity’s interaction with the biosphere, the universe, and itself. Within the burgeoning world of proxy data methodologies, nighttime light-emission data from an aging satellite network becomes a measure of GDP in developing countries where economic data is notoriously sparse; online browsing behavior is a window into the internal process of decision making; and relative regional cell phone traffic becomes a real-time indicator of the score of an international soccer match.

In 2008, when former National Science Foundation Director Rita Colwell sought to understand the spread of cholera, she and her team turned to satellite data. They identified specific environmental factors—most notably, water temperature and plankton production in local waters—that preceded outbreaks of the infectious disease in regions across the world. The predictive model they built based on this proxy data is strikingly accurate. Not only does it reliably anticipate the spread of cholera in remote areas of the developing world, where traditional measurements are fraught with difficulties, but the model also represents an important fundamental shift in epidemiology: from viewing infectious disease as spreading through transmission-based migration to an emergent phenomenon reliant on environmental conditions.

When dealing with social systems, the indirect qualitative methods for traditionally collecting data have inherent limitations. In surveys, for instance, people regularly tend to forget or misrepresent their behavior when queried about how they act in specific situations. Direct observation is slow and labor intensive, thereby making it impractical in many large-scale samples necessary for meaningful conclusions.
Brought to the personal level, a whole new ecology exists in the digital landscape. In our everyday interaction with digital infrastructures (carrying around and communicating with mobile devices, withdrawing money from ATMs, making online purchases), we leave behind a wealth of valuable information about our preferences, habits, and personal attributes. These digital footprints, combined with clickstream data (or online browsing behavior), represent a boon for social scientists in the form of massive data sets, often in real time. Proxy data as a whole represents a new toolbox for measuring the sometimes erratic nature of human behavior, reinvigorating the social sciences in ways that are just now becoming clear. We may never get to the point of knowing exactly what will happen in complicated social networks. But with human behavior such a vital component to predicting real-world outcomes, substitute data points that accurately capture this element bring decidedly closer the reality of knowing sooner and predicting some of the dramatic transitions that define our complex world.

Scientific Thinking in a Real World
The pursuit of knowing sooner may be an emerging hallmark of our age. But what does this increased prescience give us as a society? How does identifying warning signals change the equation? From preventive, truly personalized health to avoiding natural disasters, and from creating a sustainable global economy not vulnerable to the systemic risk that has defined the past century to shifting the planet-threatening trends of global climate change, early knowledge offers a veritable panacea to many of the real-world problems facing humanity. Nuanced, system-aware models, combined with more sophisticated data parsed in increasingly meaningful ways, make up a 21st century tool to be wielded by policymakers and decision makers, as well as citizens around the world.

The challenge lies in continuing to move these predictive models to the tangible veracity of the real world.

During the United Nations Climate Change Conference in Copenhagen in December 2009, a special version of the C-ROADS model—known as the “common platform,” especially tailored to decision makers—played a role in the delegates’ negotiations. Fiddaman’s models were used to drive what was dubbed the “Climate Scoreboard”—a simple online tool that displayed the significance of the proposals of the 12 nation and group delegates in concrete terms: their effect on the global temperature by the year 2100. The tracker, which was publicly available to anyone interested, updated continuously in real time while negotiations were occurring. For instance, as Japan’s position on emissions mitigation changed over the first weekend of talks, one could see the 2100 temperature rise slightly. When consensus began to form on the Reduced Emissions from Deforestation and Forest Degradation project, it had a positive effect on the temperature projection before the deal was even completed. And so on. In the end, the sum of the deals brokered over the summit resulted in a projected temperature increase over the next century of about 3.9˚C; a modest improvement from the projected temperature increase of 4.8˚C if the status quo were maintained, but significantly off the +1.5˚ mark that is widely accepted as the highest acceptable shift in global temperature change for our planet’s continued habitability.

Whether the C-ROADS climate scoreboard played any meaningful role in the actual negotiations is debatable. But it’s clear that having the capacity to accurately predict the utility of proposed policy—whether it be domestic legislature or multilateral agreements—in real time while discussions are ongoing, opens the door for an entirely new way to enact policy. It also gives advocacy groups, journalists, and constituents the ability to support or oppose such policy in time to have a greater effect on its course. US Senator John Kerry now uses the C-ROADS climate simulator on his laptop during climate negotiations. “This works, it is important, and it is already getting broad dissemination,” he said, during a meeting with the American Meteorological Society in Washington, DC in November 2009.

The practice of science has always been grounded in predicting outcomes. The hypothetico-deductive method—the due process of scientific inquiry—can be summed up by four basic steps: review data, make prediction, test, repeat. Now the ways in which we as a society are extracting information from large-scale events and systems, identifying patterns, and making predictions are clear examples of the analytical logic of science—what might be referred to as scientific thinking—transferring to the organizational principles of the public at large. In this way, scientific thinking is a nascent tool for policymaking, governance, and problem solving in general.

Our world is not the stable workhorse we once presumed it to be. Financial markets are inherently volatile; seemingly healthy ecosystems can collapse suddenly; the favorable window of life-supporting conditions that humans currently enjoy is an anomaly in the cosmic history of the planet. Change, sometimes in the form of radical, transformative shifts, is the defining characteristic of our existence. It’s how we anticipate these changes—the ways we know sooner—that will ultimately determine our long-term success in reacting to them.

Originally published December 6, 2010

Tags biology climate data economics

Share this Stumbleupon Reddit Email + More


  • Ideas

    I Tried Almost Everything Else

    John Rinn, snowboarder, skateboarder, and “genomic origamist,” on why we should dumpster-dive in our genomes and the inspiration of a middle-distance runner.

  • Ideas

    Going, Going, Gone

    The second most common element in the universe is increasingly rare on Earth—except, for now, in America.

  • Ideas

    Earth-like Planets Aren’t Rare

    Renowned planetary scientist James Kasting on the odds of finding another Earth-like planet and the power of science fiction.

The Seed Salon

Video: conversations with leading scientists and thinkers on fundamental issues and ideas at the edge of science and culture.

Are We Beyond the Two Cultures?

Video: Seed revisits the questions C.P. Snow raised about science and the humanities 50 years by asking six great thinkers, Where are we now?

Saved by Science

Audio slideshow: Justine Cooper's large-format photographs of the collections behind the walls of the American Museum of Natural History.

The Universe in 2009

In 2009, we are celebrating curiosity and creativity with a dynamic look at the very best ideas that give us reason for optimism.

Revolutionary Minds
The Interpreters

In this installment of Revolutionary Minds, five people who use the new tools of science to educate, illuminate, and engage.

The Seed Design Series

Leading scientists, designers, and architects on ideas like the personal genome, brain visualization, generative architecture, and collective design.

The Seed State of Science

Seed examines the radical changes within science itself by assessing the evolving role of scientists and the shifting dimensions of scientific practice.

A Place for Science

On the trail of the haunts, homes, and posts of knowledge, from the laboratory to the field.


Witness the science. Stunning photographic portfolios from the pages of Seed magazine.

Sites by Seed Media Group: Seed Media Group | ScienceBlogs | Research Blogging | SEEDMAGAZINE.COM