Using Data-Driven Crisis Analytics to Drive Processes, Impact, and Scale

(Illustration by Vanessa Lovegrove) 

In the humanitarian aid community, research methods have traditionally skewed toward the qualitative: Participant interviews, focus groups, and field surveys have been the predominant tools determining context-specific interventions. With advances in data science, however, aid organizations have been able to supplement these evidence-driven methods with quantitative ones.

For global humanitarian NGO Mercy Corps, supporting communities affected by crises is just the first step. Finding the means to sustain livelihoods and become more resilient to future upheaval requires longer-term, more complex and dynamic solutions. What a humanitarian organization like Mercy Corps needs, then, is the ability to gather information and data as well as the ability to synthesize and analyze the data in a continuously evolving environment.

Mercy Corps is committed to investing in and improving our capacity to apply quantitative data to drive program processes, impact, and scale. This approach extends beyond our work in crisis analytics and immediate humanitarian response to medium- and longer-term programmatic design and implementation. Through our 10-year global strategy, Pathway to Possibility, Mercy Corps has committed to be an evidence- and data-driven organization that will first focus on the foundational components of people, culture, structure, and systems as the groundwork for an overall strategy. We now know that proper collection, processing, and analysis of large amounts of data have the potential to improve our programs around the world.

When it came to finding the right technologies to accomplish such complex tasks, it was essential to our success that we had the dedicated resources and strong partnerships. Internally, our technology for development team helps our program teams leverage technology to improve their performance and quality. We also have a global crisis analytics team dedicated to in-depth analysis to support our humanitarian operations in the world’s most complex operating environments.

Externally, partnering with AWS has allowed us to leverage cloud architecture and products to facilitate advanced data storage, sharing, and analytics, as well as to have a thought partner with a vision for transforming the humanitarian and disaster-response sector. Designating ownership, leadership, and staff across both organizations, furthermore, allows us to take more risks in data-driven transformation.

A Blended Approach

“Crisis analytics” is the umbrella term used in the humanitarian sector to describe how an organization generates, aggregates, and analyzes data to improve time-sensitive decision-making. It can aid us in understanding complex contexts where crisis occurs at the nexus of political conflict, climate, and economic turmoil. While it is typical of our sector to explain these crises qualitatively, the dynamic interactions between sectors in the communities we serve require more focused, quantitative analyses. Through our work with AWS, Mercy Corps has sought to embed and build data science as a core practice, focusing on designing cloud architecture for agency-wide crisis and climate analytics. Specifically, Mercy Corps expanded our use of crisis analytics to strengthen and deepen our capacity to use quantitative data curation, collection, and analysis.

The strategy has already shown tremendous promise, particularly in our work in Syria, where we employed this new blended approach. Using qualitative methods, we performed informant interviews with those who had critical insights into what was happening in the nation. Teachers, business owners, and doctors created a picture of how the financial crisis affected their daily lives. Because of the high volatility of currency exchange rates, we added economic forecasts to model the affordability of essential goods and inform our country team about potential price fluctuations. The country team then used this information to plan for inflation. It was through quantitative methods that price affordability could be forecast to improve programming, as households dealt with shocks to their financial and food security.

To assist in this work, Mercy Corps has crisis analytics teams in 11 countries. These analysts aggregate and analyze data to improve decision-making in crisis situations. At a global level, the teams identify similarities across operations and apply both quantitative and qualitative data collection and analysis to understand situational challenges. For example, currency fluctuations can have harmful effects on household consumption and food security, threatening livelihoods and economic growth. Using Syria as a first test case in leveraging cloud technology, Mercy Corps developed a platform for price forecasting, a model that provides lessons that can be replicated in other geographies with similar challenges. Because of similarities in process and in situational challenges, our experience leveraging cloud technology for price forecasting in Syria provides lessons others can apply to increase their impact.

Based on our early success with this process, we have learned that a precise data collection strategy—especially the introduction of automated solutions for collecting, cleaning, processing, and integrating data from several sources—is a crucial first step. Ideally, this process includes a component that incorporates a country-specific understanding of data trends, patterns, and anomalies. This data strategy should also include processes and principles for secure data storage. An organization employing this strategy will also benefit from thinking about the data analysis that needs to take place through visualizing, analyzing, and interacting with data.

In working to support our crisis analysts, we discovered that we needed to tackle the challenges of data management across a large, global organization. To do so, we needed to build a cloud-based, data-science practice to bring together fragmented data from around the world into a shared space.

This transformation from a largely qualitative approach based in literature and field interviews to one that incorporates data and forecasts has opened a world of possibilities for our humanitarian work. It has accelerated our ability to scale solutions for entire communities and regions based on analysis of real-time data, including the affordability of commodities such as bread, meat, fruit, and vegetables in different zones in Syria. 

Forecasting Affordability

To test this new approach, we embarked on an experiment to model people’s ability to afford their most basic needs amid erratic price swings and currency instability in Syria. Since late 2019, the price of a food basket—the monetary threshold required for a family to meet its nutritional needs—has increased 250 percent. The concept of a food basket allows organizations to gauge how much they need to spend to provide support to families. The affordability of the basket is determined by wages and costs. When costs rise faster than wages, people must work more hours to afford the basic necessities. The basket becomes unaffordable when families are unable to work enough hours in a day to afford their minimum needs.

The ability to forecast wages and the price of a food basket over a three-to-six-month period is critical information for humanitarians, who must allocate limited resources efficiently. In this case, this information was lacking. Relevant data available to our team was fragmented, varied across formats, sources, and quality, and the collection was often inconsistent.

Mercy Corps worked with AWS to design a cloud-based approach to aggregate contextual data into a shared space, making it possible to access normalized data (i.e., data that has been standardized across all records) about the economies, populations, conflicts, climate dynamics, and food security in Syria.

For our Syrian test case, we were able to pull data into the cloud from two different sources: the World Food Programme’s Global Food Price data-set website, which is hosted by the United Nations Office for the Coordination of Humanitarian Affairs (UNOCHA) Humanitarian Data Exchange; and a second set of internal data on individual markets compiled by a Syrian analyst. With AWS, we built a model that analyzed historic patterns of commodity prices and forecasted their affordability in just four months. This work leveraged cloud-based predictive analysis tools to better understand future cash flows for food security at the household level. 

This model predicted the price of food basket items with an average accuracy of 88 percent. This ability has helped our Syria team to better target emerging crises and use data-driven information to advocate, plan for, and implement programs and interventions such as increasing supply-side production methods to encourage domestic production to protection programming around coping mechanisms (like skipping meals) taken up by households.

The promise of data-driven crisis analytics allows us to reinforce what our experts see and experience with large-scale, real-world supporting evidence.

Our Syria experience shows that when looking to apply cloud technology to crisis analytics, narrowing the scope to a clear use case leads to the best results. We are currently expanding this approach to more than 40 countries. We did not attempt to build a top-down, organization-wide data model to address every challenge. Instead, we took the approach of working backward from a single challenge and asked how we might predict the number of people who would need humanitarian food aid in Syria.

From the insights gleaned from this case study, we pursued data projects with other teams encountering challenges or data-analysis needs that could benefit from cloud technologies and predictive analytics. Over time, we have developed a broad understanding of the common data needs of our teams.

Lessons Learned

Data science and cloud technology are providing Mercy Corps with powerful new tools to balance qualitative and quantitative evidence. As we have developed our understanding of our organizational data needs, we have learned three important lessons:

• Identify narrow test cases and build a foundational, small core team to develop the data pipelines that service those cases.
• Staff the analysts as geographically proximate to the problem as possible. Data have country-specific patterns. Keeping analysts as close to complex problems as possible enables faster learning and greater adoption of new models.
• Leverage existing cloud tools before building new ones. Experimenting with standard services that minimize development costs helps prototype the systems that follow in later stages of the maturation curve.

Since our Syria study, Mercy Corps has sketched the data-science infrastructure necessary to scale our analytics efforts. This work has focused on two milestone projects: a data catalog standardizing our data pipelines, which has enabled us to build a shared data catalog where analysts can access data sets from more than 40 countries, and modeling infrastructure that uses cloud-based machine-learning tools to analyze historical trends and forecast probable futures.

We expect this work will make Mercy Corps programs more effective at the junctures that often matter the most: anticipatory action and adaptive program management. The promise of data-driven crisis analytics allows us to reinforce what our experts see and experience with large-scale, real-world supporting evidence. It means that we can determine if what we perceive about a crisis situation is in fact corroborated by data-based evidence. The data also allows for broader, more effective applications of humanitarian assistance. Because of the power of the data and analytics tools, we can do our work with more confidence and efficiency. We are now able to uncover and gather facts in an organized way to illuminate crises and trends. With this data-driven approach, Mercy Corps is effectively using information to advocate for resources and programming.

We have recently implemented cloud-based solutions gathering, analyzing, and modeling data about the economic crises in Yemen and Lebanon, as well as the ongoing political and military conflicts in Lebanon, Ukraine, and the Democratic Republic of the Congo. We also continue to explore modeling strategies. For example, we are currently developing a system-dynamics model that illustrates the complex nature of environmental, economic, and social factors in livestock systems in Somalia, where one out of three cattle have died since mid-2021 due to severe drought.

Mercy Corps will continue to approach crisis analytics in ways that create faster, more precise, and more powerful effects with the help of cloud services. We are committed to data sharing within the broader humanitarian community to better coordinate our efforts, as well as sharing our best practices and lessons learned so that we can contribute to the strengthening of data-driven transformation across the sector. For organizations thinking about developing a data or analytics strategy, it’s important to see the potential value of data for your mission, and then take the leap to commit.

Read more stories by Carolyn Florey.