Predicting Global World Events 🌎 (Actually)
So essentially,
MIRAI: Where LLMs can transform into geopolitical fortune-tellers, predicting the world's next move one event at a time.
Paper:
MIRAI: Evaluating LLM Agents for Event Forecasting (66 Pages)
Github:
https://mirai-llm.github.io/
Researchers from University of California, Los Angeles and California Institute of Technology are interested in predicting global events. Global event forecasting is crucial for stakeholders to understand geopolitical developments and make informed decisions, mitigate risks, and capitalize on opportunities in a globalized world. It’s a simple necessity.
Hmm..What’s the background?
Traditionally, experts in international relations have relied on their domain expertise to forecast events like conflicts, collaborations, or alliance shifts. This approach involves analyzing complex interactions between nations, considering factors like alliances, trade agreements, ideologies, and historical rivalries.
While data-driven neural networks have emerged as an alternative forecasting method, they rely on singular information types, like structured knowledge graphs or textual datasets. Knowledge graphs, though structured, can be incomplete or biased. Textual analyses might lack factual grounding for accurate predictions. Both approaches lack the ability to ground their reasoning in historical evidence, hindering the interpretability and validation of their forecasts.
Ok, So what is proposed in the research paper?
Large language models (LLMs) offer a potential solution by mimicking human experts and utilizing various tools to process information from diverse sources. LLMs can grasp international relations nuances, reason through intricate relationships with linguistic explanations, and plan their tool usage effectively.
Despite the potential of LLMs in event forecasting, there is a lack of standardized benchmarks to evaluate their performance within international events. To bridge this gap, researchers have introduced MIRAI (Multi-Information FoRecasting Agent Interface), a benchmarking environment designed to evaluate and advance the ability of LLMs to forecast international events over time.
MIRAI uses real-world data from the Global Database of Events, Language, and Tone (GDELT), adapting it into an event-forecasting task format across different time horizons to provide a robust assessment of LLM performance. The environment allows LLMs to interact with relational and textual databases through application programming interfaces (APIs), facilitating autonomous information gathering, processing, and contextually relevant application. Here were some key observations from current benchmark:
GPT-4o Excels at Event Forecasting: Of all the LLMs tested, GPT-4o consistently demonstrated the strongest performance in forecasting international events, achieving the highest scores across various metrics, including precision, recall, and F1 score.
"Code Block" Tool-Use Benefits Stronger LLMs: While providing more flexibility, the "Code Block" tool-use strategy, which allows for writing multi-line code snippets, proved more beneficial for stronger LLMs (like GPT-4o and GPT-4-turbo) compared to weaker models
Tool Use is Crucial for Accurate Predictions: LLMs that actively used tools to access and analyze historical data consistently outperformed models relying solely on their internal knowledge.
What’s next?
The authors outline several avenues for future work for MIRAI:
Expand LLM Evaluation: The current research primarily focuses on a few representative LLMs. Future work could assess a wider range of LLMs, particularly open-sourced models
Enhance API Functionality: The existing API primarily offers basic functions like counting, listing, and generating statistical distributions of events and news. Future enhancements could incorporate more sophisticated tools, such as time series analysis functions, to enable LLMs to analyze temporal trends more effectively
Conduct More Extensive Testing: Due to cost and time constraints, the current experiments involve a limited number of rounds, leading to potential variance in the results
Explore Advanced Agent Architectures: While the current research utilizes a ReAct-style agent, future work could investigate the effectiveness of alternative LLM agent architectures for event forecasting
So essentially,
MIRAI: Where LLMs can transform into geopolitical fortune-tellers, predicting the world's next move one event at a time.