01 / MOTIVATION
Why this problem matters
For people affected by a crisis, inconsistent responses can undermine confidence: one answer may explain recommendations professionally while another provides concrete actions but lacks context. The work examines not only the quality of each reply but also variation across replies to different needs.
Combining strengths from multiple generated answers to produce crisis responses that are consistently professional, actionable, and relevant.
02 / METHODOLOGY
How the approach works
Experiments draw on social-media data from Hurricanes Harvey, Irma, and Maria. The authors identify 540 need-related posts for response generation and compare Llama and Mistral.
- 01
Detect crisis-related information needs from social media posts.
- 02
Generate candidate answers using instructional prompting and retrieval-augmented generation.
- 03
Measure professionalism, actionability, and relevance; define consistency from variation in these dimensions.
- 04
Compare candidate responses and fuse their strongest elements using evaluation scores and structured instructions.
- 05
Evaluate weighted fusion against prompting, RAG, RAG with prompt engineering, and selection baselines.
The stages summarize the published framework. Consult the linked paper for the original figures and technical implementation.
03 / RESULTS
Experimental comparison
| Llama-8B method | Professionalism ↑ | Actionability ↑ | Overall quality ↑ | Consistency ↑ |
|---|---|---|---|---|
| Instruction prompt | 0.74 | 0.52 | 0.66 | 0.76 |
| RAG | 0.96 | 0.63 | 0.80 | 0.84 |
| Prompt and select | 0.50 | 0.98 | 0.75 | 0.78 |
| Fusion w/ evaluation | 0.98 | 0.77 | 0.86 | 0.87 |
| Fusion w/ eval + instructions | 0.92 | 0.99 | 0.92 | 0.91 |
| Weighted fusion (ours) | 0.99 | 0.99 | 0.95 | 0.94 |
Selected rows of Table 1, Llama-3.1-8B-Instruct. Means only; relevance excluded from this compact table because the paper reports approximately 0.79–0.80 across these methods.
04 / DISCUSSION
What the results tell us
For Llama, weighted fusion reaches 0.95 overall quality and 0.94 consistency; it balances professionalism and actionability instead of optimizing one at the expense of the other. The paper also evaluates Mistral, with the weighted fusion method achieving 0.95 overall quality and 0.92 consistency.
For Llama, weighted fusion reaches 0.95 overall quality and 0.94 consistency; it balances professionalism and actionability instead of optimizing one at the expense of the other. The paper also evaluates Mistral, with the weighted fusion method achieving 0.95 overall quality and 0.92 consistency.
Style consistency is evaluated through defined communication metrics and automated scoring, with human validation. These scores do not directly prove changes in crisis outcomes, and high consistency should not erase differences between users' needs.
Explore the full study
Read all the experimental results, figures, and detailed evaluation from the primary paper.