ACMMM25 - Grand Challenge on Multimedia Verification

Main Task

Overall Evaluation

Main score on each task:

Score = max(0, Q - w.T²)

• Q: Quality score assessed by professional fact-checkers (see details in Section Quality Evaluation).
• T: Time - number of hours after the task release.
• w: Weight - controls the penalty. (w = 0.001)
• Simply put, the score decreases proportionally to the square of the time spent.

Report Evaluation (Quality Q)

1. Summary & Content Classification (10%)
   • Concise Overview – Clearly summarizes the findings, highlighting uncertainties and unknowns.
   • Correctly Categorized – Correctly assign relevant tags based on platforms, people, brands, or specific topics (e.g., TikTok, Trump, Coca-Cola, Ukraine War, or AI-generated).
2. Verified Evidence (50%)
   • Source Details – Identifies where the content originates (e.g., URLs, original posts, and metadata).
   • Where? (Location) – Determines the correct geographical context.
   • When? (Time) – Establishes the accurate timeframe.
   • Who? (People, Organizations, Entities Involved) – Identifies key individuals or groups.
   • Why? (Motivation or Intent) – Provides a reasoned explanation of possible intent.
3. Forensic Analysis (15%)
   • Authenticity Assessment – Determines if the content is synthetic, modified, or recaptured.
   • Verification Tools & Methods – Clearly documents the tools and techniques used.
   • Synthetic Type (if applicable) – Identifies AI-generated content (e.g., GANs, or Stable Diffusion).
   • Other Artifacts – Notes any detected anomalies or manipulations.
4. Other Evidence & Findings (15%)
   • Supporting Sources – Uses additional fact-checks, reports, or metadata to back claims.
   • Cross-Checking Information – Ensures verification through multiple independent sources.
5. Clarity & Structure (10%)
   • Well-Organized Report – Logically structured for readability.
   • Concise & Understandable Language – Avoids unnecessary complexity or ambiguity.

Note: Not all points may be verifiable in every case. Clearly stating the failure type ("indeterminate", "inconclusive", or "not feasible") is a valid verification outcome and should be included where necessary.

Verification Summarization

The summarization subtask will be evaluated by a jury committee based on the following criteria:

1. Clarity: The summary should be well-structured, easy to read, and clearly convey the verification findings.
2. Conciseness: The summary should be brief while still covering the essential points of the verification.
3. Readability: The language should be accessible to a general audience, avoiding technical jargon.
4. Accuracy: The summary should correctly reflect the key findings from the detailed verification report.

The jury committee will assess submissions to ensure they effectively communicate the verification findings in a clear, concise, and accessible manner.

OOC Subtask: Out-of-Context Detection

Evaluation Criteria

Participants' models will be evaluated and ranked according to two aggregate scores:

• Effectiveness: accuracy, precision, recall, F1-score, and Matthews correlation coefficient (MCC). Participants are asked to calculate these 5 metrics for their model and include the values in the results section of their submission.
• Efficiency: latency, number of parameters, and model size. Participants are asked to calculate these 3 metrics for their model and include the values in the results section of their submission.

After the participants evaluate their own models on the public test split, they are asked to provide code and trained model weights to the organization committee, in order for their models to be evaluated on the hidden test split. Participants are allowed to submit their solutions in three alternative ways as described in Section Submission Guidelines, provided that they abide by the deadlines listed in Section Important Dates.