ACMMM25 - Grand Challenge on Multimedia Verification

Duc-Tien Dang-Nguyen1, Morten Langfeldt Dahlback2, Minh-Son Dao3, Anh-Duy Tran4, Sohail Khan1, Kha-Luan Pham5, Michael Riegler6, Pål Halvorsen6, Marc Gallofre Ocana2, Henrik B. Vold7, Silje Førsund2, Minh-Triet Tran8,
1University of Bergen 2Faktisk.no 3NICT 4KU Leuven 5Aalto University 6Simula Research Laboratory 7Institutt for Journalistikk 8University of Science - VNUHCM
Registration Code

Main Task

Task Description

Objective: Verify the authenticity and context of multimedia content, which may be in a language other than English, and provide both a detailed report for fact-checkers and a concise summary for general readers. The languages of the content might not be in English.

Input: Each case/task will provide (in a zip file):

  • Multimedia Content: Image(s) or video(s)
  • Associated Context: Captions, descriptions, social media posts, news articles, metadata (if available)
  • Additional Clues: Possible sources, claims, or fact-checker notes (if relevant)

Output: A verification report (as a text file) in English. The report should include the following key information:

  • Summary of Key Points: Provide a concise overview of the content, including relevant details. Clearly highlight any uncertainties and underline what is not yet known.
  • Content Category: Assign relevant tags based on platforms, people, brands, or specific topics.
  • Forensic Analysis Results:
    • Authenticity: Determine if the content is synthetic, modified, or recaptured.
    • Tools & Methods Used: Specify the verification tools and techniques applied.
    • Synthetic Type (if applicable): Identify whether it was generated using GANs, Stable Diffusion, or other AI models.
    • Other Artifacts: Note any detected anomalies or manipulations.
  • Verified Evidence: State what can be confirmed about the video/photo based on available evidence.
    • Source Details: Where the content comes from (URLs, original posts, etc.)
    • Where? (Location)
    • When? (Time)
    • Who? (People, organizations, entities involved)
    • Why? (Possible motivations or intent)
  • Other Evidence & Findings: Any additional relevant information, supporting materials, or external sources

Note: For each evidence finding, specify the failure type if verification fails: Indeterminate (insufficient or ambiguous data), Inconclusive (attempted but no definitive result), or Not Feasible (limited expertise or tools).

Sample Dataset

You can find the sample dataset here

The Main Task dataset may contain sensitive or potentially disturbing content, as it reflects real-world events and includes verification reports from fact-checking organizations. Participants are advised to exercise caution when viewing these media files.

Competition Stages

  • Stage 1 - Training (March - April)
    • Organizers provide 100 50 known cases with input and expected outputs.
    • Participants register, explore tasks, and practice verification.
  • Stage 2 - Validation (May)
    • Organizers release 20 10 new cases with input only.
    • Participants must submit verification reports and describe their methodology. Participants may also submit a scientific paper detailing their approach.
    • Submissions are evaluated, and only those who successfully pass the validation stage will qualify for the final competition.
  • Stage 3 - Real-World Verification (July 28 - August 8)
    • Only validated participants advance to this stage.
    • Organizers provide 10 real-time cases, reflecting ongoing misinformation challenges.
    • Participants verify cases, submit results, and optionally submit a camera-ready paper.

Note: To ensure the quality and accuracy of the cases, we have reduced the number of samples and only provide the high-quality ones.

OOC Subtask: Out-of-Context Detection

Task Description

In this task, participants are asked to come up with methods to determine whether a given image-caption pair is genuine (real) or falsely generated (fake). More specifically, given an <Image, Caption> pair as input, their proposed model should predict corresponding class labels 0 (real) or 1 (fake).

We acknowledge that this is a challenging task without prior knowledge of the image origin, even for human moderators. In fact, Luo et al. have verified this challenge with a study on human evaluators, who were instructed not to use search engines, where the average human accuracy was only around 65%.

Dataset

For this challenge, an augmented version of the COSMOS dataset will be used. A part of this dataset is sampled and assigned as the public dataset. The public dataset, consisting of the training, validation, and public test splits, is provided openly to participants for training and testing their algorithms. The remaining part of the COSMOS dataset is augmented with new samples and modified to create the hidden test split. The hidden test split is not made publicly available and will be used by the challenge organizers to evaluate the submissions.

You can disregard caption2 for this task and focus only on caption1 from the public test set. In NOOC (Not-Out-of-Context) cases, caption1 forms a contextually accurate image-text pair. In contrast, in OOC (Out-of-Context) cases, caption1 is intentionally incorrect, resulting in an out-of-context image-text pair. The private test set will also have only 1 caption per image. Same as in the public test set.

Challenge Dataset Statistics

Dataset Split Number of Images Number of Captions Context Annotation
Training 161,752 360,749 No
Validation 41,006 90,036 No
Public Test 1,000 2,000 Yes
Hidden Test 1,000 2,000 Yes

Resources

This section provides a list of useful resources for OSINT-based multimedia verification.

Useful Links

References

  • Dang-Nguyen, D.T., Khan, S.A., Riegler, M., Halvorsen, P., Tran, A.D., Dao, M.S. and Tran, M.T., 2024, May. Overview of the Grand Challenge on Detecting Cheapfakes at ACM ICMR 2024. In Proceedings of the 2024 International Conference on Multimedia Retrieval (pp. 1275-1281).
  • Khan, S.A., Dierickx, L., Furuly, J.G., Vold, H.B., Tahseen, R., Linden, C.G. and Dang‐Nguyen, D.T., 2024. Debunking war information disorder: A case study in assessing the use of multimedia verification tools. Journal of the Association for Information Science and Technology.
  • Khan, S.A., Sheikhi, G., Opdahl, A.L., Rabbi, F., Stoppel, S., Trattner, C. and Dang-Nguyen, D.T., 2023. Visual user-generated content verification in journalism: An overview. IEEE Access, 11, pp.6748-6769.
  • Boididou, C., Middleton, S.E., Jin, Z., Papadopoulos, S., Dang-Nguyen, D.T., Boato, G. and Kompatsiaris, Y., 2018. Verifying information with multimedia content on Twitter: a comparative study of automated approaches. Multimedia Tools and Applications, 77, pp.15545-15571.
  • Boididou, C., Andreadou, K., Papadopoulos, S., Dang Nguyen, D.T., Boato, G., Riegler, M. and Kompatsiaris, Y., 2015. Verifying multimedia use at MediaEval 2015. In MediaEval 2015 (Vol. 1436). CEUR-WS.