HEGEMON: Holistic Evaluation of Generative Foundation Models in the Security Context

Funding: Agentur für Innovation in der Cybersicherheit GmbH (Cyberagentur) through pre-commercial procurement  
Project duration: 36 months

 

HEGEMON investigates the evaluation and adaptation of generative foundation models for security-relevant applications. The project focuses on geoinformation and develops methods for assessing and using language, vision, and multimodal models in this field.

 

The project combines three geoinformation use cases with a holistic benchmarking framework. The use cases cover the generation of geospatial dossiers, the extraction and generalisation of vector data, and multimodal question answering based on maps and remote-sensing data. They provide practical requirements and evaluation scenarios for the benchmark.

Our contribution: holistic benchmarking

The UniBw AIML group leads the research and conceptual development of the overarching holistic benchmarking framework. Its purpose is to assess whether foundation models are suitable for security-relevant applications. The framework evaluates models across multiple dimensions:
  • Technical performance and robustness
  • Security
  • Trustworthiness
Our work includes analysing existing evaluation approaches, identifying research gaps, and defining evaluation criteria and metrics for each dimension. We select and develop suitable benchmarks, establish baselines, and investigate methods for normalising and aggregating heterogeneous evaluation results.
 
The framework brings results from the different evaluation dimensions together into a holistic assessment while keeping individual strengths and limitations visible.

Our aim is to provide an overarching framework for the holistic and reproducible evaluation of foundation models. The framework brings the different evaluation dimensions together and supports a transparent analysis of model capabilities, limitations, and trade-offs.
 
Project context
 
HEGEMON is funded by the Cyberagentur. The project brings together the Artificial Intelligence and Machine Learning group and the Earth Observation group at the Universität der Bundeswehr München with dida Datenschmiede GmbH.