Rybinski, Fabian

Large Language Models (LLMs) are increasingly being used in Business Process Management (BPM) to create, analyze, and interpret process models. Current research shows that the form of representation of a process model—such as a textual description, a rendered diagram, or structured serialization—has a significant impact on the quality of interpretation by LLMs. However, these studies keep the underlying modeling language constant and vary only the representation of the same model.

In practice, however, there is a wide variety of process modeling languages—ranging from Petri nets to UML activity diagrams to BPMN. Each language has its own syntactic constructs, formal semantics, and expressive power: Petri nets offer a mathematically precise execution semantics, while BPMN provides a comprehensive, practice-oriented catalog of elements. It remains unclear whether and how the choice of modeling language influences LLMs’ understanding of processes and whether this effect depends on the type of task (e.g., structural questions, trace validation, deadlock detection).

This thesis will therefore systematically investigate how different process modeling languages, when used as input for LLMs, affect the understanding of process models. The target is to determine, through controlled experiments with substantively equivalent models in different languages, which modeling languages enable the best LLM understanding for which analysis tasks.

The thesis will cover the following steps:

Literature Review: First, an initial literature review will be conducted to identify existing approaches in the LLM-based division of process model analysis, as well as the characteristics and comparability of common process modeling languages. This will also involve researching available datasets as well as methods for semantically preserving conversions between modeling languages.

Analysis: The selected modeling languages will be examined and evaluated based on defined criteria to assess their suitability as LLM input. In particular, it must be ensured that the compared models are information-equivalent, so that any observed differences can be attributed to the language rather than the content.

Design: Based on the analysis, an experimental framework will be developed that enables the systematic evaluation of process model understanding across different modeling languages. In this process, suitable task types and metrics will be developed to operationalize different levels of understanding (syntactic, semantic, inferential).

Implementation & Evaluation: Finally, a pipeline will be implemented to enable the execution of experiments with various configurations. The results will provide insights into which modeling languages are particularly suitable for LLM-based process analysis and what implications this has for the choice of modeling language in practice.

If you are interested, please apply via the following link: https://portal.wiwi.kit.edu/forms/form/Bewerbung_Abschlussarbeit_AIFB-BIS