About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Publication
SEMANTiCS 2024
Workshop paper
Automated Generation of Competency Questions Using Large Language Models and Knowledge Graphs
Abstract
This research presents a novel approach to automated competency question generation by integrating Large Language Models (LLMs) with Knowledge Graphs (KGs), particularly within the context of sustainability assessment standards like BREEAM. The study develops a comprehensive methodology combining natural language processing and knowledge representation to address the challenges of manual question generation in competency-based assessments. The methodology begins with text extraction from BREEAM standards, followed by preprocessing, transformation into graph documents, and the construction of a structured KG. Advanced LLMs, including GPT-4o and Mistral, are employed to generate competency questions based on entity-specific and community-focused retrieval methods. The system is rigorously evaluated using quantitative metrics such as cosine similarity scores and qualitative assessments using the "LLM-as-a-Judge"method. Results demonstrate that GPT-4 and Mistral models generate highly relevant, clear, and complex questions, highlighting the potential for scalable, domainspecific competency assessments. This research opens avenues for improving AI-driven educational technologies and personalised learning through automated, adaptive assessment tools.