Enhancing educational support for JetBrains MPS with a retrieval-augmented LLM chatbot: A structured knowledge integration approach.

Abstract

Model-Based Software Engineering (MBSE) with JetBrains MPS is challenging primarily because language engineering goes beyond using programming languages to designing them—working with meta-concepts, generators, and composition—so the learning curve is steep even with detailed documentation. We present an LLM-powered, retrieval-augmented chatbot for MPS education that combines official docs with expert-curated material, organizing both via composable graph indexes in LlamaIndex. We evaluate five configurations across two phases using the RAGAs framework along four dimensions: faithfulness, answer relevancy, context utilization, and harmfulness. Compared to a documentation-only baseline, faithfulness improves from 0.42 to 0.99; best context utilization reaches 0.71; answer relevancy remains 0.50–0.64 in the larger study; and harmfulness is as low as 0.05 (0.08 in the final configuration). These results indicate that (i) curated expert knowledge—beyond official docs—is crucial for onboarding to meta-level concepts, (ii) composable graphs materially improve grounding, and (iii) lightweight, targeted index summaries further boost reliability while remaining scalable. The approach generalizes to other MBSE tools where steep learning curves limit adoption, and we provide code and configuration artifacts to facilitate replication and classroom use.