is essential? – a pilot survey on views about the requirements metamodel of reqT.org

. [ Context & motivation ] This research preview paper presents ongoing work on the metamodel of a free software requirements modeling tool called reqT that is developed in an educational context. The work aims to make an initial validation of a survey instrument that elicits views on the metamodel of the reqT tool, which aims to engage computer science students in Requirements Engineering (RE) through an open source DSL embedded in the Scala programming language. [ Question ] The research question is: Which RE concepts are essential to include in the metamodel for a requirements engineering tool in an educational context? [ Principal ideas ] A survey instrument is developed, with a list of 92 concepts (49 entities, 15 relations and 28 attributes) and a set of questions for each concept, to elicit the respondents’ views on the usage and interpretation of each concept. [ Contribution ] The survey is initially validated in a pilot study involving 14 Swedish RE scholars as subjects. The survey results indicate that the survey is feasible. The analysis of the responses suggest that many of the concepts in the metamodel are used frequently by the respondents and there is a large degree of agreement among the respondents about the meaning of the concepts. The results are encouraging for future work on empirical validation of the relevance of the reqT metamodel.


Introduction
There are many challenges in teaching Requirements Engineering (RE) [4,6], including advancing students' requirements modelling skills that can be used effectively in an unstructured, non-ideal, real-world situation [1].When teaching RE modelling we may ask ourselves: What are the essential RE concepts that we should include in a taught metamodel for requirements?This paper investigates this questions in conjunction with the on-going work of developing a metamodel for reqT.org,an open source requirements engineering tool used in RE education [7].A survey instrument is presented aiming to elicit the frequency of RE term usage and the degree of interpretation agreement.The responses from 14 Swedish RE scholars are analysed and discussed and conclusions suggest that a large subset of the concepts of the current reqT metamodel can be seen as "essential" in that a majority of the subjects use them while agreeing with the concepts' definitions.The presented work represents an initial validation of the survey instrument.Further work involving more subjects is needed to draw conclusions with more certainty.

Background
There are nowadays numerous commercial RE tools available, but many are expensive, complex and not sufficiently open [2].A major aim of the reqT open source project is to provide a small but scalable, semi-formal and free software package for an educational setting [7] that can inspire code-loving computer science students to learn more about requirements modeling.The tool development started in 2011 at Lund University, where reqT is used in RE teaching at MSc level in student role-playing projects. 1 critical issue is how to choose the essential RE concepts that allows for sufficient expressiveness, while not overloading the metamodel with esoteric concepts just for the sake of completeness.
The reqT metamodel includes three types of concepts: entities, attributes and relations.Entities and attributes are nodes in a graph data structure, while relations are edges that can connect entities with sub-graphs.Thus a tree-like structure can be created of arbitrary depth spanning the graph that models some chunk of requirements.
The code below shows a toy example of an orthogonal variability model [5] expressed in the reqT Scala-embedded DSL [7] illustrating a small part of its metamodel.Other parts of the metamodel contains concepts that enable e.g.goal modelling, use case modelling, and user story modelling, see further Appendix A.
Entities in the above code listing are in bold, attributes in italics and relations start with a lower case letter.In the reqT editor, entities, attributes, and relations are syntax-coloured in blue, green and red respectively.A reqT model written in the above syntax is actually valid Scala code that, when executed, generates a data structure that can be traversed and manipulated using Scala scripts.Visualisations can be generated by export to GraphViz.Export is also available to HTML and spreadsheet formats.

Methodology and Data Collection
In order to validate RE scholar's opinions of the metamodel, a survey instrument was developed including the 49 entities, 15 relations and 28 attributes.All concepts and definitions are listed in Appendix A. 2 The concepts were gathered from various sources including the IREB Glossary3 , Wikipedia, agile development, variability [5] and goal modelling, and the text book [3] used in an RE course at Lund Univ.The data collection was made during a Swedish national network meeting with academic RE scholars in spring 2015.The survey was filled in during the meeting using the participants' own laptops in a spreadsheet shown in Fig. 1.The subjects were given around 20 minutes to complete the survey.Most of the subjects handed in the survey via email directly after the session, while a few finished it after the meeting.

Data Analysis
Subject background.The background questions in the survey regards the role of the subject, as shown in Table 1.The analyzed 4 total number of subjects is 14, of which 10 are teachers, 10 are developers and 13 are researchers.The response rate was 100% after a reminder was emailed to one missing subject.
Frequency analysis.The degree of "essentiality" is characterized as the number of subjects that has responded that they (1) use the concept at least in an informal, nonpersistent way, and that they (2) use the concept in a similar meaning as in the definition in Appendix A. Fig. 1 shows the definitions of the three-level ordinal scales of Questions Q1 usage and Q2 meaning respectively.Table 2 shows the results of the frequency counts.If an "essentiality threshold" is chosen at N/2 then only the 9 concepts from row n = 7 and below in Table 2 are considered "non-essential", hence showing that more than 90% of the metamodel concepts have a majority of the subjects that use them and agree upon their definitions.Each concept has at least one subject that uses it and agrees with its definition.
The anonymised data and analysis scripts (developed using Scala and Apache POI) are available at: https://github.com/bjornregnell/reqT-survey2. Frequency analysis, where n is the number of subjects that for the respective concept answered (Q1usage >= 1) and (Q2meaning = 2).In total there are 92 concepts (49 entities, 15 relations and 28 attributes).The higher up in the table, the more "essential".For n = 0, 2, 3, 5 the were no concepts with answers by that number of subjects.

Discussion and Conclusion
The presented survey is a pilot investigation with two main contributions: (1) the survey instrument together with the data collection and analysis approach, which are shown to be feasible in the presented context, and (2) the pilot study results: for more than 90% of the 92 reqT metamodel concepts a majority of the 14 participating RE scholars claim to use them and agree upon their definitions.Only 1 concept was considered missing by more than one subject, while in total 19 additional concepts were reported missing by some subject.
Limitations.It can be questioned if "essentiallity" of a set of RE concepts can be characterized by how many RE scholars that use them and agree upon their definition, but it can also be argued that concept usage in an educational context is interesting to investigate when developing a metamodel for an academic RE tool.A major threat to external validity is the limited number of subjects.Due to few subjects and the high degree of homogeneity among subjects with respect to background, it is difficult to analyse and draw conclusions e.g. about potential differences in opinions between e.g.teachers and developers.Some subjects needed more time and completed their survey offline, which may give a variation in how carefully the responses were considered.
Further work.When developing a metamodel it is interesting not just to ask if the concepts to include are essential, but also to pose the question if the set of concepts is complete.If some essential concept is missing from some stakeholder's viewpoint, then the metamodel is not sufficient.With more subjects participating in the presented RE metamodel survey, the analysis of answers to further questions on alternative terms and missing concepts will be enabled and beneficial to the further development of a comprehensive and complete, but not overloaded, RE metamodel.

Fig. 1 .
Fig. 1.A screen dump of a part of the survey instrument.