ORIGINAL ARTICLE
Empirical research in requirements engineering: trends
and opportunities
Talat Ambreen1 • Naveed Ikram2 • Muhammad Usman3 • Mahmood Niazi4
Received: 10 August 2015 / Accepted: 13 July 2016
Springer-Verlag London 2016
Abstract Requirements engineering (RE) being a foundation of software development has gained a great recognition in the recent era of prevailing software industry. A
number of journals and conferences have published a great
amount of RE research in terms of various tools, techniques, methods, and frameworks, with a variety of processes applicable in different software development
domains. The plethora of empirical RE research needs to
be synthesized to identify trends and future research
directions. To represent a state-of-the-art of requirements
engineering, along with various trends and opportunities of
empirical RE research, we conducted a systematic mapping
study to synthesize the empirical work done in RE. We
used four major databases IEEE, ScienceDirect,
SpringerLink and ACM and Identified 270 primary studies
till the year 2012. An analysis of the data extracted from
primary studies shows that the empirical research work in
RE is on the increase since the year 2000. The requirements elicitation with 22 % of the total studies, requirements analysis with 19 % and RE process with 17 % are
the major focus areas of empirical RE research. Nonfunctional requirements were found to be the most
researched emerging area. The empirical work in the subarea of requirements validation and verification is little and
has a decreasing trend. The majority of the studies (50 %)
used a case study research method followed by experiments (28 %), whereas the experience reports are few
(6 %). A common trend in almost all RE sub-areas is about
proposing new interventions. The leading intervention
types are guidelines, techniques and processes. The interest
in RE empirical research is on the rise as whole. However,
requirements validation and verification area, despite its
recognized importance, lacks empirical research at present.
Furthermore, requirements evolution and privacy requirements also have little empirical research. These RE subareas need the attention of researchers for more empirical
research. At present, the focus of empirical RE research is
more about proposing new interventions. In future, there is
a need to replicate existing studies as well to evaluate the
RE interventions in more real contexts and scenarios. The
practitioners’ involvement in RE empirical research needs
to be increased so that they share their experiences of using
different RE interventions and also inform us about the
current requirements-related challenges and issues that
they face in their work.
Keywords Evidence-based software engineering
Requirements engineering Systematic review
Mapping study
& Naveed Ikram
[email protected]
Talat Ambreen
[email protected]
Muhammad Usman
[email protected]
Mahmood Niazi
[email protected]
1 Department of Computer Science and Software Engineering,
International Islamic University, Islamabad, Pakistan
2 Faculty of Computing, Riphah International University,
Islamabad, Pakistan
3 Department of Software Engineering, Blekinge Institute of
Technology, 371 79 Karlskrona, Sweden
4 Information and Computer Science Department, King Fahd
University of Petroleum and Minerals, Dhahran, Saudi
Arabia
123
Requirements Eng
DOI 10.1007/s00766-016-0258-2
1 Introduction
The degree of success and failure of a software system
depends upon the level and quality of services it provides,
as required by its users and stakeholders. Requirements
engineering (RE) is the process of eliciting, analyzing,
documenting, validating and managing these requirements.
There are a number of challenges related to each of the
sub-processes within RE such as requirements articulation
problem. These challenges and problems have motivated
researchers to carry out research in different areas of RE,
since its origin in 1990s.
RE evolved tremendously with a research span of more
than 20 years. This journey of RE research has delivered
various outcomes in terms of processes, tools, techniques,
methods and frameworks as have been reported in various
RE conferences and Journals [1, 2]. A vast amount of
research is underway in various areas of RE. New RE
researchers need to have a sound knowledge of the current
state of the RE research, covering various trends, to have
an idea of the future research opportunities in this field.
Few researchers have made attempts for providing a walkthrough of the research within the RE field. Nuseibeh and
Easterbrook [3] made the first attempt in the year 2000 for
providing an overview of the field and highlighted some
key open research issues for the future. Later, in the year
2007, Cheng and Atlee [4] presented the research directions of RE by following the same pattern as of 2000s RE
Roadmap. There was also another attempt of aggregating
RE research, in the year 2007, by Davis et al. [5], by
covering a huge amount of RE publications, focusing on
thousands of RE research papers. All of these three
attempts were beneficial in their respective contributions;
however, these attempts covered both empirical and nonempirical literature and thus lacked much evidence coming
from the empirical research [6]. This notion has also been
highlighted by many researchers [7, 8], who noted that a lot
of RE research papers just propose new solutions to
existing problems without fully validating them. Empirical
studies are significant, as they determine the real value of
the research results in any field, to present progress in that
field [9, 10]. One of the major reasons behind the lack of
empirical studies in RE is the difficulty of aggregation of
empirical results [10]. So, it will be interesting to know
‘‘what has been done empirically in RE field?’’ by considering the evidence-based paradigm [11] to aggregate RE
empirical results.
To present a state of the art of RE based on empirical RE
studies, one needs to filter an enormous collection of RE
research papers. We need to extract evidence from
empirical research reported on tools, techniques, frameworks, etc. to gain insights into broader aspects of this
field. There is also a need to present the strength of these
empirical RE studies, and presenting various trends and
opportunities of this field. Such an investigation will help
RE researchers and practitioners identify RE areas, rich in
terms of tools, techniques, frameworks, and guidelines and
areas deprived of much research. It will facilitate the
practitioners and researchers in directing them to the areas
that need their attention.
There have been few attempts by various researchers to
aggregate research studies related to specific areas of RE.
However, the focus of these surveys was limited to a
specific area of RE. Also, a large number of such attempts
involved in aggregating results related to both empirical as
well as non-empirical studies of RE (shown in Appendix
1). The individual surveys do not present the themes and
trends emerging from an overall analysis of the empirical
literature of RE. A single attempt to aggregate all the
empirical studies of RE is required to present a state of the
art of this field. The purpose of this paper is to report a
systematic mapping study that has been conducted to
aggregate empirical studies of RE up to the year 2012. This
paper presents an overall analysis of the RE field, having
various trends and hinting toward various research opportunities in it.
The main motivation of this systematic mapping study
(SMS) is to aggregate and synthesize empirical studies of
the whole discipline of RE. The research questions (RQ) of
the review include:
RQ1: What is the state of the art in empirical studies of
RE?
RQ2: What is the strength of empirical evidence in
empirical requirement engineering literature?
RQ1 aims at finding existing empirical studies in RE, to
identify trends, emerging areas and future research directions. RQ2 aims at finding the strength of evidence of
empirical research in RE by analyzing the employed
research methods, data collection techniques, and type of
participants involved. RQ1 is further divided in the following sub-questions:
• Which era of RE research has maximum progress in
terms of new advances?
• Which country is frequently involved in RE research
and in which era, these countries showed maximum
progress?
• In which context, these empirical studies of RE have
been carried out over the period of RE research?
• How empirical research in various RE knowledge areas
has evolved over the years?
• What types of interventions have been proposed or
investigated in RE research?
Requirements Eng
123
• In which channels the empirical studies of RE are
published?
• How the research method, type, intervention and
domains are related to RE core areas?
RQ2 has following sub-questions:
• Which research method is frequently used in primary
studies?
• What kind of research participants are frequently
involved in primary studies?
• Which data collection method is frequently used in the
investigation of research in primary studies?
• What is the frequency of different research types of
primary studies?
• How rigorous is the reported research in different RE
core areas?
The themes emerged from overall aggregation and
synthesis of RE studies; present various interesting results
that are of benefit to both industry and academia, in terms
of research trends and opportunities. The protocol of this
SMS has already been reported in [12], while this paper
presents the results of the SMS to present a state-of-the-art
of RE, highlighting various trends and opportunities in RE
research. The rest of the paper is organized as follows:
Sect. 2 presents background and related work; Sect. 3
describes the research process; results are presented in
Sect. 4 and further discussed in Sect. 5; study limitations
are discussed in Sect. 6; lastly Sect. 7 concludes the paper.
2 Background and related work
This section deals with a terse background of evidencebased requirements engineering and the work regarding
existing systematic reviews from the RE field.
2.1 Evidence-based requirements engineering
Since the last decade, there has been an inclination toward
evidence-based software engineering (EBSE) [11], with a
focus on systematic and empirical-based research methods.
Systematic mapping study (SMS) and systematic literature
review (SLR) are the two main tools used in EBSE. The
SLRs are performed to evaluate available literature on a
research topic in a rigorous, unbiased and auditable way
[13]. The primary studies in an SLR are evaluated more
rigorously to critically appraise the reported evidence.
SMSs, on the other hand, provide a broader overview on
a research topic, and identify and quantify the available
evidence on a research area [12]. Their findings can be
used to plan future systematic reviews and also primary
studies on the identified topics/trends [12]. In a SMS, large
number of primary studies can be included, as the evaluation and critical appraisal are not very rigorous. Moreover,
the SMS is preferred over SLR in situations when the area
is too broad [13]. Therefore, we used the SMS methodology to investigate the whole field of RE.
2.2 Related work
A number of SLRs, e.g. [14–16], have been reported in
software engineering since the introduction of the EBSE
methodology. Dealing with requirements engineering, a
number of researchers have conducted systematic reviews,
mainly focusing on some specific sub-area of RE. Dieste
and Juristo [17] performed a systematic review on
requirements elicitation techniques based on 26 empirical
studies published till the year 2005. They aggregated the
results in terms of five guidelines for RE practitioners.
Using the results of the same systematic review, Davis
et al. [18] looked at the effectiveness of the requirements
elicitation techniques. Pacheco and Garcia [19] performed
an SLR on stakeholder identification during requirements
elicitation based on 47 primary studies dated from 1984 to
2011. They found that identified approaches are not able to
cover all aspects of stakeholder identification during
requirements elicitation.
SLRs have also been reported in requirements specification area. Nicolas and Toval [20] presented an SLR of 30
studies on the generation of textual requirements from
software models. Fernandez et al. [21] performed a systematic mapping study to identify what aspects of software
requirement specifications (SRS) are empirically evaluated,
in which context and by using which research methods.
They found that the understandability was the most commonly evaluated feature of SRS, and the majority of the
primary studies are experiments performed in an academic
setting to evaluate requirements specifications. Amyot and
Mussbacher [22] performed an SLR on the first 10 years of
development of User Requirements Notation (URN) and
also highlighted ongoing improvement efforts. The SLR’s
results showed that the URN is a growing requirements
modeling language in terms of its users and contribution.
A number of SLRs have also been performed in the
requirements management area, focusing on specific topics
within requirements management such as requirements
evolution management [23], requirements prioritization
[24], requirements traceability [25], requirements related
errors finding [26], causes of requirements change [27],
requirements triage and selection [28] and requirements
reuse [29].
Requirements engineering have also been investigated
within global software development (GSD) context. Lopez
et al. [30] performed an SLR to compile a repository of risk
factors that arise when RE is done in distributed software
Requirements Eng
123
development environment, along with a collection of
safeguards to overcome these risks. The SLR just presented
the risks and safeguards repository without its validation on
some real GSD project. Peng and Lai [31] performed a
review to study all the wikis (well-known knowledge
management tools that support collaborative work) used to
carry out requirements engineering activities in distributed
development. The main goals of the review were to gain an
insight into how and to what degree current distributed
requirements engineering-related wikis could support the
RE activities, and also to identify the future research
directions.
Blain et al. [32] conducted an SLR to synthesize RE
literature relevant for multi-agent systems. The aim of the
review was to investigate which requirements engineering
techniques have been applied in the development of multiagent systems (MAS) and how they were applied. This
SLR was based on 58 primary studies, but only 5 % of the
papers provided some empirical evidence about the effectiveness of their approaches. Alves et al. [33] performed an
SLR to critically appraise the available evidence on RE for
software product lines. The SLR included 49 primary
studies covering 20 years from 1990 to 2009. They found
that the evidence for adoption of the methods in the
included 49 primary studies mainly consists of toy examples and is therefore not mature.
A list of these SLRs with their overall summary is
presented in Appendix 1. However, all of these SLRs
focus on some sub-areas of RE, respectively, and cover
empirical studies related to that specific area only. To
the best of our knowledge, no SLR has aggregated
results of the existing interventions in the whole RE
discipline. Furthermore, some of the existing SLRs have
also included non-empirical primary studies [34]; contrary to it, our SMS is based on only empirical studies of
RE. Our SMS is an attempt to add to the RE body of
knowledge in its own specific way, with the aim to
aggregate results from the whole RE empirical literature,
presenting various trends and future opportunities for RE
researchers and practitioners.
3 Research method
We followed the guidelines provided in [11, 35] for conducting this mapping study. In this section, we describe the
activities of the research process we followed.
3.1 Protocol development
The initial step of this research was the development of the
SMS protocol. The authors collaboratively worked with
each other during its development. The protocol included
research questions, decisions for search strategy, data
extraction strategy, criteria for inclusion/exclusion and data
synthesis strategy.
3.2 Search string
The search string was formulated by considering the keyword software along with two sets of keywords X and Y,
where:
X: All related terms of ‘‘requirements engineering.’’
Y: All related terms of ‘‘empirical.’’
Z: software
The final search string was like: ((All related terms of
requirement engineering ORed) AND (All related terms of
empirical studies ORed) AND software)). The final string
consisted of the terms shown in Table 1.
A generic query string was developed to search various
databases as:
(Software AND (‘‘requirements engineering’’ OR
‘‘requirements process’’ OR ‘‘requirements elicitation’’ OR ‘‘requirements gathering’’ OR ‘‘requirements identification’’ OR ‘‘requirements discovery’’
OR ‘‘requirements analysis’’ OR ‘‘requirements validation’’ OR ‘‘requirements verification’’ OR ‘‘requirements specification’’ OR ‘‘requirements
development’’ OR ‘‘requirements documentation’’
OR ‘‘requirements management’’ OR ‘‘requirements
change management’’ OR ‘‘requirements negotiation’’ OR ‘‘requirements testing’’ OR ‘‘requirements
checking’’) AND (‘‘case study’’ OR ‘‘industrial
Table 1 Search string terms
X1: requirements engineering
X2: requirements process
X3: requirements elicitation
X4: requirements gathering
X5: requirements identification
X6: requirements discovery
X7: requirements analysis
X8: requirements validation
X9: requirements verification
X10: requirements specification
X11: requirements development
X12: requirements documentation
X13: requirements management
X14: requirements change management
X15: requirements negotiation
X16: requirements testing
X17: requirements checking
Y1: case study
Y2: industrial report
Y3: experiment
Y4: experience report
Y5: empirical
Y6: observational study
Z: software
Requirements Eng
123
report’’ OR experiment OR ‘‘experience report’’ OR
empirical OR ‘‘observational study’’)).
The search string was validated by applying it iteratively
on the databases, for checking few (8–10) well-known
papers from RE, during pilot testing steps of the protocol.
This generic string of a query was modified to specific
queries according to each database. The query was also
broken down into sub-queries due to the limitations provided by each database for the maximum number of terms
in a query. The queries were applied on the title and
abstracts of the papers. We have applied queries in the
databases in the year of 2012.
3.3 Search strategy
An automated search process was employed to find all the
relevant studies of RE. Four major databases were
searched:
• Association for Computing Machinery (ACM).
• The Institute of Electrical and Electronics Engineers
(IEEE).
• ScienceDirect.
• SpringerLink.
During protocol development, we also planned to search
EI Compendex, but later on at the execution phase of the
SMS, we failed to cover this database because of
unavailability of it due to subscription issues. The items to
be searched included:
• Journal papers.
• Conference papers.
• Peer-reviewed workshop papers.
The publication period of the studies included in the
SMS was decided to be from the start of the period specified in various databases till the year 2012, and only the
papers in the English were included.
3.4 Inclusion and exclusion criteria
The study was decided to be included that would fit the
criteria as:
• The study was about RE.
• OR the study was about any of the sub-areas of RE.
• AND the study had empirical evidence (i.e., it is a case
study, experiment, survey, or experience report-based).
The study was decided to be excluded that was:
• In the form of books, literature surveys, SLRs, mapping
studies, thesis, unpublished articles, tutorials, summaries, discussions, prefaces, comments and editorials.
• OR the study did not directly address RE or any of its
sub-areas.
• OR the study lacked empirical evidence.
• OR the study was not in the English language.
3.5 Quality assessment strategy
Quality assessment is used to evaluate the quality of the
empirical evidence described in the primary studies. These
criteria were adopted from the SLR guidelines [13, 36–39].
Appendix 2 shows various sections of the checklist with
respective questions of each section. The questions included in the checklist were answered either ‘‘yes,’’ ‘‘no’’ or
‘‘partial,’’ rated as 2, 1 or 0, respectively. The sum of the
scores for all these questions was used to assess the quality
of a primary study. We, however, did not exclude any study
based on its quality score; rather, this score just depicts
quality rank of the primary studies.
3.6 Data extraction strategy
Appendix 3 enlists the data items we extracted from each
primary study. To find various core/main and sub-areas of
RE in the data extraction scheme, we consulted SWEBOK
[40] and REBOK [41]. The type of research in data
extraction scheme had been formulated according to the
research types provided in [42]. The rest of the items were
extracted to carry out a rich analysis and present various
themes and trends as advised in [35].
The data of this SMS were extracted and saved in
Microsoft Excel sheets where data for each primary study
was saved in a separate row of excel sheet with a reference
ID corresponding to its ID in endnote library (that contained all the references in it). Use of excel sheets for
saving data and its analysis saved a lot of time, as it was
easier for cross-referencing of primary studies. The next
section describes the whole process of the SMS.
3.7 Review process
The overall process of this mapping study has been divided
into four phases: phase 1 involves the research questions
and search strategy formalization, phase 2 involves
searching the references from the databases by using search
strings, saving of references in endnote, and then removing
duplicates, phase 3 involves studies’ screening (level 1 and
level 2) and data extraction, while phase 4 involves the
quality assessment of the studies. The whole process of the
mapping study is shown Fig. 1.
The initial search string yielded thousands of research
papers, which were exported to endnote, which is a widely
Requirements Eng
123
used tool for reference management. Separate endnote
libraries were created for the set of research papers
obtained by applying queries on ACM, SpringerLink,
IEEEXplore and ScienceDirect. Duplicates were removed
from each database endnote library, after that all the
research paper references from the four databases libraries
were merged and again duplicates were removed.
The final endnote library contained all the research
papers’ references contained in four separate libraries of
databases, merged into a single endnote library. It was the
final set of references on which level 1 screening was
applied. Level 1 screening involved studying the paper
title, abstract and keywords to find:
• The study was relevant to RE field.
• And the study was also empirical based.
Initial query application on the four databases yielded
380 results from IEEE, 88 from ScienceDirect, 5841 from
SpringerLink and 643 from ACM. It is worth mentioning
here that the criteria of paper search from the four databases were initially set to be based on the empirical nature
of the study. Therefore, queries applied to databases contained empirical factors to get only empirical studies. After
individual searches from the four databases, there were a
total of 6952 studies retrieved. A total of 2229 studies were
left after discarding duplicates from the search results of
Fig. 1 Systematic mapping study process
Requirements Eng
123
the four databases. All of these studies in separate endnote
files, belonging to the four databases, were then merged
into a single master file of endnote, and there emerged 227
duplicates again from all the references’ merger. We
removed these duplicates and a total of 2002 studies were
finally left, on which level 1 screening was applied.
During level 1 screening, each of the 2002 studies was
screened by reading the title, abstract and keywords, to find
if the study was relevant to the RE field and was also
empirical. We found that a large number of abstracts of the
studies were not detailed enough to prove its empirical
nature, so we had to pass these studies to the next level
(level 2), for further screening. Level 1 screening was the
primary responsibility of the first author, while third author
reviewed a randomly selected sample (10 %) of these
studies to validate the results.
Level 2 screening involved reading the whole text of the
studies obtained from level 1 screening, with the same
criteria as for level 1 screening. Here, the studies were
excluded based on two factors: either their main focus was
not on RE or the studies did not present a real empirical
work. While reading the whole text of the papers, we found
that most of the studies that employed case study
methodology did not employ it in a real manner. These
studies presented examples and scenarios rather than cases
from the real world. Therefore, such studies were not
selected as primary studies. During level 2 screening, some
of the studies were selected as candidate primary studies,
while some were rejected. The third category contained the
studies about which the primary reviewer had some doubt
to include or exclude, so these studies were then discussed
with secondary reviewers and resolved. Unfortunately, we
could not have access to the full text of 35 studies. After
level 2 screening, by reading the whole text of the papers,
there were 270 studies that had finally been selected as
primary studies of the SMS. A complete list of these primary studies is provided online (a link at [43]). The third
author of this paper contributed during level 2 screening
also, by reviewing a randomly selected sample (8 %) of
studies to validate the results. The next section describes
the results of this SMS.
4 Results
This section describes the results along with the analysis of
the data extracted from the studies, to answer both the
research questions of this SMS.
4.1 State of the art in empirical studies of RE (RQ1)
To answer the first research question, we formulated few
sub-questions to investigate various aspects of primary
studies. This section describes the details of the state of the
art in RE.
4.1.1 Which era of RE research has maximum progress
in terms of new advances?
This SMS includes 270 primary studies spanning over the
era of two decades. Figure 2 shows the frequency of RE
empirical studies reported in the period from 1991 up to
2012. The Empirical work in RE started in the era of 1990s.
The empirical work in RE was not significant till the year
2000; however, there has been an increased attention paid
by researchers toward empirical RE studies during the last
decade of reported RE studies.
4.1.2 Which country is frequently involved in RE research
and in which era, these countries showed maximum
progress?
It is interesting to know the regions where the empirical
research of RE has been conducted. There are 29 different
countries involved in reporting empirical RE research.
Figure 3 shows the frequency of studies reported from the
top 20 countries involved in RE research. The maximum
frequency of studies is reported from USA (39 studies).
Fig. 2 Year-wise distribution of RE studies
Requirements Eng
123
Germany (38 studies), UK (22 studies) and Australia (20
studies) are also significantly involved in empirical RE
research. Sweden, Italy and Canada also reported a considerable number of empirical studies in RE. The empirical
work reported in other countries is very little, so they have
not been shown in the above figure. The status of 39 studies
is unclear for the country to which they belong.
To investigate the second part of sub-question: ‘‘in
which era a country showed maximum progress?’’ we
mapped the top 10 countries in terms of frequency of
studies against per year reporting of studies in these
countries as shown in Fig. 4. The earliest empirical RE
study is reported in the USA in the year 1991, which
indicates that the USA was the first country reporting the
empirical work related to RE. The frequency of empirical
studies in the top 10 countries is minimal before the year
2000, except for the USA, Germany and UK, that were
involved in empirical research work of RE before the year
2000.
4.1.3 In which context, these empirical studies of RE have
been carried out over the period of RE research?
The primary studies included in this SMS belong to various
domains as shown in Appendix 4. The most significant
domains are embedded (13 %), telecom (7 %), MIS
Fig. 3 Countries involved in RE studies
Fig. 4 Yearly distribution of
studies in top 10 countries
Requirements Eng
123
(management information systems) (7 %), finance (7 %)
and web (6 %), whereas a small percentage of studies
belong to education (3 %), e-commerce (2 %, and manufacturing (1 %) domains. 16 % of studies belong to multiple domains, while 24 % of studies belong to the generic
software development domain. There are 14 % of studies
that belong to some other domains.
We also tried to investigate the most active domains of
RE research recently. So, Fig. 5 shows the yearly distribution of empirical RE studies in various domains. From
Fig. 5, it can be seen that MIS is the oldest domain where
empirical work of RE appeared, and then, in late 1990s the
work in other domains like telecom and finance started.
However, embedded domain showed a progressive
increase in empirical studies from 1995 to 2012. Also, it is
the most active domain of empirical RE work recently. The
embedded domain in turn contains other various domains
in it, like avionics, medical and automotive [44], and we
also categorized electronics and control systems under the
embedded domain. In the last decade, some empirical work
in education and e-commerce domains can also be seen in
Fig. 5. However, in the recent era, the most active domains
of RE empirical research are embedded, telecom, web and
finance.
4.1.4 How empirical research in various RE knowledge
areas has evolved over the years?
To check knowledge areas of RE where research has been
conducted, we first had to decide what are the various
knowledge areas of RE? So, we consulted SWEBOK [40]
and REBOK [41]. The terminologies and concepts of these
knowledge areas are, however, finally adopted from
REBOK [41].
Table 2 shows the quantification of empirical research
done so far in each RE core area. It can be seen that major
work is done in requirements elicitation (22 %), requirements analysis (19 %) and RE process (17 %) core areas.
Fig. 5 Yearly distribution of empirical RE studies in various domains
Table 2 Core RE areas
Core areas of RE No. of empirical studies Percentage
Requirements engineering fundamentals 9 3
Requirements engineering process 46 17
Requirements elicitation 59 22
Requirements analysis 50 19
Requirements specification 28 10
Requirements verification, validation and evaluation 13 5
Requirements planning and management 36 13
Practical considerations of requirements engineering 29 11
Requirements Eng
123
A significant number of primary studies have reported
research in the area of requirements planning and management (13 %). A considerable number of studies have
reported research in requirements specification (10 %) and
practical considerations of RE (11 %). The studies dealing
with the practical consideration of RE have mostly presented research interventions in terms of some lessons
learned, so these research interventions have been categorized as ‘‘guidelines’’ during data extraction. Only 5 % of
studies deal with requirements verification, validation or
evaluation, and only 3 % deal with fundamental aspects of
requirements engineering.
We also categorized some sub-areas of RE shown in
Table 3. However, the studies related to sub-areas do not
exist independently, rather all of these studies also fall in
any of the core areas too. The sub-areas shown in Table 3
present significant work in requirements negotiation
(5 %), requirements prioritization (5 %), requirements
traceability (4 %) and requirements modeling (4 %).
However, only a small amount of empirical RE work is
discovered in requirements risk analysis (2 %), requirements impact analysis (1 %) and enterprise analysis
(1 %).
Figure 6 shows the yearly distribution of studies in RE
core areas to visualize research trends in these areas. The
results show a consistent trend across almost all core areas,
highlighting the fact that the interest in empirical research
in RE has been on the rise after year 2000. The quantity of
empirical research in all RE core areas has progressed well
from year 2005 onwards.
During the data extraction and analysis stages of this
SMS, we observed few interesting trends. Some new areas
of research have been emerging in various RE core areas.
Figure 7 presents some areas that have emerged in
empirical RE research. Non-functional requirements are by
far the most active among these emerging research areas.
Fig. 6 Yearly distribution of studies in core areas of RE
Table 3 Sub-areas of RE
Sub-areas of RE No. of empirical studies Percentage
Requirements negotiation 13 5
Requirements prioritization 12 5
Requirements traceability 10 4
Requirements modeling 10 4
Requirements risk analysis 4 2
Requirements trade-off analysis 3 1
Requirements impact analysis 2 1
Enterprise analysis 2 1
Requirements Eng
123
Some of these areas, such as formal methods and process
improvement in RE, have been under investigation since
1990s. However, empirical research in the lead emerging
areas increased considerably after the year 2005, see nonfunctional requirements and global RE in Fig. 8, for
example. Some areas, on the other hand, have only
emerged in twenty-first century and have not been investigated very extensively so far. These include: goal-oriented RE, requirements change management, agile RE,
value-based RE, etc.
4.1.5 What type of interventions have been proposed
or investigated in RE research?
Figure 9 shows various types of interventions that have
been investigated over the years in RE research. These
Fig. 7 Emerging areas of RE
Fig. 8 Yearly distribution of emerging areas of RE
85
48
32
23
20
15 13
8 7 6 5 4
2 1 1
0
10
20
30
40
50
60
70
80
90
No. of Studies
Research Outputs
Frequency of Research Outputs
Fig. 9 Research outputs in RE studies
Requirements Eng
123
results indicate that guidelines, techniques and processes
are highly investigated intervention types in RE empirical
research. 31 % (85) of studies proposed guidelines in
various RE areas. New techniques and processes have
been proposed in 18 % (48) and 12 % (32) studies in our
SMS. It was also interesting to note that there is more
interest in proposing new interventions (guidelines, technique, processes, etc.) in empirical RE research, while
relatively little attention is paid to use and evaluate
existing interventions.
To analyze the investigated interventions with respect to
each RE core area, we mapped them to each area in
Appendix 5. RE area-wise classification of interventions in
Appendix 5 can be used by RE researchers and practitioners to identify relevant types of interventions that are
proposed and/or validated empirically in their area of
interest. The research output ‘‘guidelines’’ is missing in
Appendix 5, as we could not represent guidelines with a
short name/label as such, but in the future we plan to
summarize these guidelines too.
Most of the names of these interventions in Appendix 5
are used as reported by authors in the primary studies of the
SMS. However, if we could not find any proper name of
the research output mentioned in the paper, we named it by
using keywords from the paper. The complete list of primary studies reporting these interventions is available in
[43].
4.1.6 In which channels the empirical studies of RE are
published?
Publication channels for primary studies include conference (62 %), journals (36 %) and peer-reviewed workshops (2 %) as shown in Fig. 10. The International
Requirements Engineering Conference (ICRE), with 21 %
of the studies, is the top publication channel of primary
studies. The Requirements Engineering journal with 16 %
is the second in the list. These results confirm the results of
the study reported in [5].
Other significant journal publication channels, shown in
Fig. 11, include Journal of Information and Software
Technology, Journal of Empirical Software Engineering,
IEEE Transactions on Software Engineering and Journal of
System and Software. Conference channels include REFSQ
(International Conference on Requirements Engineering:
Foundation for Software Quality), APSEC (Asia–Pacific
Software Engineering Conference), ICGSE (International
Conference on Global Software Engineering) and ICSE
(International Conference on Software Engineering). There
are also many other conferences and journals whose percentage of studies was not that significant to be presented
here. These sources of RE studies can be helpful for RE
researchers to hunt for the desired empirical work of RE and
for seeking a chance to publish new RE research papers.
ICRE: International Conference on Requirements
Engineering.
REFSQ: Requirements Engineering Foundation for
Software Quality.
APSEC: Asia Pacific Software Engineering Conference.
COMPSAC: International Computers, Software &
Applications Conference.
ICSE: International Conference on Software
Engineering.
PROFES: International Conference on Product-Focused
Software Process Improvement.
HICSS: Hawaii International Conference on System
Sciences.
ER: International Conference on Conceptual Modeling.
SAC: Symposium on Applied Computing
Fig. 10 Types of publication channel Fig. 11 Publication channels
Requirements Eng
123
4.1.7 How the research method, type, intervention
and domain are related to RE core areas?
This research question aims to characterize the research
conducted in different RE core areas with respect to the
research method, type, output and domain of the studied
projects/products. Figure 12 classifies the primary studies
with respect to the RE core areas and research types. The
results show that most of the research in RE is of ‘‘evaluation’’ and/or ‘‘validation’’ type. This pattern is consistent
across almost all RE core areas. The fact that research is
evaluated in laboratory settings (validation) or implemented and/or evaluated in practice (evaluation) is positive
in terms of rigor. However, relatively fewer experience
papers in different RE core areas (e.g., Analysis, Planning
& Management) point to the need for encouraging practitioners, to share their firsthand experiences more frequently
at academic venues.
Figure 13 classifies research in RE core areas with
respect to the type of interventions proposed or
investigated. The results indicate that the dominant trend,
across almost all RE core areas, is that of proposing new
interventions. These intervention types are new guidelines, new techniques, new processes and frameworks.
‘‘Guidelines’’ by far is the leading intervention for
‘‘Practical Considerations of RE’’ and ‘‘RE Fundamentals’’ core areas. Various types of interventions have been
proposed in ‘‘Elicitation’’ and ‘‘Planning & Management’’
core areas including guidelines, techniques, processes and
frameworks. Technique-oriented research dominates
‘‘Analysis,’’ while ‘‘RE Process’’ core area has high
number of studies proposing new processes, guidelines
and frameworks. These results can be used by researchers
in initiating efforts to organize and classify knowledge in
different core areas of RE. There are relatively fewer
studies that report results of using or modifying existing
interventions in new contexts. It is important for the
maturity of RE discipline that more studies are conducted
to further validate and enhance existing interventions. The
interventions that have been validated in number of
Fig. 12 Research types versus
RE core areas
Requirements Eng
123
contexts would be more acceptable for software companies and practitioners.
Figure 14 presents classification of primary studies with
respect to the domain of the studied project or product or
organization for all RE core areas. General software
development and embedded systems development are the
leading domains across all core areas. The results show that
research in ‘‘RE Process,’’ ‘‘Elicitation’’ and ‘‘Analysis’’
core areas has been conducted in variety of domains.
However, it is not the case for core areas like ‘‘Specification,’’ ‘‘Planning & Management’’ and ‘‘verification, Validation and Evaluation’’ wherein the most research efforts
are limited to relatively fewer domains. We also tried to
investigate which RE areas are investigated by which
research methods, see Fig. 15.
From Fig. 15, it can be seen that surveys have mostly
been conducted for investigation of practical consideration of RE, for investigation of requirements elicitation
and for exploring fundamental aspects of RE, while a
large number of case studies have been conducted for
investigating issues related to requirements engineering
process, requirements elicitation, requirements
specification and requirements planning and requirements
management areas of RE. The experimentation methodology has mostly been employed to explore requirements
elicitation and analysis areas of RE. However, for
investigation of issues related to requirements analysis, an
equal amount of case studies and experiments have been
conducted in the empirical studies. The maximum number
of experience reports has been reported related to RE
process area. It can be seen that the case study is the most
popular research method for almost all the areas of RE
except, RE fundamentals and verification, validation and
evaluation area of RE.
4.2 Strength of empirical evidence of RE (RQ2)
The aim of the second question of this SMS was to find out
the strength of empirical studies by finding the source of
empirical studies along with the research methods and data
collection methods employed in the studies. Following
section deals with various question used to explore the
strength of empirical studies of RE.
Fig. 13 Research interventions versus RE core areas
Requirements Eng
123
4.2.1 Which research method is frequently used in primary
studies?
The results show that the half of the primary studies used
case study as the research method (see Fig. 16). The
experiments have been used in 28 % of primary studies,
surveys in 16 % of primary studies, whereas only 6 % of
the studies reported the experience reports as an evidence
type. It points out the need for more experience reports to
let RE researchers and practitioner gain benefit from these
experiences. Also, a large number of case studies and
experiments offer opportunities for replication of the
studies.
We used the quality assessment criteria, given at the
Appendix 2, to assess the quality of the primary studies.
The quality scores for half of the primary studies using case
study was 50 % or less. The quality score for 37 % of the
primary studies using the experiment as research method
was more than 75 %. Figure 17 shows the research
methodology-wise quality scores segments of the primary
studies.
4.2.2 What kind of research participants are frequently
involved in primary studies?
From Fig. 18, it can be seen that 59 % of the studies
involved practitioners as subjects, while 27 % of the
studies used students as subjects. There are only 4 % of
studies where subjects both from industry and academia
participated, indicating a collaborative research. The
type of subjects of investigation in 10 % of the studies is
not mentioned in the primary studies. These results
indicate that RE research is more practice-oriented as it
involved more practitioners and professionals from the
industry.
It is interesting to note that 66 % of studies using case
studies as research method have been investigated in an
industrial setting, as shown in Table 4. Only 16 % of case
studies have been investigated in an academic setting.
Similarly, surveys have also been investigated largely in an
industrial setting (76 %) and comparatively less in an
academic setting (7 %). This trend is reversed for experimentation as 59 % experiments have been investigated
Fig. 14 Domains versus RE area-wise studies
Requirements Eng
123
using subjects from academia, i.e., students and 32 % from
industry. This result may be due to the factors of cost and
effort, as it is more expensive and difficult to conduct the
Fig. 15 Research method versus RE area-wise studies
Fig. 16 Types of research methods
Fig. 17 Quality scores of primary studies (research methods-wise)
Requirements Eng
123
experiments in the industrial setting. However, the credibility of the results using students as subjects is debatable.
We see an opportunity for repeating these experiments
using subjects from the industry to improve the credibility
of results from these experiments.
4.2.3 Which data collection method is frequently used
in investigation of research in primary studies?
Figure 19 shows the frequency of various data collection
methods used in the empirical studies of RE. There are
four distinct data collection methods that have been used
overall during the investigation of the primary studies,
including questionnaire, interview, archive analysis and
observations. 27 % (74) of studies, however, employed a
combination of data collection methods, marked as
‘‘Mixed’’ in Fig. 19. Observation (21 %, 57), questionnaire (17 %, 45), archive analysis (14 %, 39) and
interviews (13 %, 36) have also been used frequently in
empirical studies. The data collection method of 19
primary studies was not mentioned in primary studies of
the SMS.
4.2.4 What is the frequency of various research types
of primary studies?
The types of research reported in empirical studies shown
in Fig. 20 have been decided by consulting the research
types provided in [42]. Validation research type covers
empirical studies used to present new research outputs of
RE (new techniques, new tools, etc.), ‘‘Evaluation’’
research type has been assigned to empirical studies where
usage experience, modification or comparison of existing
research outputs of RE has been done, while solution
proposal’’ research type has been assigned to the empirical
studies, where some new research output has only been
proposed without a full validation. The research type
‘‘experience paper’’ represents all the studies based on
experience reports. The ‘‘philosophical paper’’ represents
the study where some whole new philosophy has been
presented, and there is only one such empirical study that
we categorized in this type.
Fig. 18 Subjects of investigation
Table 4 % of subjects of
investigation in research
methods
Type of research method Subjects of investigation
Case study Academia: 16 % Industry: 66 % Mixed: 3 %
Experiment Academia: 59 % Industry: 32 % Mixed: 5 %
Survey Academia: 7 % Industry: 76 % Mixed: 7 %
Fig. 19 Data collection
methods
Requirements Eng
123
From Fig. 20, it can be seen that 49 % (132) of studies
belong to evaluation research type, while 43 % (117) deal
with validation research. Only a handful of studies are
categorized as experience papers and solution proposals.
There is only one paper that is of philosophical research
type, while opinion papers, understandably, were not represented in our study due to the absence of empirical evidence in them.
4.2.5 How rigorous is the reported research in different
RE core areas?
This research question aims to assess the quality of
reported research in all RE core areas. Figure 21 presents
the classification of primary studies with respect to the
quality scores for all RE core areas. The quality assessment
instrument presented in Appendix 2 was used to assign a
quality score to all the primary studies.
The minimum and maximum quality scores were in the
range of 6–26, respectively. We divided the quality score
range from 6 to 26 in four groups as depicted in Fig. 21
and assigned all studies to the relevant groups based on
their scores. The results show a similar pattern across all
core areas, i.e., most of the studies have quality scores
falling in two middle groups, i.e., 11–15 and 16–20.
‘‘Elicitation’’ is one core area that has relatively higher
percentage of studies in high-quality-score group (21–26),
as compared to low-quality-score group (6–10). Other RE
core areas, such as RE process, analysis and planning &
management, have almost same number of studies both in
low-quality-score (6–10) and high-quality-score (21–26)
groups. The future studies in these core areas need to
properly address questions related to rigor and relevance
of the research.
5 Discussion
This section presents discussion on major findings of the
mapping study to highlight trends and opportunities for
future research.
5.1 Findings for RQ1
The empirical research in RE mainly started in 1990s and is
on the rise since the year 2000. This has also been observed
in a recent editorial [45]. A number of publication venues
emerged during this time line that might have contributed
in this trend, e.g., in the year 1991, the systems engineering
symposia of the International Council on Systems Engineering (INCOSE) started working, then in 1993, IEEE
International Symposium on RE started, later in the year
1994, IEEE International Conference on RE and International Conference on Requirements Engineering: Foundation for Software Quality (REFSQ) started and Springer’s
Journal for RE started in 1996 [5]. The emergence of
evidence-based software engineering (EBSE) paradigm
around the year 2004 also resulted in the increased
awareness and interest in performing empirical studies in
all the fields of software engineering, including RE. The
rise in the number of studies after 2005 (see Fig. 2) coincides with the emergence of EBSE paradigm [45]. The
emergence of new areas in software engineering, such as
global software development or value-based software
engineering, also contributed in this rising trend by giving
rise to the need for re-investigation of existing RE
practices.
Requirements elicitation is the leading empirically
researched RE core area, and the interest in investigating
it further is still on the rise (see Fig. 6). Does this mean
132
117
14
6
1 0
Evaluation
Research
Validation
Research
Experience Paper Solution Proposal Philosophical
Paper
Opinion Paper
Fig. 20 Types of research in RE studies
Requirements Eng
123
that we have failed to solve the problems in the
requirements elicitation area or we are facing new
problems? The later seems plausible as empirical studies
in RE have been conducted for a variety of domains (see
Fig. 5). The extensive body of knowledge in requirements elicitation needs to be organized in such a way
that it is readily available to software practitioners.
Requirements analysis, which is very closely linked with
elicitation, has also been an active area of research. A
large number of studies have also been performed to
investigate RE processes. These leading topics have been
investigated in variety of domains (see Fig. 6) such as
embedded, finance, information systems, etc. Some
domains, such as embedded or information systems,
have relatively high number of empirical studies. It
could be interesting to initiate efforts to organize existing RE knowledge in such domains. What interventions,
for instance, have been proposed and evaluated in
information systems domain?
The interest in three RE core areas (specification,
requirements planning and management, and practical
considerations of RE) has gone up since year 2005.
Practical considerations of RE covers best practices and
patterns of RE, gained mostly through research methods of
surveys or experience reports. Researchers and practitioners can gain benefit from these experiences in various
small- and medium-sized organizations [46, 47], or companies located in some specific countries such as Australia
[48], New Zealand [46], China [49, 50], Malaysia [51] and
Europe [52].
The empirical research in the requirement validation and
verification (V&V) is little and has a decreasing trend.
Validation and verification of requirements helps the
development team to check whether systems meet its
business objectives and stakeholders’ needs, and all the
documented requirements have been implemented or not.
Given its importance, it is surprising to see lack of interest
in V&V area. There is a need to investigate these topic
further, in future empirical studies.
Non-functional requirements (NFRs) is most extensively
researched emerging area. Table 5 presents various NFRs
and corresponding number of empirical studies. We identified, like [45], that security requirements are investigated
relatively extensively followed by usability requirements,
Fig. 21 Quality score versus studies of RE core areas
Requirements Eng
123
while other NFRs lack empirical research. The need to
investigate security requirements has been stressed elsewhere at [4] as well. However, the researchers need to look
into the emerging non-functional requirements such as
privacy, and regulatory requirements.
Future research efforts should be directed at other NFRs
as well such as performance, sustainability requirements.
Other emerging areas, where a reasonable amount of RE
research has been published (see Fig. 8), include distributed/global requirements engineering, requirements
process improvement and goal-oriented RE. Chang and
Atlee [4] highlighted globalization as ‘‘RE research hotspots’’ in 2007, since then number of studies have investigated this topic.
Value-based requirements engineering is also an
emerging area. However, it has only seven empirical
studies. Other interesting emerging areas, where a relatively small amount of empirical research has been conducted so far, include RE for embedded software (6
studies), agile RE (6 studies), RE and software architecture’s relationship (5 studies), and RE patterns and
requirements ontology (5 studies). The relatively limited
empirical work in these emerging areas so far presents
opportunities to explore in future research efforts.
Researchers in recent years have also investigated smalland medium-sized enterprises (SME) from various aspects,
as these companies are considerably different to large
companies. Following this notion, RE researchers have also
investigated SMEs for requirements engineering aspects in
recent few years in a small number (5 studies) of empirical
studies. Some other topics also emerged in the last 5 years
with a very few studies (3 or less than 3) including power
and politics in RE, requirements inspection, requirement
conflicts resolution and requirements information modeling.
Formal methods in RE and requirements change management are two areas that have been around for quite
some time. However, these areas lack empirical research.
For requirement change management, an SLR conducted in
[53] also pointed out the same notion that there are only a
handful empirical studies in this area. For formal methods
in RE, we found many studies initially. However, later
during data extraction, majority of these studies were
excluded due to lack of empirical evidence. We did not
select such studies wherein only toy examples, illustrations
or scenarios are used as empirical work. The interest in
requirements process improvement area has been on the
decline after year 2006. It needs to be investigated what
does it indicate, saturation and/or maturity of the area.
We identified 43 RE topics investigated in our sample of
empirical studies. We have categorized them in three categories in Table 6: core areas, sub-areas and emerging
areas. Some of the emerging areas are also highlighted in
other works [3, 4].
The main interest in RE research is on proposing new
interventions in the form of new guidelines, techniques,
processes, frameworks, etc. There are a few studies which
evaluate and modify existing interventions. The researchers
ought to work and solve problems by ‘‘standing on the
shoulders of giants.’’ However, we do not see this trend in
RE field. The researchers in future, besides proposing new
guidelines or techniques, should also focus on validation of
existing guidelines and techniques in different contexts,
and attempt to build and improve on top of existing work.
The interventions that have been evaluated in multiple
studies in different contexts would be more acceptable for
software practitioners.
5.2 Findings for RQ2
A half of empirical studies included in this mapping study
have used case study as a research method. Two-thirds of
these case studies are conducted in industrial settings, a
good sign of involvement of industry participants. Experiments have been conducted in 28 % of the primary studies
and mostly student were used as subjects. Therefore, there
is a need for replication of these experiments using
industrial subjects to improve the credibility of results.
The researchers also need to replicate case studies to
validate the credibility of existing results. The researchers
reporting case studies and experiments must provide
detailed protocols for their studies enabling other
researchers to replicate. The purpose behind such replication is to prove the validity of the results from the original
study to a larger population [54]. Various authors have
pointed out toward the lack of replication and a need of
replication of empirical studies in research [55–57]. The
lack of replications might be due to the difficulty inherent
in the process of replication of studies, because of the
involvement of human subjects [58, 59]. The results of this
mappings study identified that case study and experiment
are the two main research methods used in empirical RE
Table 5 Research on NFRs
Type of NFR Frequency
Generic NFRs 6
Security requirements 16
Usability 7
Legal/regulatory requirements 3
Performance requirements 2
Sustainability requirements 2
Requirements Eng
123
research; this provides opportunities to replicate these
studies in future.
The use of survey research method is not very popular as
only 16 % of primary studies presented surveys. However,
there is a need for more surveys to collect best practices
and patters of RE from a large population. There is only
6 % of studies reporting experiences of RE practitioners.
We need to learn from practice. Therefore, practitioners
should be encouraged to share their firsthand experiences
by publishing industry reports. An alternative approach for
researchers is to work more closely with practitioners, not
just to evaluate a proposed invention but also to work
together during problem identification and solution design
phases [4, 33]. The practitioners may benefit from the
catalogue of tools/techniques given at the Appendix 5. We
encourage practitioners to uses these tools/techniques and
share their usage experiences with the RE community for
further improvements.
6 Study limitations
The main limitation of this study is the inaccessibility to
the full text of some studies as mentioned earlier. Another
limitation of this study is that we could only cover four
databases, although we had various other options too, but
due to time availability, we could not extend the studies
search to other databases.
The decision to categorize various emerging areas was a
subjective decision as we have decided it solely on the
results presented in the primary studies so that it might be
subject to criticism. Moreover, there were a number of
overlapping topics of the studies that deal with emerging
areas of RE, so we selected the topics on our personal
judgment. We faced a critical difficulty during execution of
this SMS, when the platform of SpringerLink was under
construction for modification, and query results used to be
changed so frequently, so we might have missed some
Table 6 Topics of empirical research of RE
Core areas Sub-areas Emerging areas
Requirements engineering fundamentals
Requirements engineering process
Requirements elicitation
Requirements analysis
Requirements specification
Requirements verification, validation and evaluation
Requirements planning and management
Practical considerations of requirements engineering
Requirements negotiation
Requirements prioritization
Requirements traceability
Requirements modeling
Requirements risk analysis
Requirements trade-off analysis
Requirements impact analysis
Enterprise analysis
Non-functional requirements
Distributed/global RE
RE process improvement
Goal-oriented RE
Requirements change management
Formal methods in RE
Value-based RE
RE for embedded software
Agile RE
Relationship of requirements and SA
RE patterns
Requirements ontology
RE for SME
Power and politics in RE
RE for scientific computing projects
Requirements inspection
Requirement conflicts resolution
Requirements information modeling
Requirements communication
Requirements consolidation
Creativity in RE
Decisions in RE
Requirements’ analyst skills
RE for market-driven Software development
RE for software product lines
RE for web application
User management
Requirements Eng
123
studies during that. The threats to validity of our SMS are
as follows:
Construct validity describes the correct operational
measures of the concepts that are being studied. The key
constructs for our study are the concepts related to ‘‘software requirements engineering’’ and the methodology of
‘‘systematic mapping review.’’ For the first construct of
software requirements engineering, we tried to find the
basic concepts and core areas of RE and tried to compared
the related work. For the second construct, i.e., systematic
mapping study, we properly followed the guidelines of the
SMS to formulate our research questions, search strategy
and the protocol of our study. For constructing the search
string, we tried to use related terms of the various activities
involved in the requirements engineering process to get as
many results as possible. For targeting the maximum
search results, we covered the four major databases to
collect as many publications as possible. But still we could
not cover many other databases as Scopus, Compendex,
Citeseer, etc. So this can be a possible threat to construct
validity in our case.
Internal validity determines a causal relationship, where
specific conditions lead to other conditions. Regarding
internal validity, the key threats might be the primary
studies selection and individuals’ bias in their assessment.
The sources of studies in our case were conferences,
journals and peer-reviewed workshops of requirements
engineering. We followed an automated search process by
properly formulating search strings according to the rules
defined for searching each database to find the relevant
studies. During hunting the studies from various databases,
we have applied queries on titles and abstracts of the papers
only. Therefore, we may have missed the paper if it has not
mentioned any of the keywords we used, in its title and
abstract, but the probability of this is very low. So this can
be a threat to internal validity. The other threat originates
from the bias introduced by the individual researchers
during assessing their assigned primary studies. We handled this threat by defining a proper protocol, pilot testing
of the protocol and then solving the problems and issues
collaboratively during each and every step.
External validity is about generalization of the results. It
involved the areas and domain to which findings of a study
can be generalized. For handling external validity, we did
not limit the start of the period to which studies belong, and
the ending period was set to be 2012, to get a large number
of studies. But as we only selected empirical studies in our
mapping study, the studies appeared from 1991 till 2012.
Due to the empirical nature of the studies, we did not select
many types of studies as theses, technical reports and
books, etc. But this cannot be a threat to external validity in
our case as the nature of our SMS was purely empiricalbased. This is the reason we have rejected so many studies
based on toy examples and scenarios in the name of case
studies in our SMS. Also, our search strategy was based on
an automatic search having defined search strings consisting of many related terms to our topic of SMS, to possibly
get a large number of primary studies.
Conclusion validity is about getting the same results in
case of replication of a study. To handle conclusion
validity, we followed the guidelines [11, 35] to conduct the
systematic mapping study, with distinct steps of SMS and
proper criteria for searching and data extraction. But as
during replication of the study, the choice of databases,
search string terms and research questions might vary, so
results might differ to some extent, but overall trends
should remain same.
7 Conclusion
In this paper, we reported results of a mapping study on
empirical research in RE. The mapping study is based on
270 empirical studies from four databases ACM, IEEE,
SpringerLink and ScienceDirect till the year 2012. The
interest in the empirical research in RE is on the rise after
year 2000. Requirements elicitation, analysis and management were identified as leading areas with highest
number of empirical studies. Despite being an important
topic, requirements verification and validation lacks
empirical evidence. Non-functional requirements and global RE were identified as the lead emerging areas of
research. Lately, topics such as RE patterns, RE for small
and medium enterprises and requirements ontologies have
also received some attention.
There is limited interest in evaluating and comparing
existing interventions, rather the focus is on proposing new
ones. Guidelines and techniques are most frequently proposed types of RE interventions. There is need to replicate
studies in different contexts wherein existing RE interventions are evaluated and implemented in practice.
Although most of the case studies involve practitioners as
participants, there is a need to work more closely with
practitioners. Practitioners’ involvement should not be
limited to their role as subjects, wherein they help
researchers in just evaluating a proposed intervention. They
should also be involved in problem identification and
solution formulation stages. Only 6 % of the studies were
identified as experience papers. Software requirements
practitioners should be encouraged to share their experiences as experience/industry reports.
Requirements Eng
123
Appendix 1: List of existing systematic reviews
of RE
This appendix describes a summary of existing systematic
reviews of RE. These systematic reviews have been discussed in body section of this paper, but their overall
summary in terms of year of publication, number of primary studies included, source of primary studies, the range
of databases covered and the nature of the primary studies
included in the systematic study (empirical/non-empirical)
is listed here, to give an overview to readers through a
bird’s eye view.
Sr# Year Title of systematic study # of
studies
Source of primary studies Covered till the
time
Empirical/
nonempirical
Existing systematic reviews of RE
1 2006 Effectiveness of Requirements
Elicitation Techniques [18]
26 SCOPUS, IEEEXPLORE, ACM DL March 2005 Empirical
2 2008 Requirements Prioritization
Based on Benefit and Cost
Prediction [24]
240 ACM Digital Library, Compendex, IEEE Xplore, ISI
Web of Science, Kluwer Online Science Direct
Elsevier, SpringerLink, Wiley InterScience, and
manual search
Feb 2008 Empirical
and nonempirical
3 2009 RE in the Development of
Multi-Agent Systems [32]
58 ACM DL, IEEExplore, Inspect, and Science Direct 1998 to March
2009
Empirical
and nonempirical
4 2009 Software Requirements
Specifications Techniques
[21]
46 Scopus, IEEE Digital Library, ACM Digital Library,
and manual search
1987–2008 Empirical
5 2009 Software Requirement Errors
[26]
149 Databases:IEEExplore, INSPEC, ACM Digital
Library, SCIRUS (Elsevier), Google Scholar,
PsychINFO (EBSCO), Science Citation Index
Not mentioned Empirical
and nonempirical
6 2009 Generation of Requirements
Specifications from Software
Engineering Models [20]
24 IEEE Digital Library,ACM Digital
Library,ScienceDirect, MetaPress
(Kluwer ? springer), Wiley InterScience, Google
scholar
Not mentioned Empirical
7 2009 Risks in RE Process in Global
Software Development [30]
36 IEEE Digital Library,ACM Digital Library
Metapress, Google Scholar
2000–2009 Empirical
and nonempirical
8 2009 Technology transfer decision
support in RE [60]
97 RE Journals in Inspec Start time: not
mentioned end
time: June,
2008
Empirical
and nonempirical
9 2009 Systematic Review of
Requirements Reuse [29]
18 IEEE Xplorer digital library, ACM digital library,
Springer Link and Science Direct
2004–2009 Empirical
and nonempirical
10 2010 Managing Quality
Requirements [53]
18 ACM Digital Library, Compendex and Inspec, IEEE
Xplore, Wiley Inter Science Journal Finder
2008 Empirical
11 2010 Requirements Engineering for
Software Product Lines [33]
49 ACM Digital Library, IEEE Xplore, Science Direct
Elsevier, Wiley Inter Science Journal Finder.
1990–2009 Empirical
Requirements Eng
123
Appendix 2: Quality assessment checklist
This appendix describes the quality instrument that we used
to access the quality of studies. It consisted of 5 sections, a
section having general checklist items which was applied to
all the studies included in the SMS, while other 4 sections
were decided specifically for various research methods used
in the study, i.e., experiment, survey, case study and experience report. These criteria were adopted from SLR guidelines [13, 36–39]. The questions included in the checklist
were answered either ‘‘yes,’’ ‘‘no’’ or ‘‘partial’’ and were
given rates as 2, 1 or 0, respectively. The sum of the scores for
all of these questions was used for assessing the quality of a
primary study.
Sr# Year Title of systematic study # of
studies
Source of primary studies Covered till the time Empirical/
nonempirical
12 2011 Elicitation Techniques
[17]
26 SCOPUS, IEEEXPLORE, ACM DL databases, as
well as Google
Start time: unlimited,
ending time: March
2005
Empirical
13 2011 User Requirements
Notation [22]
281 IEEE Xplore, ACM Digital Library, Google Scholar,
SpringerLink, Scopus
Start time: not
specified ending
time: 2010
Empirical
and nonempirical
14 2012 Stakeholder Identification
Methods [19]
47 ACM Digital Library, IEEE Xplore, Springer Verlag,
Google Scholar, ScienceDirect, Metapress, Wiley
InterScience
1984–2011 Empirical
and nonempirical
15 2012 Software Requirements
Triage and Selection
[28]
23 Scopus, INSPEC, EI Compendex, IEEExplore, ISI
web of science
Not mentioned Empirical
and nonempirical
16 2012 Requirements Evolution
[23]
125 ACM Digital Library, IEEE Xplore, Science Direct,
Springerlink, InterScience
1994–2009 Not
mentioned
17 2012 DRE-Specific Wikis for
Distributed RE [31]
27 ACM portal, Elsevier’sScience Direct, IEEE Xplore,
Springer-Verlag’s Link;
Start time: unlimited,
ending time: 2011
Empirical
18 2012 Causes of Requirement
Change [27]
5 Springer link, IEEE Explore, ACM Digital library,
Cite Seer library, Science Direct, EI Compendex
December 2008 to
March 2009
Empirical
19 2012 Creativity Techniques for
Requirements
Engineering [61]
25 IEEE Xplore, ACM, Compendex, Inspec,
Springerlink, Science Direct
Start time: not
mentioned ending
time: 2011
Empirical
and nonempirical
Quality assessment checklist
Generic

Requirements Eng
123
Appendix 3: Data extraction scheme
This appendix enlists the data extraction items that have been
extracted from the primary studies of the SMS. The RE core/main areas and sub-areas in this data extraction scheme were
decided by consulting SWEBOK [40] and REBOK [41], while
the type of research in this data extraction scheme was formulated according to the research types provided in [42]. The
rest of the items were extracted to carry out a rich analysis and
present various themes and trends as advised in [35].
Quality assessment checklist
Experiment
Were treatments randomly allocated? Yes/no
If there is a control group, are participants similar to the treatment group participants in terms of variables that may affect
study outcomes?
Yes/no

Data extraction items
1. Study ID 2. Reference
type
3. Conference/
Journal
4. Title
5. Authors 6. Publication
year
7. Countries
involved in
research
8. Conference/Journal name 9. Aim of
study
10. Results of
study
11. RE Core Area (RE Fundamentals/RE Process/Reqs Elicitation/Reqs Analysis/Reqs
Specification/Reqs Validation, Verification & Evaluation/Reqs Planning & Management/
Reqs Practical Consideration)
12. RE Sub Area (Reqs Modeling/Enterprise Analysis/Product Analysis/Reqs Prioritization/
Reqs Tradeoff Analysis/Reqs Impact Analysis/Reqs Risk Analysis/Reqs Traceability)
13. RE Emerging Trends
14. Technique/Process/Tool/Framework Name
15. Research Output (New Technique/Tool/Process/Framework, Modification of Technique/
Tool/Process/Framework, Usage experience of Technique/Tool/Process/Framework,
Guidelines, Other)
16. Company Size (Small/Large/Medium/Mixed) 17. Name of
Company
18. Industry/Domain(Telecom/Web/Finance/Automation/Automotive/Medical/Manufacturing/
Governoment/Ecommerce/Education/Generic Software Development)
19. Type of Evidence (Experiment/Case Study/Survey/Experience Report)
20. Data Collection Method (Questionnaire/Interview/Archive Analysis/Observation/Mixed)
21. Type of Research (Evaluation Research/Validation Research/Solution Proposal/
Philosophical Paper/Opinion Paper/Experience Paper)
22. Subjects of Investigation(Academia/Industry/Mixed)
Requirements Eng
123
Appendix 4: Domains of empirical studies of RE
This appendix describes various domains to which empirical
studies of RE belong, along with frequency and percentage of
studies for each domain. Some studies belong to software
development generally, without mentioning of some specific
domain in them, so we have categorized such studies under
‘‘generic software development’’ domain. The studies dealing
with the domains of avionics, medical, automotive, electronics and control systems have been categorized under the category of ‘‘embedded’’ domain. Some studies belonged to
more than one domain, so we categorized such studies under
‘‘multiple’’ domain category.
Appendix 5: Research interventions in empirical
studies of RE
This appendix describes various interventions reported in
empirical RE research. The type of these interventions has
already been discussed in body section of this paper.
These interventions in this appendix have been provided
per each core area of RE, to let practitioners get a handful
of these empirically evaluated interventions while practicing some activities from the RE process. The organization of interventions this way can also be helpful for the
RE researchers to get a collection of various existing
empirically evaluated interventions in case they are
attempting to develop new interventions of RE or want to
modify/replicate existing interventions. The research output ‘‘guidelines’’ is missing in this appendix, as guidelines
cannot be summarized like this; also we left this part for
the future work.
Domains Frequency of
studies
Percentage
Domains of empirical studies
Generic software development 64 24
Multiple domain 44 16
Embedded 35 13
Telecom 19 7
Management information
systems
19 7
Finance 18 7
Web 17 6
Education 9 3
E commerce 5 2
Manufacturing 3 1
Other 37 14
Requirements Eng
123
Research interventions in empirical studies of RE
Requirements elicitation
New technique ROADMAP
QRF (Quality requirements of a software Family)
Structured digital storytelling
CREE
StakeRare
Agent-based goal elicitation (ATABGE)
Interview-driven requirements elicitation
Scenario weaving
ORE (Ontology-based requirements elicitation)
Confidentiality requirements elicitation and engineering
New tool Gaius T
New process Display-Action-Response Model
IRIS (Integrating Requirements and Information Security)
Enterprise Analyzer
UEProject (Usability Evaluation Project)
Domain-specific requirements model for scientific computing
CelLEST
Cognitive-Driven Requirements Prioritization Process
Cognitive Psychology Approach for Balancing Elicitation Goals
SQUARE (Security Quality Requirements Engineering)
Model describing the relationships between
Threats, security requirement types and related IT infrastructure components
New framework RE-GSD (Requirement Elicitation for Global Software Development projects)
A framework to support alignment of secure software engineering with legal regulations
RE-GSD
Strategy-based process for requirements elicitation
Non Functional Model
A framework of analysis of group performance in synchronous text-based
distributed requirements elicitations and negotiations.
Modification of technique EasyWinWin modified to WikiWinWin
Usage experience of technique Group story telling
Prospect theory
Scenarios
Appreciative Inquiry
Scenario Acting
Usage experience of tool Cerno
REE(Requirements Engineering Environment)
Usage experience of process GORE(Goal Oriented Requirements Engineering)
SREP(security requirements engineering process)
Usage experience of framework Nomos
New tech &tool OREW (domain Ontology Reconstruction Environment by Web search)
Comparison of techniques Attack trees & Misuse cases
Full EPMcreate & Power-Only EPMcreate
Optimization of full EPMcreate &Traditional Brainstorming
Requirements Eng
123
Research interventions in empirical studies of RE
Requirements analysis
New technique Human facilitation in computer-mediated requirements meetings
SIREN (SImple REuse of software requirements)
heuristic decision making algorithm
Automated similarity analysis
l-Strategy
Lightweight Semantic Processing
SBSE (Search-Based Software Engineering)
Business process modeling method
Fuzzy QFD (fuzzy quality function deployment)
RA (Relationship Analysis)
Security Requirements Analysis and Secure Design Using Patterns and UMLsec
Scenario transformation method
New tool IntelliReq
JSPWikiWinWin
useystem case retrieval system
RE Context
requirements analysis supporting system
New process Distributed Prioritization Process
NFR Evaluation Model
New framework VOP (Value-oriented Prioritization)
Goals-Skills-Preferences Framework
Staged Modelling Methodology
Modification of technique Use Case
Usage experience of technique Scenarios
AHP for requirements prioritization
WinWin
Prototyping
Heuristic Critiques
Usage experience of tool QuARS
QARCC-1
Usage experience of process SFMEA(Software Failure Modes and Effects Analysis)
RAM(Requirements Abstraction Model)
Usage experience of framework i* Modelling Framework
New tech &tool Automated requirements classification technique
Comparison of techniques Use Cases & Tropos
F2F communication & COFFEE & Second Life
Lexical similarity & Searching and filtering
Single-Objective GA & FOOM & OPM (Object-Processes Methodology)
UML Use Case (UC) model & OODFD Transaction
Analytic Hierarchy Process method (AHP) & Case-Based Ranking method (CBRank)
Non-dominated Sorting Genetic Algorithm-II (NSGA-II) & Pareto GA
Comparison of tools ARENA II(Anytime, Anyplace REquirements Negotiation Aids) & ARENA-M
((Anytime, Anyplace REquirements Negotiation Assistant – Mobile)
Requirements Eng
123
Research interventions in empirical studies of RE
Requirements specification
New technique Clone Detection
SCR(software cost reduction)
Structuring specification documents by using temporally adjacent topics
High-level requirements engineering methodology for electronic system-level design
Information model approach
CSRML (Collaborative Systems Requirements Modelling Language)
SOFL(Structured-Object-based-Formal Language)
New tool LAMPS(Learning Action Model from Plan Samples)
New process AutoRELAX
New framework PDS(Problem Decomposition Scheme)
DAM(domain analysis methodology)
Theoretical Framework of Requirements Documentation Styles
Modification of technique TORE(Task and Object Oriented Requirements Engineering)
Use case
Z Language
Usage experience of technique OCL (Object Constraint Language)
GQM (Goal-Question-Metric)
FRORL (Frame-and-Rule Oriented Requirements specification Language)
Usage experience of tool SeCSE’s Service Discovery Environment
Usage experience of process Performance Refinement and Evolution Model
Usage experience of framework AUTOSAR
Comparison of techniques Use case & textual Approach
F2F communication & Think-Pair-Square
Requirements verification, validation and evaluation
New technique SQ2E (Scenario Question Query Engine)
Requirement Error Taxonomy
ALIGNMENT OF ONTOLOGY AND MODELS
New tool SRA (System Reliability Analyser)
MEG
New framework and tool GRIP (Groupware-supported Inspection Process)
Usage experience of technique UML Diagrams
CBR(checklist-based reading) and SBR (scenario-based reading)
Symbolic Model Checking
Perspective-Based Reading (PBR)
New tech &tool CREWSAVRE (Scenarios for Acquisition and Validation of Requirements)
I VT (Input Validation Testing)
MICASA (Method for Input Cases and Static Analysis)
Requirements Eng
123
Research interventions in empirical studies of RE
Requirements planning and management

RC Cost Pre-evaluation
Owner ship based user group model
Automatically Structuring Textual Requirement Scenarios
Traceability-Based Notification Strategy
UMGAR (UML Model Generator from analysis of Requirements)
Rule-based generation of requirements traceability relations
PiLGRIM (Propagating i*-Led Goal-Requirement Impacts)
FoCM (Feature-oriented requirements Change management Method)
Value-based analysis method for variability evolution
Business Process-driven Approach for Requirements Dependency Analysis

User-centered requirements engineering
RE process for Web Service project
domain requirements development process
Evolutionary model of RE
RDMod (Requirements data model)
SREP (Security Requirements Engineering Process)
SecuRUP(security requirements engineering conformed to RUP)
RE process model for projects in emerging markets
SREPPLine (Security Requirements Engineering Process for software Product Lines)
Requirements Engineering using Prototyping Projects in Healthcare Diagnostic Software Application
Requirements Eng
123
References
1. IEEE, International Conference on Requirement Engineering.
http://www.requirements-engineering.org/
2. Springer, International Journal of Requirement Engineering.
http://www.springer.com/computer/swe/journal/766
3. Nuseibeh B, Easterbrook S (2000) Requirements engineering: a
roadmap. In: Proceedings of the conference on the future of
software engineering. ACM, pp 35–46
4. Cheng BH, Atlee JM (2007) Research directions in requirements
engineering. In: 2007 Future of software engineering. IEEE
Computer Society, pp 285–303
5. Davis A, Hickey A, Dieste O, Juristo N, Moreno A (2007) A
quantitative assessment of requirements engineering publications—1963–2006. In: Requirements engineering: foundation for
software quality. Springer, Berlin, pp 129–143
6. Lami G, Ferguson RW (2007) An empirical study on the impact
of automation on the requirements analysis process. J Comput Sci
Technol 22(3):338–347
7. Davis AM, Hickey AM (2002) Requirements researchers: do we
practice what we preach? Requir Eng 7(2):107–111
8. Wieringa RJ, Heerkens JMG (2006) The methodological soundness of requirements engineering papers: a conceptual framework
and two case studies. Requir Eng 11(4):295–307
9. Wasson KS (2004) Requirements metrics: scaling up. In: Proceedings of 2nd international workshop on comparative evaluation in requirements engineering (CERE’04), Kyoto, Japan,
pp 51–55
10. Seaman CB (1999) Qualitative methods in empirical studies of
software engineering. IEEE Trans Softw Eng 25(4):557–572
11. Kitchenham BA, Dyba T, Jorgensen M (2004) Evidence-based
software engineering. In: Proceedings of the 26th international
conference on software engineering. IEEE Computer Society,
pp 273–281
12. Ambreen T, Usman M, Ikram N, Bano M (2011) Evidence in
requirements engineering: a systematic literature review protocol.
In: The Sixth international conference on software engineering
advances. ICSEA, pp 576–580
13. Kitchenham B, Charters S (2007) Guidelines for performing
systematic literature reviews in software engineering (version
2.3). Technical Report, Keele University and University of
Durham
14. Bjørnson FO, Dingsøyr T (2008) Knowledge management in
software engineering: a systematic review of studied concepts,
findings and research methods used. Inf Softw Technol
50(11):1055–1068
15. Jorgensen M, Shepperd M (2007) A systematic review of software development cost estimation studies. IEEE Trans Softw Eng
33(1):33–53
16. Qureshi N, Usman M, Ikram N (2013). Evidence in software
architecture, a systematic literature review. In: Proceedings of the
17th international conference on evaluation and assessment in
software engineering. ACM, pp 97–106
17. Dieste O, Juristo N (2011) Systematic review and aggregation of
empirical studies on elicitation techniques. IEEE Trans Softw
Eng 37(2):283–304
Research interventions in empirical studies of RE
New framework Organization modeling-based requirements engineering framework
Software requirements management (Quality Function Deployment) framework based on QFD
REPI (Requirements Engineering Process Improvement)
IterativeC-K requirements engineering process
FRERE (Framework for requirements engineering process development)
REF (Requirements engineering framework)
Iterative requirements engineering process using FCA(Formal concept Analysis)
Modification of process XP requirements process
Praxis
Usage experience of Technique Gilb Style
SQUARE (Security Quality Requirements Engineering)
VORD (The Viewpoint-Oriented Requirements Definition method)
Usage experience of tool PVCS
Wikis
Usage experience of process Requirements Process Maturity Model
Usage experience of framework TORE (task-oriented requirements engineering framework)
Requirements practical considerations
New technique Gilb Style
New tool BBN-based expert system
Requirements Quality Model
KASRET (Knowledge-based Approach for the Selection of Requirements Engineering Techniques)
Requirements engineering fundamentals
New process Ontology for Requirements Engineering
New framework Framework of Power & Politics in RE
Requirements Eng
123
18. Davis A, Dieste O, Hickey A, Juristo N, Moreno AM (2006)
Effectiveness of requirements elicitation techniques: empirical
results derived from a systematic review. In: 14th IEEE international conference requirements engineering. IEEE, pp 179–188
19. Pacheco C, Garcia I (2012) A systematic literature review of
stakeholder identification methods in requirements elicitation.
J Syst Softw 85(9):2171–2181
20. Nicola´s J, Toval A (2009) On the generation of requirements
specifications from software engineering models: a systematic
literature review. Inf Softw Technol 51(9):1291–1307
21. Condori-Fernandez N, Daneva M, Sikkel K, Wieringa R, Dieste
O, Pastor O (2009) A systematic mapping study on empirical
evaluation of software requirements specifications techniques. In:
Proceedings of the 2009 3rd international symposium on empirical software engineering and measurement. IEEE Computer
Society, pp 502–505
22. Amyot D, Mussbacher G (2011) User requirements notation: the
first ten years, the next ten years. J Softw 6(5):747–768
23. Li J, Zhang H, Zhu L, Jeffery R, Wang Q, Li M (2012) Preliminary results of a systematic review on requirements evolution. In: Evaluation & assessment in software engineering (EASE
2012), 16th International Conference on, IET, pp 12–21
24. Herrmann A, Daneva M (2008) Requirements prioritization based
on benefit and cost prediction: an agenda for future research. In:
16th IEEE international requirements engineering. RE’08. IEEE,
pp 125–134
25. Winkler S, Pilgrim J (2010) A survey of traceability in requirements engineering and model-driven development. Softw Syst
Model (SoSyM) 9(4):529–565
26. Walia GS, Carver JC (2009) A systematic literature review to
identify and classify software requirement errors. Inf Softw
Technol 51(7):1087–1109
27. Bano M, Imtiaz S, Ikram N, Niazi M, Usman M (2012) Causes of
requirement change—a systematic literature review. In: 16th
international conference on evaluation and assessment in software engineering (EASE 2012). IET, pp 22–31
28. Khurum M, Uppalapati N, Veeramachaneni RC (2012) Software
requirements triage and selection: state-of-the-art and state-ofpractice. In: 2012 19th Asia-Pacific Software Engineering Conference (APSEC), vol 1. IEEE, pp 416–421
29. De Azambuja FB, Bastos RM, Bacelo APT (2009) Systematic
review of requirements reuse. In: SEKE, pp 67–72
30. Lopez A, Carrillo-de-Gea JM, Toval A (2009) Risks and safeguards for the requirements engineering process in global software development. In: Fourth IEEE international conference on
global software engineering. ICGSE 2009. IEEE, pp 394–399
31. Peng R, Lai H (2012) DRE-specific wikis for distributed requirements engineering: a review. In: 2012 19th Asia-Pacific Software
Engineering Conference (APSEC), vol 1. IEEE, pp 116–126
32. Blanes D, Insfran E, Abraha˜o S (2009) Requirements engineering
in the development of multi-agent systems: a systematic review.
In: Intelligent Data Engineering and Automated Learning-IDEAL
2009. Springer, Berlin, pp 510–517
33. Alves V, Niu N, Alves C, Valenc¸a G (2010) Requirements
engineering for software product lines: a systematic literature
review. Inf Softw Technol 52(8):806–820
34. Cruzes DS, Dyba˚ T (2011) Research synthesis in software engineering: a tertiary study. Inf Softw Technol 53(5):440–455
35. Cruzes DS, Dyba T (2011) Recommended steps for thematic
synthesis in software engineering. In: 2011 International symposium on empirical software engineering and measurement
(ESEM). IEEE, pp. 275–284
36. Kitchenham BA, Brereton OP, Budgen D, Li Z (2009) An
evaluation of quality checklist proposals: a participant-observer
case study. In: Proceedings of the 13th international conference
on evaluation and assessment in software engineering. British
Computer Society, pp 55–64
37. Kitchenham B, Sjøberg DI, Brereton OP, Budgen D, Dyba˚ T,
Ho¨st M, Runeson P (2010) Can we evaluate the quality of software engineering experiments? In: Proceedings of the 2010
ACM-IEEE international symposium on empirical software
engineering and measurement. ACM, p 2
38. Ho¨st M, Runeson P (2007) Checklists for software engineering
case study research. In: ESEM, pp 479–481
39. Budgen D, Zhang C (2009) Preliminary reporting guidelines for
experience papers. In: Proceedings of EASE, vol 2009, pp 1–10
40. Bourque P, Dupuis R (2004) Guide to the software engineering
body of knowledge 2004 Version. In: SWEBOK
41. Aoyama M, Nakatani T, Saito S, Suzuki M, Fujita K, Nakazaki
H, Suzuki R (2010) A model and architecture of REBOK (Requirements Engineering Body of Knowledge) and its evaluation.
In: 2010 17th Asia Pacific Software Engineering Conference
(APSEC). IEEE, pp 50–59
42. Wieringa R, Maiden N, Mead N, Rolland C (2006) Requirements
engineering paper classification and evaluation criteria: a proposal and a discussion. Requir Eng 11(1):102–107
43. https://drive.google.com/file/d/0B8nZBcJOcv2MYVJEa3VPbHR
sWlU/view?usp=sharing
44. Sikora E, Tenbergen B, Pohl K (2012) Industry needs and
research directions in requirements engineering for embedded
systems. Requir Eng 17(1):57–78
45. Daneva M et al (2014) Empirical research methodologies and
studies in requirements engineering: how far did we come? J Syst
Softw 95:1–9
46. Talbot A, Connor AM (2014) Requirements engineering current
practice and capability in small and medium software development
enterprises in New Zealand. arXiv preprint arXiv:1407.6102
47. Aranda J, Easterbrook S,Wilson G (2007) Requirements in the wild:
how small companies do it. In: 15th IEEE international requirements
engineering conference, 2007. RE’07. IEEE, pp 39–48
48. Sadraei E, Aurum A, Beydoun G, Paech B (2007) A field study of
the requirements engineering practice in Australian software
industry. Requir Eng 12(3):145–162
49. Liu L, Zhang H, Ma W, Shan Y, Xu J, Peng F, Burda T (2009)
Understanding chinese characteristics of requirements engineering. In: 17th IEEE international requirements engineering conference, 2009. RE’09. IEEE, pp 261–266
50. Liu L, Li T, Peng F (2010) Why requirements engineering fails: a
survey report from china. In: 2010 18th IEEE international
requirements engineering conference (RE). IEEE, pp 317–322
51. Solemon B, Sahibuddin S, Ghani AAA (2010) Adoption of
requirements engineering practices in Malaysian software
development companies. In: Advances in software engineering.
Springer, Berlin, pp 141–150
52. Juristo N, Moreno AM, Silva A (2002) Is the European industry
moving toward solving requirements engineering problems?
IEEE Softw 19(6):70–77
53. Svensson RB, Ho¨st M, Regnell B (2010) Managing quality requirements: a systematic review. In: 36th EUROMICRO conference on
software engineering and advanced applications (SEAA), pp 261–268
54. Wohlin C, Runeson P, Ho¨st M, Ohlsson MC, Regnell B, Wessle´n
A (2000) Experimentation in software engineering—an introduction. Kluwer Academic, Boston
55. Lindsay RM, Ehrenberg A (1993) The design of replicated
studies. Am Stat 47(3):217–228
56. Popper K (1959) The logic of scientific discovery. Hutchinson &
Co, Bangkok
57. Schmidt S (2009) Shall we really do it again? The powerful
concept of replication is neglected in the social sciences. Rev Gen
Psychol 13(2):90
Requirements Eng
123
58. Franc¸a ACC, da Cunha PR, da Silva FQ (2010) The effect of
reasoning strategies on success in early learning of programming:
lessons learned from an external experiment replication. In:
International conference on evaluation and assessment in software engineering, pp 81–90
59. Lung J, Aranda J, Easterbrook S, Wilson G (2008) On the difficulty of replicating human subjects studies in software engineering. In: ACM/IEEE 30th international conference on
software engineering, 2008. ICSE’08. IEEE, pp 191–200
60. Ivarsson M, Gorschek T (2009) Technology transfer decision
support in requirements engineering research: a systematic
review of REj. Requir Eng 14(3):155–175
61. Saha SK, Selvi M, Buyukcan G, Mohymen M (2012) A systematic review on creativity techniques for requirements engineering. In: 2012 International conference on informatics,
electronics & vision (ICIEV). IEEE, pp 34–39
Requirements Eng
123