Cost and benefit analysis of quality requirements in competitive software product management: A case study on the QUPER model

In market-driven product development, it is important that the software product is released to the market at the right time, and offers higher quality than the competitors. In release planning, the allocation of development effort in investments into product enhancements, functions are competing with quality requirements for limited resources. In addition, it is important to find the right balance between competing quality requirements. In this paper, we present an industrial evaluation of the benefit and cost views in the QUality PERformance (QUPER) model. The results indicate that the model is relevant in the release planning process, and that the combination of the benefit and cost views provides a clear picture of the current market situation.


INTRODUCTION
Software development for the mass-market (marketdriven development), rather than for a specific customer, is becoming more common in the software industry [7,22].One major goal of market-driven product development is to develop a product with high customer value.To increase the chance of market success, it is important that the software product is released to the market at the right time, and offers a higher level of quality than the competitors' products [2].However, one problem is to decide what to include in the next release of the product, hence release planning is a major determinant of the success of a product [17,22].Moreover, lack of good release planning practices may result in unsatisfied customers [22].
Release planning involves activities such as selecting what features and requirements should be in a certain release, when it should be released and at what cost [22].The release planning process is closely associated with product management and requirements engineering decisions [23].In literature, there are several approaches that provide support for release planning and prioritization.Examples of release planning approaches are Release Planning Prototype [5], EVOLVE [10], and release planning through optimization and what-if analysis [1].Although it may be possible to define release planning as a mathematical optimization problem, it may not be worthwhile to apply complex mathematics or advanced computational algorithms to achieve "optimum", if the input data to the optimization process is highly uncertain.To the best of our knowledge, little research has looked into release planning and prioritization of quality requirements (QR) [6], despite that QR are of major importance for market success [2,11].To support release planning and roadmapping of QR, we have developed the QUality PERformance (QUPER) model [3,4,16,18], with the goal to provide concepts for qualitative reasoning of orders of magnitude rather than precise mathematical formulas.
QUPER was developed at two case companies in the mobile handset domain [18].Generic guidelines of how to apply QUPER in practice were developed in cooperation between academia and industry [16].Furthermore, the tailoring and evaluation of QUPER's benefit view (see Section II) were carried out at a case company using action research [3].Moreover, modeling guidelines of QUPER's cost view (see Section II) were carried out at a case company [4].
This paper presents one case of an evaluation of QUPER's benefit and cost views in the electronic payment domain.Evaluations of QUPER have been performed in [3,4]; however, the evaluation in [3] only concerned the benefit view, while in [4] the evaluation concerned the creation of guidelines for QUPER's cost view.The electronic payment domain allows us to validate QUPER's usefulness in, not only a different domain than the mobile handset domain, but also in a non-simulated environment in real projects using real requirements.The main objectives and contributions of the paper are to show that QUPER is applicable in domains other than the mobile handset domain, but in particular the focus on the evaluation of the industrial introduction of using both the benefit and cost views together.
The paper is organized as follows.Section 2 presents a short introduction to the QUPER model and related work.In Section 3, the case company and its products where QUPER was evaluated is presented, while Section 4 presents the research methodology.Section 5 presents the results from the evaluation and Section 6 concludes the paper.

II. BACKGROUND AND RELATED WORK
In this section, the notations and a short introduction to the QUPER model are presented.Furthermore, related work to release planning and prioritization is presented.

A. QUPER
QUPER aims to support requirements prioritisation and roadmapping of quality requirements at early stages of release planning when making high-level scoping decisions and creating roadmaps.A major objective of QUPER is to define a feature prioritisation model that includes a third dimension related to quality, in addition to cost and value that are used in prioritisation of functional requirements [14].The QUPER model is based on the observations that quality is continuous and non-linear.The quality level is typically not viewed as either good or bad, but rather as something with different shades of goodness on a sliding scale.
The QUPER model has two main concepts: breakpoints and barriers.A breakpoint is an important aspect of the nonlinear relation between quality and benefit, while barriers represent an interesting aspect of the non-linear relation between quality and cost.The two concepts of breakpoints and barriers form the basis of QUPER's three views: (1) the benefit view, (2) the cost view, and (3) the roadmap view.
The benefit view (see Figure 1) includes three breakpoints.The utility breakpoint marks the border between useless and useful quality.Useless means that a product is not accepted on the market and does not recognize its value.The differentiation breakpoint marks the shift from useful to competitive quality, which makes them have a competitive market position.The saturation breakpoint implies a change in quality level from competitive to excessive quality, where higher quality levels have no practical impact on the benefit in the particular usage context considered.The cost view (see Figure 2) includes foreseen cost barriers.For a specific quality aspect in a specific context, we approximate the quality-cost relation to have two different steepness ranges.A typical cost barrier may be the result of that a quality increases is not feasible without a large reconstruction of the product architecture, while a typical cost plateau is exemplified by the case where comparatively inexpensive software optimizations may result in high gains of performance.The roadmap view (see Figure 3) combines the benefit and cost views by position the breakpoints and barrier together ordered on the same scale.This view enables visualization of benefit breakpoints and cost barriers in relation to the current quality level of a product and the qualities of competing products.This view also combines the notation of targets for coming releases with the aim of supporting roadmapping.The QUPER model's benefit view is evaluated in Berntsson Svensson et al. [3].The main lessons learnt by applying QUPER in an industrial context show that it was a good basis for decisions about introduction of new products to the market and it provided more informed decisions and a good overview of the current market.The main identified challenge by using QUPER' benefit view was difficulties to identify and specify the values for the differentiation and saturation breakpoints.
The practical guidelines for QUPER's cost view are evaluated in Berntsson Svensson et al. [4].In general, the practitioners believe it is possible to identify and estimate the cost barriers by applying the practical guidelines.In addition, the evaluation indicates that the cost view may be of great value for creating business cases.The main identified challenge was the uncertainty of the estimated cost for the second cost barrier.

B. QUPER Guidelines
QUPER as presented in Section II.A and its practical application in Regnell et al. [16] are generic in nature.The practical application of QUPER's benefit view in [3] and the cost view in [4] are adopted prior to the model being set into operation at the companies.The practical guidelines were developed in cooperation between academia and industry.
To apply QUPER's benefit and cost views in practice, the following steps have been identified [3,4]: 1. Define quality aspects.In step one, when defining quality aspects, it is important to identify relevant qualifiers and consider their potential consequences.For example, different products offered to different market segments may have different levels of quality for the same quality requirement.
In step two, when quality aspects are identified, identify reference levels based on actual products, your own (Qref) and your competitors'.These reference levels helps in calibrating the breakpoints.
Step three defines the market's expectations in terms of breakpoints (see Figure 1).First, estimate the utility breakpoint (UBP) -the lowest acceptable level of quality on the market.Then, estimate the saturation breakpoint (SBP).The SBP represents quality levels that are considered excessive.Finally, estimate 1the differentiation breakpoint (DBP), which represents quality levels that give market advantages.
In step four, estimate the first cost barrier (CB1).CB1 may relate to software changes, e.g.requires a change in one or a few parts of the architecture.Moreover, CB1 may only affect your own and/or closely related projects' architecture.It is important to note that C1 represents the penalty of raising the quality level from the current product's (Qref) quality level.
In step five, estimate the second cost barrier (CB2).CB2 may affect major, if not all, parts of the entire product's architecture.The hardware's physical constraints may be used as input for Q2.It is important to note that C2 represents the cost penalty given that given that C1 investment has been made.
In the final step, the estimated candidate targets are requirements with potential quality commitments.The actual requirement is specified as an interval by a min target (lowest acceptable quality for this requirement) and a max target (the highest needed quality level for this requirement).The specification of the requirements is assisted by the steps prior to this one.Actual targets for the upcoming release or future releases are chosen from the defined candidate targets.This is an example of how QUPER supports release planning.For further discussion on QUPER and release planning see [16].

C. Related Work
The process of selecting requirements for a certain release of a software product is called release planning [17,22], where the primary goal is to maximize the expected value of a product release.In literature there are several approaches for release planning, for example, EVOLVE [10], EVOLVE* [20], F-EVOLVE [15], Release Planner Prototype [5], and a method for software release planning through optimization of what-if analysis [1].All of these methods use generic algorithms and linear programming to solve the problem of release planning.Input data such as requirements value, requirements cost, resources, stakeholder importance, and budget constraints are used to come up with an "optimum" release.In addition, steps for analyzing dependencies among requirements are presents in the above mentioned approaches.
Prioritization of requirements is often conducted prior, or as a part of the release planning process [9].Several prioritization techniques and cost-benefit models are introduced in the literature.The contributions in this area include: Kano [12], Planguage [8], a cost-value approach [14] based on the analytical hierarchical process [21], and quality function deployment (QFD) [13].Planguage has roadmap related concepts such as past, record, and trend in templates for quality requirements.QUPER could be used together with the planguage method to express breakpoints, barriers, and targets related to, for example, express competing products in different market segments.
The cost-value approach uses a two-dimension graph that displays the requirements value against its cost.AHP is used from a customer and user perspective to assess the value of each requirement, followed by an assessment of the requirements cost from an implementation perspective.The next step is to plot these into a cost-value diagram, which is used to analyze and discuss the requirements.This approach, supporting trade-off analysis and is mainly used for functional requirements.Quality requirements can be included as objects of prioritization in AHP, but as discrete objects are compared against each other, the relation to a sliding scale is not explicitly addressed.The QUPER model thus goes further by introducing a third dimension related to the continuous nature of quality attributes.There are potential strategies for combining QUPER with AHP-based approaches, e.g. by comparing breakpoints of different use cases.
Similar to QUPER, Kano et al. developed a model for evaluating patterns of quality [12].The evaluation is based on customer's satisfaction with specific quality attributes.Kano's model explains the relationship between customer satisfaction and the degree of achievement of a specific quality attribute in a two-dimension graph.Kano's approach views quality as non-linear.However, Kano's approach does not include a cost dimension.Moreover, Kano's model is not related to roadmapping, benefit breakpoints, or cost barriers to indicate important aspects of quality relations.
QFD is a comprehensive, customer and user oriented approach to product development.The QFD process starts by organizing the project, including the formation of a cross-functional team, followed by the establishment of relationships among requirements and then prioritization.To fully implement QFD, customers and users need to be visible; however, not all market-driven projects have access to customers and users [13].Furthermore, QFD measures quality attributes using a scale where no clear distinctions between the values are provided.While QFD may require a complete change of current practices, QUPER is a simple reference model to be used in combination with current practices to support communication of quality attributes.

III. CASE COMPANY DESCRIPTION
The case company employs more than 250 employees, has more than 120,000 customers' and business partners, and operates in the electronic payment processing market.The case company is an international organization that specializes in payment terminals, transaction processing and development of saving-and customer card systems.In order to structure and regulate the electronic payment market, third party organizations coordinate and set standards for the market, and how organizations are allowed to operate with other parties, e.g.banks and acquirers.Moreover, the third parties define a majority of the functionalities and requirements, which leaves little room for the case company to differentiate itself from its competitors.Hence, quality requirements play an important part.
At the moment, the case company does not have any defined quality standard to adhere to in relation to product requirements.First, product requirements are defined at a high abstraction level with input from the market.Then, the requirements are specified in more details for the software development team.It is the responsibility of the development team to estimate the realization costs and to communicate this with management.Then, management decides, in collaboration with the development team, what requirements should be included in the upcoming release, and what requirements are planned for future releases.After a realization acceptance test is performed, a pilot test is conducted to test the requirements.
Since the case company is growing internationally, there is a higher focus on identification and gathering of quality requirements.In order to monitor the quality levels, a more structured approach to deal with quality requirements is needed.However, at the moment, when a software upgrade is about to be released, quality requirements are dealt with in an ad-hoc approach, unlike the product requirements.One of the case company's larger software upgrade releases for 2010 is the first release where quality requirements are elaborated.
Daily challenges faced by the case company include, how can we measure the quality of our products?When is the quality level good enough, and how much would it cost us?The case company does realize that quality requirements need to be planned and decided in the early decision-making process (such as release planning), before the initiation of the software development process.

IV. RESEARCH METHODOLOGY
The research was carried out in cooperation between academia and industry using an action research approach [19].In action research, the researcher enters a project where tasks are performed by using the researchers method.The aim of action research is to influence or change some aspects of the research focus.Moreover, action research aims to improve: practice, the understanding of practitioners, and the situation in which the practice took place [19].The cycle of action research comprises of four steps [19]: (1) plan to improve current practice, (2) implementation of the plan (action), (3) observe effects, and (4) reflection.After the reflection step, the researcher evaluates the performance of the used method and draw conclusions.
The aims of this study are to improve the release planning process with regards to quality requirements by introducing QUPER's cost and benefit views, to understand how practitioners use QUPER in an industrial environment, and how QUPER adapts to existing processes.
Four interview subjects were chosen to give a rich picture.The four subjects represent different roles at the case company.The roles chosen for the evaluation are, two product managers, one product quality manager, and one acceptance manager.
The study consists of the following three phases, where the first phase is related to the first step in the action research cycle, while the second phase related to step two and three.The third phase in this study is related to step four in the action research cycle.

A. Phase 1 -Planning
Phase 1 involved a brainstorming and planning meeting to plan the study and how to apply QUPER at the case company, and to identify different areas of interest for the evaluation.Six quality requirements were identified for applying QUPER in different areas such as: maintainability, usability, efficiency and reliability.The interview instrument (see Table I) was designed with respect to the different areas of interest and inspiration from [3].To test the interview instrument, a pilot interview was conducted prior to the industry study.Some questions were clarified and the structure of the interview instrument was improved before interviewing proceeded.

B. Phase 2 -Workshop
Presentations: QUPER and how to use QUPER in practice was presented in several steps.First, an introduction of QUPER was presented to a product manager.When the product manager showed an interest in QUPER, a presentation of QUPER and its practical application was given to several managers at different departments.The next step involved planning the approach of applying QUPER at the case company using the guidelines presented in Section II.B.As a result of the planning step, a final presentation of QUPER and its guidelines was given to managers and experts.These representatives are selected based on their roles and expertise by the local managers.
Apply QUPER in practice: During the application of QUPER on real requirements, the second author provided help and feedback to the subjects about applying QUPER on their requirements.In addition, the second author had the opportunity to observe the effects of QUPER's practical guidelines.Due to time availability of the case company and absence of vital information for the application of QUPER on some quality requirements, the case company gave priority to one maintainability and one efficiency quality requirement.
As the workshop is concluded, the main goal is to achieve an understanding of how to use QUPER on real requirements in coming projects.Within 3 months, it was possible for the second author to study, introduce, apply and evaluate QUPER at the case company without any direct support.The people involved in the application of QUPER at the case company spent between 10 and 13 hours in performing QUPER activities.The spent hours are an accumulation of: an introduction to QUPER, presentations and general meetings, brainstorm and planning meetings, how to apply QUPER in practice (using QUPER guidelines in Section II.B), result meetings and evaluation meetings.Except for two evaluation meetings, all meetings were between one and two hours.

Data collection:
The study used a semi-structured interview strategy [19].All interviews were attended by one interviewee and one interviewer.First, the purpose of the study was presented and questions about the different areas of interest in relation to QUPER were discussed in detail.Interviews varied between 35 and 60 minutes.Transcripts of all interviews were made in order to facilitate and improve the analysis process.
Data analysis: The content analysis [19] involved creating categories where interesting parts from the interview transcripts were categorized and discussed.The second author examined the categories from different perspectives and searched for explicitly stated or concealed pros and cons in relation to QUPER.The results from the analysis are found in Section V.

D. Validity
In this section, threats to validity in relation to research design and data collection are discussed.We consider the four perspectives of validity threats as presented in Wohlin et al. [24].

Conclusion validity:
Threats to conclusion validity arise from the ability to draw accurate conclusions.Each interview was done in one uninterrupted work session.This, the answers were not influenced by internal discussions about the questions.To obtain highly reliable measures and to avoid poor question wording and poor layout, a pilot study was conducted.The sampling technique can pose a threat to the validity of the investigation.The subjects selected may not be representative of the role they represent at the company.The main assurance that this misrepresentation is minimized is the fact that the subjects were selected in cooperation with senior manager.The selection was based on what individuals had been using QUPER at the company.
Internal validity: This threat is related to issues that may affect the causal relationship between treatment and outcome.In our study, the research instrument was developed with close reference to literature relating to quality requirements, and influenced by a previously validated research instrument [3], which mitigates the instrumentation threat.In addition, maturation threats are handled by reducing the duration of interview sessions by collecting background information before the interview, and by keeping the interview session to 60 minutes.As their answers were registered by the researcher this could have constrained people in their answers.This potential problem was alleviated by the guarantee of anonymity as to all information divulged during the validation, and that recorded answers was only to be used by the researcher, i.e., not to be showed or used by any other party.
External validity: This threat is concerned with the ability to generalize the findings beyond the actual study, i.e., in this case the applicability of QUPER in industry at companies other than the case company.Some of the problems introduced, as a motivation behind QUPER, could to some extent be general for organizations faced with developing embedded systems for markets.However, strictly speaking, it is not possible to generalize the results from this evaluation based on the case company; although from a perspective of the concepts and practical application of QUPER as described in this paper and in [3,4,16,18] can give an overview of the challenges facing the companies where QUPER has been implemented.

Construct validity:
The construct validity is concerned with the relation between theories behind the research and the observations.The variables in our research are measured through interviews, including open-ended aspects where the participants are asked to express their own opinions.Monooperation bias was avoided by collecting data from four subjects representing four different roles and backgrounds on the topic of the study.To avoid evaluation apprehension, complete anonymity from other participants, the companies, and researchers was guaranteed.

V. RESULT AND ANALYSIS
The three following sub-sections present and discuss general evaluation results of QUPER, evaluation of the benefit view, and evaluation of the cost view.

A. QUPER -General
Table II illustrates the general results from this study.One major difference between how quality requirements were handled in their previous process and QUPER is that most of the estimations were based on a gut feeling.When QUPER was introduced, quality requirements where clearly defined and measured.Furthermore, with the previous process, the case company did not know how they were positioned on the market compared to its competitors.Due to the introduction of QUPER, the company constantly thinks of, measures, and defines its position on the market, which improves the early decision-making process by raising discussions.

General about QUPER General comments
(1) Improves the early decision-making process (2) a better understanding of the competitors, the aims of the level of quality and at what cost Challenges (1) Moving from no method/process of dealing with QR to a method with new terminology (2) find time to use QUPER Using QUPER (1) QUPER is understandable, not difficult (2) after one presentation QUPER was clear and the views where easy to remember Decisionmaking (1) Supports and coordinates release planning and product roadmapping (2) more informed decisions and a visual roadmap improves the communication between the decision makers Improvements compared to previous way of working (1) QR are clearly defined and quantified (2) a clear view of when to stop improving the quality The combination of the benefit and cost views provides a clear picture of the market situation, what quality level to aim for and at what cost.
One of QUPER's goals is that the model should be easy to use and learn.All subjects agreed that it is easy to decide which quality requirements that are applicable to apply and use the concepts of QUPER for.In addition, the subjects viewed QUPER as understandable, not very easy, but definitely not difficult.This view of QUPER is partly inline with the results in Berntsson Svensson et al.where all subjects confirmed that QUPER is easy to learn and use [3].In addition, a common terminology among the staff improves the communication.QUPER's views help to improve the communication to management; there is no need for a long description when a roadmap like QUPER's is presented.
In general, QUPER does not only help in creating a more aligned view of quality requirements, but also to use one method to measure all quality requirements.All subjects confirmed that QUPER would support and coordinate the early decision-making process, e.g. release planning.The view of an improved release planning process is inline with the results in Berntsson Svensson et al. [3].The subjects agreed that the roadmap view provides a clear and understandable representation of the market as a basis for decisions.Particular the information about targets, what the estimated cost for reaching the targets is, competitors and your own product's level of quality, and the identified saturation breakpoint is of major help in release planning.The importance of a rich understanding of the market as input for release planning and early decision-making is inline with the results in Berntsson Svensson et al. [3].
During the interviews, a general challenge of applying QUPER was identified, to translate the theory of QUPER into practice.According to the subjects, it was difficult to apply a new model and its terminology.The reason for this was that the subjects were used to estimate quality requirements based on gut feeling.Meaning, no process or common terminology was previously used.Working with quality requirements in a more structured way was completely new to the subjects.

B. QUPER -Benefit view
Table III displays the evaluation results of the benefit view.In general, the subjects agreed that the benefit view is rather complete and could not identify any need for additional information that may be needed.The subjects believed that the benefit view captures the market perspective very well, particular the information about the competitors' products.

About QUPER's benefit view General comments
(1) The saturation breakpoint is the most important one (2) brings in the market perspective and the competitors very well (3) easy to communicate within and with management Challenges (1) Hard to gather competitor information and to define the differentiation breakpoint (2) difficult to identify the saturation breakpoint Estimating breakpoints A combination of results from the latest samples, knowledge and experience Preferred expertise knowledge needed (1) A combination of marketing and system knowledge should form great domain knowledge (2) one without the right domain knowledge will only add more uncertain estimations Estimations uncertainty magnitude (1) From reasonable to quite high uncertainty (2) estimates based on small samples that do not reflect the "real world" (3) no uncertainty for the utility breakpoint due to regulations from third party organizations The subjects liked the concepts of the breakpoints.The utility breakpoint is mostly defined by third party organizations, which all actors on the electronic payment processing market must adhere to.The differentiation breakpoint was seen as valuable once the case company had gathered information (current level of quality) from its competitors.Similar to the results in Berntsson Svensson et al. [3], the main benefit of the breakpoints was the saturation breakpoint, where the quality level changes from competitive to excessive quality.Meaning higher quality has no practical impact on the benefit in the particular usage context considered.However, the saturation breakpoint was experienced as the most challenged breakpoint to estimate, it was difficult to know at which level the quality was seen as excessive.This challenge was closely related to the lack of input from the case company's main customers.
Another identified challenge of the benefit view was to gather information about the competitors; with only three competitors in a relatively dense market it is not desirable to give away information that competitors can use.Hence, it was a challenge to identify the value for the differentiation breakpoint.The main input when estimating the differentiation breakpoint was a competitor analysis.On a quarterly basis, the case company's Marketing & Sales department calls its competitors, as private individuals that are interested in their products, to collect sales information, e.g.offer-prices.However, to collect information about the current level of quality with regards to the selected quality requirements, a short questionnaire was developed in collaboration with the Marketing and Sales department.The questionnaire is based on questions related to the two quality requirements that are used to evaluate QUPER.The second author called the competitors to conduct the competitor analysis (using the developed questionnaire) in a discrete manner.Since sales-employees gave the answers to technical questions, it was decided to conduct the competitor analysis a second time to confirm the answers.Another reason for conducting the analysis a second time is because of unrealistic answers of certain quality levels according to the case company's experts.The second competitor analysis was conducted by using the same questionnaire with as different sales-employee that answered the questions the first time; names of the sales-employee were noted.
The challenges related to the identification of the values for the differentiation and saturation breakpoints are the same challenges reported in Berntsson Svensson et al. [3].Despite experiencing the same problems, the reasons behind the challenges differ between the two studies.While the case company in this study had problems with gathering competitor information to be able to calibrate the differentiation breakpoint, in Berntsson Svensson et al., the company's problems was related to when to stop calibrating those breakpoints [3].
When estimating the three breakpoints, a combination of expertise in the area, the latest tests, and years of domain knowledge and experience of the system were used.In general for all three breakpoints, the estimated values should all be acceptable from a customer point of view.For the utility breakpoint, the required quality level defined by the third party organizations was the main source of information when estimating the value.However, even if the estimated value marks the shift from useless to useful quality, it may not be acceptable from a customer point of view.For the differentiation breakpoint, the subjects believe when all of the competitors level of quality is gathered, it is useful to calculate an average quality level to know how much more it is required to improve in order to differentiate yourself from your competitor.When estimating the saturation breakpoint, the value depends on what the customer expects or desires.To gather this information, collecting and organizing customer feedback is important.Then, it is possible to define the saturation breakpoint with as high accuracy as possible.
When asked about the importance of having the right domain knowledge when estimating the breakpoints, the subjects believe it is of major importance.If the experts do not have the right domain knowledge, the estimates will be much more uncertain.It is preferable to have a combined knowledge of the market and of technology and development.
Similar to Berntsson Svensson et al. [3], the subjects believed the estimates to have an increase in accuracy compared to the previous estimates, which did not include a structured way of measuring quality requirements.However, the estimates ranged from reasonable to quite some doubts.This is mainly due to the fact that the case company had never before actively upgraded their products remotely.The estimates were based on small tests, which did not represent the actual numbers in the "real world".According to the subjects, the uncertain estimates are related to lack of gathering and using information about their own product's level of quality, i.e. the existing information was not very reliable.

About QUPER's cost view General comments
The cost view was considered very useful to understand the cost of different levels of quality Challenges (1) Difficult to estimate an investment per QR (2) much time would be needed to integrate the cost view in the current process Estimating cost barriers (1) The first cost barrier is easier to estimate than the second one (2) The first cost barrier is viewed as a reasonable improvement, while the second cost barrier is to be differentiated from the competitors Preferred expertise knowledge needed (1) A combination of market experts and experts on technology and development (2) more domain knowledge was needed than first expected Estimations uncertainty magnitude (1) Greater uncertainty compared to the estimates in the benefit view (2) intervals to map the uncertainties of the estimates would be useful In general, the subjects agreed that the cost view is very useful in their situation.To map the cost of potential levels of quality improvements at an early phase, before the development starts, is very important to the case company.The importance of understanding what different levels of quality cost in an early phase is inline with the results from Berntsson Svensson et al. [4].The subjects believe improvements (major ones where cost barriers are identified) should be between the differentiation and saturation breakpoints.However, the subjects experienced difficulties with determining an improvement where a significant investment (time and money) is required in order to get closer to the saturation breakpoint.One reason why the subjects experienced difficulties with estimating the cost barriers was lack of in-depth knowledge and experience in estimating the cost of implementing quality requirements.
Another identified challenge of the cost view was to estimate an investment for each quality requirement, define available resources, and then stay within the estimated cost without exceeding it.
When estimating the cost barriers, the subjects tried to relate as much as possible to the products where they have a direct influence since not all mobile products are always accessible for the subjects.The cost barriers were defined based on market expectations and the subjects' knowledge.However, when the market expectations were not fully understood, the estimated improvements may not be realistic, which may not be in the best interest of the customers.The subjects explained, there are values that must be adhered to in their market segments, but also important quality aspects for the customers that should have a dedicated part of the total improvement budget.In this case, the subjects considered how important such quality requirements are, and how much (percentage of the total budget) each of those quality requirements should have.
The subjects considered the first cost barrier as the most reachable improvement with minor effort.While the second cost barrier was seen as a differentiator from the competitors and provides the customers with the best available quality.The subjects agreed that the first cost barrier is easier to estimate than the second cost barrier.The reason is that the first cost barrier is closer to the current quality level than the second one.This view of the differences of estimating the two cost barriers is inline with the results in Berntsson Svensson et al. [4].
Similar to the practical guidelines of QUPER's cost view in [4], the subjects believe that the first cost barrier is related to software optimization, while the second cost barrier may be related to new hardware or new architecture.However, if the product requires new hardware or a new architecture, this product would normally (in the case company) turn into a new product.There are major differences between the levels of quality of the case company's large set of requirements.Meaning, some requirements already have a high level of quality, and therefore, the first cost barrier may relate to new hardware or a completely new architecture.
Similar to the required domain knowledge for the benefit view, the subjects believe a combination of market experts and experts on technology and development are of major importance.The subjects realized that more domain knowledge than expected is necessary to be able to make better, and more accurate estimates.
According to the subjects, the estimates for the cost view have a greater uncertainty than the estimates of the breakpoints in the benefit view.Instead of estimating the cost as an absolute value, the subjects would prefer to use an interval (of the estimated cost) to better capture the uncertainties.Two main reasons were discovered for the uncertain cost estimates.First, the subjects did not have any reliable data to support the estimates of the cost barriers, or any historical data that could have been used for determining more certain estimates.Second, lack of in-depth knowledge and experience of cost estimation of quality requirements led to more uncertain estimates of the cost barriers.

VI. CONCLUSIONS
In conclusion, this paper presents the result of an empirical study using action research that examines the improvement of the release planning process with regards to quality requirements by introducing QUPER at a case company in the electronic payment processing market.The QUPER model was implemented and evaluated by applying it in real projects, using real requirements, by industry professionals.Data is collected from four subjects.
The overall result indicates that QUPER's benefit and cost views are relevant in the early decision-making process, e.g. release planning.Similar to [3], the concepts of breakpoints, competitor analysis, and identification of own products quality level provides a greater understanding of the needed quality level for a particular release.In addition, the concept of cost barriers provides a great understanding of what different levels of quality cost.The combination of the benefit and cost views provides a clear picture of the current market situation, and what quality level to aim for at what cost.Furthermore, the concepts behind QUPER improve the communication to management, which is inline with [3].
Two main challenges are identified, (1) difficulties to identify and specify the values for the differentiation and saturation breakpoints, and (2) determining an improvement where a significant investment is required, i.e. identifying and specifying the values for the cost barriers.
The evaluation indicates, not only that QUPER is feasible and relevant to the selected domain, but also to other domains than the mobile handset domain [3].
The QUPER evaluation described in this paper, was aimed to observe how QUPER could be applied in a different domain than the mobile handset domain, and the use of both the benefit and cost views together.The next phase that will be undertaken in the validation and evolution of QUPER can be described as a two-step process.
First, the observed findings in this evaluation provides an important feedback regarding the usability and usefulness of QUPER, but nonetheless enabling further improvements on the model's concepts and practical guidelines of the benefit and cost views, e.g.incorporating support for the uncertainty of cost estimates.Moreover, practical guidelines for the roadmap view will be developed, and to find heuristics to efficiently support and incorporate dependencies into the QUPER model.All the improvements and developed guidelines will be used as input to the next step, large-scale (use QUPER for more than a few quality requirements) piloting of QUPER in industry.The present plan is for QUPER to be tailored to, and subsequently piloted in two organizations.Data, both qualitative and quantitative will be collected during the large-scale piloting.In addition, the natural evolution of QUPER will continue, and tool support for the model will be developed.

TABLE III .
BENEFIT VIEW EVALUATION RESULTS Table IV illustrates the evaluation results of the cost view in this study.

TABLE IV .
COST VIEW EVALUATION RESULTS