Measurement and evaluation methodology

Measurement and evaluation methodology

This note proposes a methodology for measurement and evaluation of the process of software distribution. In deriving the adequate methodology, the following standards and methods have been studied:

• ISO 9126 standard, which deals with software product quality,
• ISO 14598 standard, which is concerned with the process of evaluation,
• Goal/Question/Metric (GQM) Method, developed at NASA, mostly used for the improvement of software development.

Links:

• King: Living up to standards - ISO standards 9126 and 14598 (for software product quality) http://www.dcs.shef.ac.uk/~katerina/EACL03-eval/eacl-doc/King.pdf
• Basili, Caldiera, Rombach: The goal/question/metric (GQM) approach http://www.cs.umd.edu/projects/SoftEng/ESEG/papers/gqm.pdf
• The goal/question/metric method http://irb.cs.uni-magdeburg.de/sw-eng/us/java/GQM/

The methodology consists of the following steps:

Identifying goals

In order to be effective, a measurement should be focused on specific and explicitly stated goals. In specifying a goal of the measurement, the following questions should be answered:

• What? the object under measurement (e.g. a product, a process)
• Why? the purpose (e.g. improving, comparing)
• What aspect? the quality focus of the object that the measurement focuses on
• Who? the person or team from whose point of view the object is measured

For instance, our goal can be to: _improve the quality of service in the process of software distribution from the viewpoint of the end user_.

It is recommended to choose only the most important goals.

Specifying a quality model

A quality model consists of a set of quality characteristics, each of which is decomposed into a set of quality sub-characteristics. The characteristics and sub-characteristics depend on the purpose of the evaluation. The structure is hierarchical, and, theoretically of unlimited depth.

For the goal defined above, the characteristics can be:

• _quality of version_, and
• _ease of use_.

For the quality of version, the following sub-characteristics can be defined:

• _freshness_,and
• _consistency_.

Similarly, the sub-characteristics for the ease of use are defined.

In this phase a set of questions can also be derived in order to characterize the object of measurement with respect to a selected quality issue and to determine its quality from the selected viewpoint. It should be defined what it means to satisfy the goals as completely as possible in a quantifiable way. The question types can vary from rather general (e.g. what is the consistency requirement?) to more specific (e.g. how many files are missing for a certain version? or how much time is needed to download the whole version?).

Defining metrics

Each terminal node of the quality model hierarchy will have at least one metric associated with it.

If the set of questions has also been specified, these questions can be further refined into metrics. A set of attributes, which have to be measured in order to answer the questions, is generated.

Example of a metric: _average time a user needs to download a complete version_.

It is recommended to identify a significant and minimal set of metrics that are clearly related to the goals of the measurement activity.

Establishing a measurement and evaluation plan

Rating levels for metrics

Establishing rating levels for metrics involves determining the correspondence between the score produced on some scale and the degree of satisfaction of the requirements.

Criteria for evaluation

Establishing criteria for evaluation involves defining a procedure for summarizing the results of the evaluation of the different characteristics, using for example weighting functions of different kinds.

Methods and schedule

Methods and schedule for measurement and evaluation should be described. It should be defined who is responsible for collecting metrics data values.

Simulation of data interpretation

A simulation of data interpretation should also be done before the actual measurement starts. The simulation contains expected values of metrics, graphs and charts.

Data collection - measurement, rating, evaluation

As already mentioned, measurement gives a score on the scale, rating determines the correlation between the raw score and the rating levels, and evaluation is a summary of the set of rated levels. The individual ratings are put together in order to get an overall picture, which reflects the relative importance of different characteristics in the light of the particular quality requirements.

Interpretation

The results of the measurements are discussed and conclusions are made.

{metadata}

Topics Wp5

{metadata}