Quality Management for the Web

From WebScience

Jump to: navigation, search

Fact Box
Module Foundations and Principles I
Mario Winter
Credits 3
Term Term 1, Term 2
Course is not required
Current course page Winter 2017
Active Yes

The course Quality Management for the Web (QM4Web) is dedicated to aspects of quality management for the development, deployment, operation, evolution, assessment, and governance of web-based systems. Covered aspects are

  • the history of quality management (QM),
  • the basic principles of QM,
  • the role of quality models,
  • the components of quality management systems (QMS), and
  • some important techniques and tools for QM.

The Big Picture

Contemporary web-based systems not only support the presentation of and interaction with rich media pages, but also the execution of business-critical processes. Additionally, web based-systems, and especially rich internet applications (RIA), bear many aspects of multi- resp. hypermedia systems. To achieve, sustain, and optimize the quality of such systems, quality management and the adoption of an appropriate quality management system should be a strategic decision of each organization developing, maintaining, or operating such a system.

Since web-based systems are primarily software systems, all aspects of software quality management also are applicable to web-based systems. Additionally, some aspects peculiar to the quality management for web-based systems have to be accounted for.

According to the ISO-Standard 9000[1] quality is the degree to which a set of inherent characteristics fulfills requirements. The term "quality" can be used with adjectives such as poor, good, or excellent. "Inherent", as opposed to "assigned", means existing in something, especially as a permanent characteristic. A peculiarity of the very general definition of quality cited above is the total absence of an object which bears the characteristics fulfilling (or not) the requirements. Indeed the object could be a product or service, a single project, or a whole organization and its resources.

Quality management comprise all coordinated activities to direct and control an organization with regard to quality. Thus quality management generally includes establishment of the quality policy and quality objectives, quality planning, quality control, quality assurance and quality improvement.

Quality management systems are a means to assist organizations in enhancing customer satisfaction. Customers require products with characteristics that satisfy their needs and expectations. These needs and expectations are expressed in product specifications and collectively referred to as customer requirements. Customer requirements may be specified contractually by the customer or may be determined by the organization itself. In either case, the customer ultimately determines the acceptability of the product. The ISO-Standard 9001[2] states general requirements on quality management systems.The application of ISO 9001 to the development of Software is described in the ISO-Standard 90003[3].

Because customer needs and expectations are changing, and because of competitive pressures and technical advances, organizations are driven to continually improve their products, and processes. The quality management system approach encourages organizations to

  • analyze customer requirements,
  • define the processes that contribute to the achievement of a product which is acceptable to the customer, and
  • keep these processes under control.

A quality management system can provide the framework for continual improvement to increase the probability of enhancing customer satisfaction and the satisfaction of other interested parties. It provides confidence to the organization and its customers that it is able to provide products that consistently fulfill requirements.

According to the ISO/IEEE Standard 12207 Software Life Cycle Processes[4], quality management plays a key role both within the whole product life cycle processes (covering acquisition/supply, development, operation, maintenance, and phaseout), and the organizational processes (covering management, infrastructure, improvement, and training). Thus QM enacts many supporting processes like documentation, configuration management, quality assurance, verification and validation, joint reviews and audits. Thus it partially overlaps with IT-governance, compliance, and optimization processes, as covered in the module on Web Analytics.

Intended Learning Outcomes

The students are able to identify the primary needs for and the implementation of quality management for the development of web-based systems. Therefore they:

  • know existing approaches, concepts and methods for (software) quality and quality management and are able to discuss their similarities and differences,
  • know established techniques of quality planning, quality assurance, quality control, and quality optimization on organizational, project, and development-phase scale and are able to judge them critically in terms of business, risk, and other kinds of objectives, and
  • are able to derive appropriate requirements for quality management systems from high-level (e.g. business) objectives.

Structure of the Course


The course starts with a brief history on (software) quality and quality management, as sketched out e.g. by Schulmeyer in Chpt. 2 of his Handbook of Software Quality Assurance[5]. Quality management (QM) is a responsibility at all levels of management and involves all members of an organization. For this reason, in the 1980s, total quality management (TQM) as a business management strategy aimed at embedding awareness of quality in all organizational processes found very great success. Early works on QM were led by experts like Kaoru Ishikawa, Joseph M. Juran, Yoji Akao, W. Edwards Deming, Genichi Taguchi, Shigeo Shingo, Philip Crosby, and Watts Humphrey. The work of these contributors each contain important quality messages, which have been applied to product development. Typically, quality applications to product development have been supplied by product (i.e., software) specialists. However, the concepts available that address quality production must now be used for the production of quality products containing software. The definition of "quality", more narrow than "fitness for use", is that supplied by Philip Crosby, which is "conformance to requirements". The logical extension of that definition to software quality is "to conform to software requirements and to provide useful services" ("the fitness for use of the software product").

After studying the historical roots of QM, the students shall gain basic contemporary terminology on quality management as defined in the ISO 9000 standard[1]. Besides consulting this and other standards on quality management themselves, their descriptions in wikipedia[6] and the survey found in chapter 3 of Schulmeyers Handbook of Software Quality Assurance[5] contains sufficient introductions for this course.

According to the ISO 9000 series of standards, the basic steps for developing and implementing a quality management system are:

  • determination of needs and expectations of customers and other involved parties;
  • definition of the organization's quality policy and quality objectives;
  • determination of processes and responsibilities for quality assessment;
  • identification and choice of production resources necessary to attain the quality objectives;
  • determination and application of methods to measure the effectiveness and efficiency of each process within the production system;
  • prevention of nonconformities and deletion of the related causes;
  • definition and application of a process for continuous improvement of the quality management system.

The second pillar for this course is an introduction to the challenges the web presents to software engineering and application development. A good starting point to the recent evolution of the Web is given in the article on Web Science written by Hendler et.al.[7].

In this course, when talking of web applications, we bear in mind any application delivered over the Web that combines characteristics of both multi- resp. hypermedia and software applications, as described in chapter 1 of Mendes and Mosley's book on Web Engineering[8]. The development of the latter is done largely by IT- or software professionals bearing a sound knowledge of development and project management methods, techniques, and tools. In contrast, the development of web applications encompasses a much wider variety of developers, such as amateurs with no programming skills, graphics designers, writers, database experts, and IT professionals, to name but a few. This is possible as Web pages can be created by anyone without the necessity for programming knowledge. Web applications by default use communications technology and have multi-platform accessibility. In addition, since they employ a hypermedia paradigm, they are non-sequential by nature, using hyperlinks to interrelate Web pages and other documents. Therefore, navigation and pluralistic design become important aspects to take into account. Finally, the multitude of technologies available for developing Web applications means that developers can build a full spectrum of applications, from a static simple Web application using HTML to a fully fledged distributed e-commerce application.

Additionally, Web applications are aimed at wide-ranging groups of users. Such groups may be known ahead of time (e.g. applications available within the boundaries of the intranet). However, it is more often the case that Web applications are devised for an unknown group of users, making the development of aesthetically pleasing applications more challenging. In contrast, conventional software applications are generally developed for a known user group (e.g. department, organization) making the explicit identification of target users an easier task.

Regarding the quality challenges, read the chapter on web quality of Olsina, Covella and Rossi (chapter 4 of Mendes and Mosley[8]). This chapter sketches Web applications as encompassing technical computing, information architecture, contents authoring, navigation, presentation and aesthetic, multiplicity of user audiences, legal and ethical issues, network performance and security, and heterogeneous operational environments. Chapter 4.1 introduces some quality aspects for web applications.

Basic Principles and Tools

To achieve, sustain, and optimize the quality of web-based systems, one needs to know

  • how to built quality during the initial development of new systems,
  • how to measure the quality of new or existing systems before and during their operation, and
  • how to steer quality optimization to balance the effort used for and the value earned by it.

Each trip - be it in a car on the road or in an organization paving on its way to high quality products and processes - has to determine where one is and where one wants to be. The equivalents of map and compass for the road are statistics and measurements for the quality. Therefore we start our course with an introduction of a basic statistic and measurement toolbox.


According to the wikipedia[6], Statistics is considered to be a mathematical science or branch pertaining to the collection, analysis, interpretation or explanation, and presentation of data. Statisticians improve the quality of data with the design of experiments and survey sampling. Statistics also provides tools for prediction and forecasting using data and statistical models.

For an introduction to statistics, we recommend the site[9] Online Statistics: An Interactive Multimedia Course of Study as an introductory-level statistics book. The material is presented both as a standard textbook (a PDF-version of which is available as Online_Statistics_Education.pdf[10]) and as a multimedia presentation featuring interactive demonstrations and simulations, case studies, and an analysis lab. Read the introductory chapters and consult the more advanced ones when unfamiliar with notions, methods, or techniques in the sequel of our course. Alternatively, chapter 12 of Mendes and Mosley[8] contains an introduction to the basic notions and techniques of statistics used in this course.

Measurement and Metrics

You cannot control what you cannot measure - this often cited sentence, attributed to Lord Kelvin, states the importance for measurement in the science and engineering domain. Unfortunately, in the software domain, we most often can't rely on traditional measurement techniques for physical values like size or time, but have to measure intangible properties of software and services like complexity, usability, or correctness. According to chapter 16 of Schulmeyers Software Quality Assurance Handbook[5], the purpose of software quality metrics is to assess throughout the development cycle whether the software quality requirements are being met. The use of metrics reduces subjectivity in the assessment of software quality by providing a quantitative basis for making decisions about software quality. The use of metrics, however, does not eliminate the need for human judgment in software evaluations. The use of software quality metrics within an organization or project is expected to have a beneficial effect by making software quality more visible.

Metrics allow quantitative statements regarding the quality of a product and its development and quality assurance processes. They must therefore be built on solid measurement theoretical foundations. Metrics form the basis for transparent and traceable quality management planning and control. Based on different measurement objectives, we can define metrics for product quality, process quality, and an organizations overall quality. In this chapter, we focus on product quality, where in later chapters we are going to widen our view w.r.t. to process and project quality.

Chapter 16 of Schulmeyers Handbook of Software Quality Assurance[5] contains an introduction to software quality assurance metrics. An introduction to the goal-directed definition of metrics in the software engineering domain can be found in the report of Park et al.[11]). Some interesting insights into the usage of metrics in the software engineering domain can be found in the paper of Kaner and Bond[12]). Moreover, Rice gives some hints to possible consequences of measurement[13]).

Product Quality

In Sect. 4.2 of Mendes' and Mosley's book on Web Engineering[8], the different perspectives of quality for software are discussed. In particular, Sect. 4.2.1 reviews the state of the art of quality according to the ISO standards for software quality. Also the importance of distinguishing between (internal and external) quality and quality in use is addressed (see Sect. 4.2.2), and how different requirements, from diverse users' standpoints, should be considered (see Sect. 4.2.3).

Regarding software quality, the most important quality model is specified in the ISO 9126 series of standards[14]. The ISO/IEC 9126 part 1 defines a general purpose quality model, quality characteristics, and sub-characteristics. It divides approaches to product quality into quality in use, external quality, and internal quality. The definitions of these quality aspects are as follows:

  • Quality in use is the extent to which a product used by specified users meets their needs to achieve specified goals with effectiveness, productivity and satisfaction in a specified context of use;
  • External quality is the extent to which a product satisfies stated and implied needs when used under specified conditions;
  • Internal quality is the totality of attributes of a product that determines its ability to satisfy stated and implied needs when used under specified conditions.

ISO/IEC 9126 parts 2-4 give examples of metrics to define and to measure quality attributes contributing to these characteristics.

The ISO/IEC 14598 series[15] gives an overview of software product evaluation processes and provides guidance and requirements for evaluation. Parts 2 and 6 relate to corporate or department level evaluation management and support, while parts 3, 4 and 5 give requirements and guidance for evaluation at the project level.

Note that the ISO 9126 and ISO 14598 series of standards is currently being replaced by the ISO/IEC 250xx series[16]. SQUaRE will comprise 5 divisions of standards, namely (note that the "x" in the indicated numbers stand for one of the 10 digits)

  • ISO/IEC 2500x - Quality Management Division. The standards that form this division define all common models, terms and definitions referred further by all other standards from SQuaRE series. Referring paths (guidance through SQuaRE documents) and high level practical suggestions in applying proper standards to specific application cases offer help to all types of users. The division provides also requirements and guidance for a supporting function which is responsible for the management of software product requirements specification and evaluation.
  • ISO/IEC 2501x - Quality Model Division. The standard that forms this division presents a detailed quality model including characteristics for internal software quality, external software quality and software quality in use. Furthermore, the internal and external software quality characteristics are decomposed into subcharacteristics. Practical guidance on the use of the quality model is also provided. This division replaces ISO 9126-1.
  • ISO/IEC 2502x - Quality Measurement Division. The standards that form this division include a software product quality measurement reference model, mathematical definitions of quality measures. and practical guidance for their application. Presented measures apply to internal software quality, external software quality and quality in use. Quality measure elements forming foundations for the latter measures are defined and presented. This division replaces ISO 9126-2-4.
  • ISO/IEC 2503x - Quality Requirements Division. The standard that forms this division helps specifying quality requirements. These quality requirements can be used in the process of quality requirements elicitation for a software product to be developed or as input for an evaluation process. The requirements definition process is mapped to technical processes defined in ISO/IEC 15288- Information Technology -Life Cycle Management - System Life Cycle Processes.
  • ISO/IEC 2504x - Quality Evaluation Division. The standards that form this division provide requirements, recommendations and guidelines for software product evaluation, whether performed by evaluators, acquirers or developers. The support for documenting a measure as an Evaluation Module is also presented. This division will replace ISO 14598-x.
  • ISO/IEC 25050 - 25099 are reserved for SQuaRE extension international standards and technical reports.

As noted in the introduction of this course description, the descriptions in wikipedia[6] and the survey in chapter 3 of Schulmeyers Handbook of Software Quality Assurance[5] contain sufficient information on ISO 9126 and ISO 250xx (SQuaRE) for this course.

Quality in use

Quality in use measures the quality perceived by real users using a web application in their real contexts. Because quality in use is of utmost importance to Web quality, it is at the core of the design-related modules of the web-science curriculum. Thus in this module we don't need to dive into this matter. For the interested student, chapter 5 of Mendes' and Mosley's book on web engineering[8] covers principles and evaluation methods for web usability.

External and internal Quality

Sect. 4.3 of Mendes' and Mosley's book[8]presents the Web Quality Evaluation Method (WebQEM) as a model-centered evaluation method for the external quality of web sites. Using WebQEM to assess Web applications helps to meet quality requirements in new Web development projects and to evaluate requirements in operational phases. It also helps discover absent attributes or poorly implemented requirements, such as interface-related designs, and implementation drawbacks or problems with navigation, accessibility, search mechanisms, content, reliability and performance, among others.

Regarding the internal quality of html-based web sites, the international Standard Software Engineering - Recommended Practice for the Internet - Web Site Engineering, Web Site Management, and Web Site Life Cycle[17] describes recommended practices for World Wide Web page engineering for Intranet and Extranet environments, based on World Wide Web Consortium (W3C) and related industry guidelines. This recommended practice does not address stylistic considerations or human-factors considerations in web page design beyond limitations that reflect good engineering practice.

Since web sites quality models are important to rank presences in the web, to identify best practices, or pinpoint poor implementations, another quality model especially for web sites was developed by Rio and Abreu [18]. In contrast to the WebQEM-Model cited above, the work of Rio and Abreu presents a preliminary step of a website quality evaluation approach based on fully automatic collection of quality metrics, grouped by ISO9126 quality characteristics.

Information Quality

Consulting the Wikipedia, Information quality (IQ) is a term to describe the quality of the content of information systems. It is often pragmatically defined as The fitness for use of the information provided. Also in ch. 4.2 of Mendes' and Mosley's book[8], information quality is mapped to the content aspects of web-based systems. To this end, the four categories information accuracy, information suitability, accessibility, and legal compliance define information quality requirements of Web applications. According to Mendes and Mosley, also sub-concepts of information structure and organisation should be addressed. Note that many of these sub-characteristics, such as global understandability, learnability, and even internationalisation, are also be related to the usability and the quality in use characteristics.

As an example of defining and measuring the information quality of web sites, Capiello[19] et al. state that modern Web 2.0 applications are characterized by high user involvement: users receive support for creating content and annotations as well as "composing" applications using content and functions from third parties. This latter phenomenon is known as Web mashups and is gaining popularity even with users who have few programming skills, raising a set of peculiar information quality issues. Assessing a mashup's quality, especially the information it provides, requires understanding how the mashup has been developed, how its components look alike, and how quality propagates from basic components to the final mashup application.

A most specialized product quality standard is the EN IEC 62503:2008, Multimedia quality - Method of assessment of synchronization of audio and video[20]. It specifies a subjective (or perceptible) and statistical method of assessment of overall difference of delays between a real world and a reproduced scene and sound. It addresses this quality aspect of typical multimedia content such as bust shots of news casters because of easiness in defining synchronization of audio and video. Since a range of perceptibly allowable miss-synchronization and sensitivity of human audience for lead and delayed audio against accompanied video depends on human sensation and the conditions for assessment, a clearly defined method of assessment of such characteristics should be standardized.

Quality Management Methods, Techniques, and Tools

As stressed in Mendes' and Mosley's book on Web Engineering[8], the Web plays a central role in diverse application domains such as business, education, government, industry, and entertainment. The Web's growing importance heightens concerns about Web applications' development and evaluation methods, and requires the systematic use of engineering models, methods, and tools. In this part of the course we look at

  • organizational measures,
  • preventive (constructive) measures, and
  • analytical (detective) measures.

Organizational Measures

The main organizational measure regarding quality management is the incorporation of appropriate processes guarded by an appropriate quality management system. This section covers the latter, while the section on process quality focuses the former.

The ISO 9000 family distinguishes between requirements for quality management systems and requirements for products. Requirements for quality management systems are specified in ISO 9001[2]. Requirements for quality management systems are generic and applicable to organizations in any industry or economic sector regardless of the offered product category. ISO 9001 itself does not establish requirements for products (c.f. section 'Product Quality'). ISO 90003[3] describes the application of ISO 9001 to computer software.

As stressed in ch. 14 of [5], to successfully apply quality management to the activities performed by an information technology (IT) organization, it is necessary to be familiar with the key IT processes. IT services in organizations are provided by an internal department or an outsourced organization known as the IT organization. The IT organization consists of resp. provides the IT infrastructure. An IT infrastructure describes all the components used in the delivery of the IT services to users, including the computing and telecommunications hardware, software, accommodation, people, documentation, and meta-data. These components and their use must be managed - hence the term IT infrastructure management. Collectively, IT services and the management of the IT infrastructure is referred to as IT service management (ITSM). ITSM is the principles and practices of designing, delivering, and maintaining IT services to an agreed-upon level of quality, all in support of a customer activity. To this end, the ISO standard series 20000[21], which is based on the Information Technology Infrastructure Library (ITIL), defines a framework to direct, monitor and control the service management activities of a IT service provider.

Preventive (Constructive) Measures

To prevent quality issues from creeping into web sites and applications, their development should be done according to the state of the art. To this end, quality managements main job is to give an organizational framework for processes, methods, techniques, and tools, which is tailorable to the quality requirements for the product to be developed or the service to be delivered. Since concrete methods, techniques, and tools for the development of web sites and applications count among the operational resp. technical roles and responsibilities, they are not focused by this course. The interested student is referred to chapters 9 and 10 of Mendes' and Mosley's book on Web Engineering[8] or to the module on web architectures.

Analytical (Detective) Measures

One needs sound evaluation methods for obtaining reliable information about product quality and quality in use. There are different categories of methods (e.g. inspection, testing, inquiry, simulation) and specific types of evaluation methods and techniques (e.g. heuristic evaluation technique, the concept model-centered evaluation method). In this section we look at some techniques for static analysis and dynamic testing of web applications.

Static analysis comprises all techniques focusing on an development artifact as is. For example, program code would be analyzed by static analysis without executing it. Chapter 7 of Schulmeyers Handbook of Software Quality Assurance[5] describes inspections as the main measure to assure the quality of all development artifacts perceivable by humans, such as textual descriptions, diagrams, program code, or design mockups. Some techniques and tools for the analysis of web sites are already sketched in the section on external and internal quality and on information quality.

Dynamic testing needs executable development artifacts, such as running programs or loadable web sites. During the execution of the software with predefined data, deviations of actual outcomes from expected ones may be detected. While static analysis focuses directly on defects in the development artifacts (i.e. bugs in some code) and may prove that there are no errors, dynamic testing can only detect the presence of failures. Chapter 7 of Mendes' and Mosley's book on Web Engineering[8] surveys some methods for web application testing.

Process and Organization Quality

Up to now the course focused mainly on product quality. In this last part, this focus is widened to the quality of single projects (developing a product or service) and whole organizations (conducting such projects). We already took a look at the ISO 9001[2], which states general requirements on quality management systems. The application of ISO 9001 to the development of software is described in the ISO-Standard 90003[3]. In the following, the two contexts of process improvement and process capability determination are characterized by two widely used standards, namely CMMI (the american view) and SPICE (the european answer). Chapter 8 of Mendes' and Mosley's book on Web Engineering[8] gives an overview on software process improvement (SPI) for small organizations.

Capability Maturity Model Integration (CMMI)

CMMI is a process assessment and improvement approach that provides organizations with the essential elements of effective processes that ultimately improve their performance. CMMI can be used to guide process improvement across a project, a division, or an entire organization. It helps integrate traditionally separate organizational functions, set process improvement goals and priorities, provide guidance for quality processes, and provide a point of reference for appraising current processes.

CMMI offers two representations for process assessment and improvement: continuous and staged.

  • The continuous representation allows to focus on the specific processes that are considered important for an organization's immediate business objectives, or on those to which the organization assigns a high degree of risks. To each of these specific processes a capability level will be assigned, resulting in a differentiated achievement profile.[6]
  • The staged representation characterizes whole project or organization by one of five so-called maturity levels (1 - Initial, 2 - Managed, 3 - Defined, 4 - Quantitatively Managed, 5 - Optimizing). It provides a standard sequence of improvements, and is more convenient for comparing the maturity of different projects and organizations.

The maturity of an organization (or division) is rated by conducting a so-called appraisal. To this end, CMMI defines the Standard CMMI Appraisal Method for Process Improvement (SCAMPI). Depending on the type of the appraisal, the organization can be awarded a maturity level (1-5), or it will receive a capability level achievement profile over the specific processes.

An introduction to CMMI is given in chapter 11 of Schulmeyers Handbook of Software Quality Assurance[5]. The original documentation can be found at the SEI CMMI Version 1.3 Information Center[22]).

ISO 15504 (SPICE)

According to wikipedia[6], ISO/IEC 15504 Information technology - Process assessment[23], also known as SPICE (Software Process Improvement and Capability Determination), is a set of technical standards documents for the computer software development process and related business management functions. It is comparable to the staged representation of the CMMI.

Didactic Concept, Schedule and Assignments

The course concept comprises basic readings, online workshops, online group work and an introductory and final on site presence. After a first introductory lecture on site, the subject is treated in three online workshops, that are supplemented by a session on site. Online workshops are held on three evenings with a duration of three hours each.

Introductory lecture on site

The introductory meeting deals with organizational course details and a workshop to address basic definitions and concepts of quality management for the web.

1st online workshop

This workshop consists of two phases: The first part is held as a seminar. The subject of the first workshop, ch. 3.2, "Basic Principles and Tools", and ch. 3.3. "Product Quality of Web Applications", is presented in a highly interactive matter, open questions are discussed with the lecturer and among the students.

For the second phase students form groups and choose a case study to address issues of chapter 3.3 (product quality).

Group presentations and a discussion conclude this session.

2nd online workshop

The second workshop is based on the basic readings cited in chapter 3.4 above (Quality Management Methods, Techniques, and Tools). Starting with a given case study (describing an organization and its development process), the participants form groups, fill the role of stakeholders, form and negotiate high-level objectives for the QMS of the case study. In the second phase they apply some quality management methodology to the exemplary organization of the case study and form a basic concept to carry out short- and mid-term quality management activities.

Findings are discussed in a concluding plenary session.

3rd online workshop

Main topics of the third workshop are the concepts studied in chapter 3.5 (process and organization quality). The participants continue with their work from the 2nd workshop, select an appropriate process quality framework, and plan optimization goals and a preliminary assessment for the case study. They discuss possible achievements according to high-level objectives.

Findings are discussed in a concluding plenary session.

Wrap-up session on site

This on site workshop is dedicated to clarify open questions of the students concerning all course issues and to summarize workshop content.

The wrap-up session is also used to complete the written examination (45 minutes).


The module grade arises to 70% from the written module examination grade (45 minutes) and to 30% from the score for the group work elaborated during the course.

The written module examination consists of 3 knowledge questions (~30 min, restricted response or multiple choice) and 1 complex achievement question (~15 min, scenario based restricted response, covering a range of core course topics, which the students have to debate).


  1. 1.0 1.1 ISO 9000:2005 Quality management systems - Fundamentals and vocabulary. ISO. 2005. 
  2. 2.0 2.1 2.2 ISO 9001:2008 Quality management systems - Requirements. ISO. 2008. 
  3. 3.0 3.1 3.2 ISO 90003:2004 Software engineering — Guidelines for the application of ISO 9001:2000 to computer software. ISO. 2004. 
  4. ISO 12207:1995 Standard for Information Technology - Software life cycle processes. ISO. 1995. 
  5. 5.0 5.1 5.2 5.3 5.4 5.5 5.6 5.7 Schulmeyer, G. Gordon (ed.) (2008). Handbook of Software Quality Assurance (4. Ed.). ARTECH HOUSE. pp. 1--62. 
  6. 6.0 6.1 6.2 6.3 6.4 "WIKIPEDIA - The Free Encyclopedia". Wikimedia Foundation, Inc.. 2011-11-21. http://en.wikipedia.org. Retrieved 2011-01-21. 
  7. Hendler, James and Shadbolt, Nigel and Hall, Wendy and Berners-Lee, Tim and Weitzner, Daniel (July 2008). "Web science: an interdisciplinary approach to understanding the web". Commun. ACM (ACM) 51 (7): 60--69. http://doi.acm.org/10.1145/1364782.1364798. 
  8. 8.00 8.01 8.02 8.03 8.04 8.05 8.06 8.07 8.08 8.09 8.10 Emilia Mendes and Nile Mosley (2006). Web Engineering. Springer. doi:10.1007/3-540-28218-1. 
  9. David M. Lane et Al. (2006-01-06). "Online Statistics: An Interactive Multimedia Course of Study". Rice University et Al.. http://onlinestatbook.com/. Retrieved 2011-01-21. 
  10. David M. Lane et Al. (2006-01-06). "PDF of Online Statistics: An Interactive Multimedia Course of Study". Rice University et Al.. http://onlinestatbook.com/Online_Statistics_Education.pdf. Retrieved 2011-01-21. 
  11. Robert E. Park and Wolfhart B. Goethert and William A. Florac (1996). "Goal-Driven Software Measurement - A Guidebook". 10TH INTERNATIONAL SOFTWARE METRICS SYMPOSIUM. http://www.sei.cmu.edu/reports/96hb002.pdf. Retrieved 2011-02-19. 
  12. Cem Kaner and Walter P. Bond (2004). "Software Engineering Metrics: What Do They Measure and How Do We Know?". 10TH INTERNATIONAL SOFTWARE METRICS SYMPOSIUM. http://www.kaner.com/pdfs/metrics2004.pdf. Retrieved 2011-02-19. 
  13. Riley Rice (2008). "Unintended Consequences of Measurement: Causes and Cures". SEPG Conference. http://www.sei.cmu.edu/library/assets/Rice08.pdf. Retrieved 2011-02-19. 
  14. ISO 9126-1:2001 Information technology - Software product quality - Part 1: Quality model. ISO. 2001. 
  15. ISO/IEC 14598-1:1999 Information technology - Software product evaluation - Part 1: General overview. ISO. 1999. 
  16. ISO 25000 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) - Guide to SQuaRE. ISO. 2005. 
  17. ISO / IEEE (2006). ISO/IEC 23026 (IEEE Std 2001-2002): Software Engineering - Recommended Practice for the Internet - Web Site Engineering, Web Site Management, and Web Site Life Cycle. ISO / IEEE. 
  18. Americo Rio and Fernando Brito e Abreu, (2010). "Websites Quality: Does It Depend on the Application Domain?,". Proc. International Conference on the Quality of Information and Communications Technology (IEEE Computer Society): 493-498. 
  19. Cinzia Cappiello and Florian Daniel and Maristella Matera and Cesare Pautasso (2010). "Information Quality in Mashups". IEEE Internet Computing (IEEE Computer Society): 14-22. doi:http://doi.ieeecomputersociety.org/10.1109/MIC.2010.60. 
  20. Multimedia quality - Method of assessment of synchronization of audio and video. ISO. 2008. 
  21. ISO/IEC 20000-1, Information technology - Service management - Part 1: Service management system requirements. ISO. 2010. 
  22. "SEI CMMI Version 1.3 Information Center". SEI. 2009. http://www.sei.cmu.edu/cmmi/tools/cmmiv1-3/. Retrieved 2011-02-19. 
  23. ISO/IEC 15504-1, Information technology - Process assessment - Part 1: Concepts and vocabulary. ISO. 2004. 

Past Course Pages