About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Publication
CMG 1992
Conference paper
Experimental design and computer performance analysis
Abstract
While queueing theory provides substantial insight into the solution of many computer performance prediction problems, there remain situations for which queueing theory is inadequate, e.g., predicting the effect of a change in the memory size of a multiprogramming system; or determining the impact of a change to an operating system tuning parameter. Experimental design provides an effective methodology for addressing these kinds of problems. This tutorial introduces experimental design as a tool for the computer performance analyst. The experimental design methodology covers both the actual design and planning of an experiment, as well as the analysis of its results. After some background discussion of the methodology and its terminology we proceed with a discussion of design issues that confront a computer performance experimenter. The different types of variability that are present in an experiment are discussed, together with techniques for their control or elimination. Next we review the analysis of experiments, stressing the importance of assumption validation. For example, the analysis associated with experimental design (e.g., the analysis of variance, or ANOVA) is straightforward and available in many statistical packages. However, the validity of this analysis depends on several key assumptions that are rarely examined. Indeed, it is the nature of computer performance data that it frequently violates these assumptions! We describe and illustrate graphical techniques for assessing compliance with these assumptions, as well as approaches for ensuring experimental data is appropriate for this analysis.