# Experimental Design and Computer Performance Analysis

## Abstract

While queueing theory provides substantial insight into the solution of many computer performance prediction problems, there remain situations for which queueing theory is inadequate, e.g. predicting the effect of a change in the memory size of a multiprogramming system; or determining the impact of a change to an operating system tuning parameter. Experimental design provides an effective methodology for addressing these kinds of problems. This tutorial introduces eqperimental design as a tool for the computer performance analyst. The e:q>erimental design methodology covers both the actual deágn and planning of an experiment, as weÚ as the analysis of its results. Añer some backgroiud discussion of the methodology and its terminology we proceed with a discussion of design issues that confront a computer performance experimenter. The different types of variability that are present in an experiment are discussed, together with techniques for their control or eliioination. Next we review the analysis of experiments, stressing the importance of assumption validation. For example, the analysis associated with e:q)erimental design (e.g., the analysis of variance, or A N O V A) is straightforward and available in many statistical packages. However, the validity of this analysis depends on several key assumptions that are rarely examined. Indeed, it is the nature of computer performance data that it frequentiy violates these assumptions! We describe and illustrate graphical techniques for assessing compliance with these assumptions, as well as approaches for ensuring e:iiperimental data is appropriate for this analysis.