QDOE (quantitative design of experiments): Some lessons from field experience and a connection to Bayesian methods

The purpose of this talk is to comment on the purpose of experiments and the nature of the risks associated with them. The connection of this topic to Bayesian methods is that the interpretation of the experiment is conditional on one's prior belief about how the data arose. Factors subject to prior belief include the details of the items under test, the details of test conditions, and the details of sensor arrangement and performance. An incomplete range of alternatives in these areas of the prior will lead to misinterpretation. This talk will sketch out present methods of QDOE developed by the Navy and illustrate common pitfalls as experienced in major tests. The final box score on seven such tests, designed and conducted during the period 1981 - 1996, could be given as 3-1-3. That is, using QDOE we experienced three wins (i.e., foresaw a problem and avoided it), one loss (could not recover from a problem), and three ties (ran into a problem but recovered from it). Without QDOE, despite the fact that our goals would have been less ambitious, the probable outcome even for those lower goals would have been something like 2-4-1 - many more losses.