In Response to ‘What was the Original Purpose of MTBF Predictions?’
In the section on predictions you mention Dr. Box’s oft quoted
statement that “..all models are wrong, but some are useful.” In the
same book Dr. Box also wrote, “Remember that all models are wrong; the
practical question is how wrong do they have to be to not be useful.” [see these and other quote by Dr. George Box here]
Reliability predictions are intended to be used as risk and resource
management tools. For example, a prediction can be used to:
- Compare alternative designs.
- Used to guide improvement by showing the highest contributors to failure.
- Evaluate the impact of proposed changes.
- Evaluate the need for environmental controls.
- Evaluate the significance of reported failures.
None of these require that the model provide an accurate prediction of
field reliability. The absolute values aren’t important for any of the
above tasks, the relative values are. This is true whether you express
the result as a hazard rate/MTBF or as a reliability. Handbook methods
provide a common basis for calculating these relative values; a
standard as it were. The model is wrong, but if used properly it can
Think about the use of RPN’s in certain FMEA. The absolute value of
the RPN is meaningless, the relative value is what’s important. For
sure, an RPN of 600 is high, unless every other RPN is greater than
600. Similarly, an RPN of 100 isn’t very large, unless every other RPN
is less than 100. The RPN is wrong as a model of risk, but it can be
I once worked at an industrial facility where the engineers would dump
a load of process data into a spreadsheet. Then they would fit a
polynomial trend line to the raw data. They would increase the order
of the polynomial until R^2 = 1 or they reached the maximum order
supported by the spreadsheet software. The engineers and management
used these “models” to support all sorts of decision making. They were
often frustrated because they seemed to be dealing with the same
problems over and over. The problem wasn’t with the method, it was
with the organization’s misunderstanding, and subsequent misuse, of
regression and model building. In this case, the model was so wrong it
wasn’t just useless, it was often a detriment.
Reliability predictions often get press. In my experience, this is
mostly the result of misunderstanding of their purpose and misuse of
the results. I haven’t used every handbook method out there, but each
that I have used state somewhere that the prediction is not intended to
represent actual field reliability. For example, MIL-HDBK-217 states,
“…a reliability prediction should never be assumed to represent the expected field reliability.”
I think the term “prediction” misleads
the consumer into believing the end result is somehow an accurate
representation of fielded reliability. When this ends up not being the
case, rather than reflecting internally, we prefer to conclude the
model must be flawed.
All that said, I would be one of the first to admit the handbooks could
and should be updated and improved. We should strive to make the
models less wrong, but we should also strive to use them properly.
Using them as estimators of field reliability is wrong whether the
results are expressed as MTBF or reliability.