As is the case for
hardware, software reliability
engineering involves much more than
analyzing test results, estimating
remaining faults, and modeling future
failure probabilities.
Although in most
organizations software test is no
longer an afterthought, management is
almost always surprised by the cost
and schedule requirements of the test
program, and it is often downgraded in
favor of design activities. Often
adding a new feature will seem more
beneficial than performing a complete
test on existing features. A good
software reliability engineering
program, introduced early in the
development cycle, will mitigate these
problems by:
-
Preparing program
management in advance for the
testing effort and allowing them to
plan both schedule and budget to
cover the required testing.
-
Offering
management a quantitative assessment
of the dependence of reliability
metrics (software/system
availability; software/system
outages per day etc) on the effort
(time and cost) allotted to testing.
-
Providing the
most efficient test plan targeted to
bringing the product to market in
the shortest time subject to the
reliability requirements imposed by
the customer or market expectations
-
Continuous
quantitative assessment of
software/system reliability and the
effort/cost required to improve
these by a specified amount.
SoHaR software
reliability engineers are experienced
in all the stages and tasks required
in a comprehensive software
reliability program. We can support or
lead tasks such as:
Reliability Allocation
Defining and Analyzing Operational
Profiles
Test Preparation and Plan
Software Reliability Models
Reliability Allocation
Reliability
allocation is the task of defining the
necessary reliability of a software
item. The item may be part of an
integrated hardware/software system,
may be a relatively independent
software application, or, more and
more rarely, a standalone software
program. In either of these cases our
goal is to bring system reliability
within either a strict constraint
required by a customer or an
internally perceived readiness level,
or optimize reliability within
schedule and cost constraints.
SoHaR will assist
your organization in the following
tasks:
-
Derive software
reliability requirements from
overall system reliability
requirements
-
When possible,
depending on lifecycle stage and
historical data, estimate schedule
and cost dependence on software
reliability goals
-
Optimize
reliability/schedule/cost based on
your constraints and your customer's
requirements
Ideally,
reliability allocation is performed
early in the lifecycle and may be
modified and refined as both software
and other system components are
developed. At these early stages,
SoHaR can assist in the above tasks
with limited design and requirements
inputs. As the system develops
software reliability allocation
becomes more accurate. The dependence
of reliability allocation on cost and
schedule also solidifies when the
software goes into testing. Although
it is ideal to begin these tasks early
on and follow during system evolution,
often organizations do not implement a
software reliability program until
very late in the software development
cycle. The delay may be to the time of
test preparation and plan, or even
later when testing is yielding results
that need to be interpreted to verify
or ascertain reliability.
Defining and Analyzing
Operational Profiles
The reliability of
software, much more so than the
reliability of hardware, is strongly
tied to the operational usage of an
application. A software fault may lead
to system failure only if that fault
is encountered during operational
usage. If a fault is not accessed in a
specific operational mode, it will not
cause failures at all. It will cause
failure more often if it is located in
code that is part of an often used
"operation" (An operation is defined
as a major logical task, usually
repeated multiple times within an hour
of application usage). Therefore in
software reliability engineering we
focus on the operational profile of
the software which weighs the
occurrence probabilities of each
operation. We will prioritize our
testing according to this profile.
SoHaR will work
with your system and software
engineers to complete the following
tasks required to generate a useable
operational profile:
-
Determine the
operational modes (high traffic, low
traffic, high maintenance, remote
use, local use etc)
-
Determine
operation initiators (components
that initiate the operations in the
system)
-
Determine and
group "Operations" so that the list
includes only operations that are
significantly different from each
other (and therefore may present
different faults)
-
Determine
occurrence rates for the different
operations
-
Construct the
operational profile based on the
individual operation probabilities
of occurrence.
Test Preparation and
Plan
Test preparation is
a crucial step in the implementation
of an effective software reliability
program. A test plan that is based on
the operational profile on the one
hand, and subject to the reliability
allocation constraints on the other,
will be effective at bringing the
program to its reliability goals in
the least amount of time and cost.
Software
Reliability Engineering is concerned
not only with feature and regression
test, but also with load test and
performance test. All these should be
planned based on the activities
outlined above.
The reliability program will inform
and often determine the following test
preparation activities:
-
Assessing the
number of new test cases required
for the current release
-
New test case
allocation among the systems (if
multi-system)
-
New test case
allocation for each system among its
new operations
-
Specifying new
test cases
-
Adding the new
test cases to the test cases from
previous releases
Software Reliability
Models
Software
reliability engineering is often
identified with reliability models, in
particular reliability growth models.
These, when applied correctly, are
successful at providing guidance to
management decisions such as:
The application of
reliability models to software testing
results allows us to infer the rate at
which failures are encountered
(depending on usage profile) and more
importantly the changes in this rate
(reliability growth). The ability to
make these inferences relies
critically on the quality of test
results. It is essential that testing
be performed in such a way that
failure incidence is accurately
reported.
SoHaR's software reliability engineers
will work with developers, testers and
program management to apply an
appropriate model to your failure
data. In order for the model
prediction to be useful we must ensure
that the assumptions and structure of
the model coincide with the underlying
coding and testing process. It is not
sufficient to find a mathematical
function that best fits the data. In
order to infer future failure behavior
it is crucial that the underlying
assumptions of the model be understood
in terms of program management and
progress towards release. This
requires experience working with the
software reliability models as well as
an understanding of latent issues in
the development and testing process
that may influence the test data.
For
more information about SoHaR's
Software Reliability and Safety
program please contact us at
becky@sohar.com
