Proceedings of the Seventeenth Symposium on Explosives and Pyrotechnics, Essington, PA, April 1999.

ISO 14304 Annex B
All-Fire/No-Fire Test and Analysis Methods

Barry T. Neyer James Gageby
EG&G Optoelectronics JVG Associates
Miamisburg, OH 45343-0529 Palm Desert, CA 92211-1399

Contact Address
Barry T. Neyer
PerkinElmer Optoelectronics
1100 Vanguard Blvd
Miamisburg, OH 45342
(937) 865-5586
(937) 865-5170 (Fax)
Barry.Neyer@PerkinElmer.com

Abstract

A new international standard for explosive systems and devices has been developed. It combines key attributes of multi-national standards, specifications, regulations, manuals, publications and journals into a uniform set of criteria that can be used to safely and reliably apply explosive systems onto space vehicles. This paper presents some of the details of this standard, with the major emphasis on the All-Fire and No-Fire test methods.

Technical Papers of Dr. Barry T. Neyer

Introduction

A new international standard has been developed. It combines key attributes of multi-national standards, specifications, regulations, manuals, publications and journals into a uniform set of criteria that can be used to safely and reliably apply explosive systems onto space vehicles.

The new standard is known as ISO 14304, Criteria for Explosive Systems and Devices Used on Space Vehicles. It provides improved understandings of all areas of the field of explosive systems, also known as pyrotechnics. The criteria outlined in this new international standard are a composite of those verified by previous use in space vehicle applications.

Described are essential design characteristics, suggested manufacturing controls, and methods for certifying performance, acceptance, qualification and useful life. It is intended to be universal sets of tools for explosive system manufacturers and users during all phases of development, certification and application.

Criteria are defined here as standards, rules or tests by which items can be judged. They are measures of value that have proven to be effective in numerous space vehicle applications. These same criteria may also be used in applications unrelated to space vehicles. A goal of this effort was to produce a quality document that can be used by all.

Explosive systems include components and assemblies that provide stimuli for initiation and propagation of explosive trains used to activate explosive devices. Explosive trains and devices are components or assemblies containing or operated by explosive materials. The latter are, by design, "one-shot" components that cannot be tested completely before use. Performance confidence of "one-shot" components can only be obtained by destructive tests of like samples from common production lots.

Key attributes of this new international standard:

The group responsible for this new international standard is the International Organization for Standardization (ISO), Technical Committee 20 (Aircraft and Space Vehicles), Subcommittee 14 (Space Systems Operations), Working Group 1 (Design Engineering), otherwise known as ISO/TC20/SC14/WG1.

Note that the acronym ISO was used in lieu IOS to preclude language confusion (English – IOS, French – OIN). It is a derivative of the Greek isos, meaning equal.

The main body of ISO presently has more than 166 technical committees working on standardization of a multitude of items and subjects. The organization is based in Geneva and was initiated in 1947 via a United Nations resolution establishing universal compatibility of commonly used items. An overview of the ISO is given at website http://www.iso.ch/

Compliance to international standards created by ISO/TC20/SC14 is voluntary. Their intent is to enhance scientific cooperation and promote trade. They do not require individual countries to change or discard their existing specifications or standards. Rather, individual countries are encouraged to exchange information that can be included in ISO standards. ISO/TC20/SC14 goals are given at website www.aiaa.org/information/professional/standards/tc20sc14.html.

Within ISO/TC20/SC14, voting member nations include Canada, China, France, Germany, Italy, Israel, Japan, Russia, the United Kingdom and the United States. Non-voting member nations include Argentina, Brazil, India, Poland, Slovakia and Spain.

The American National Standards Institute (ANSI) is the primary U.S. member body overseeing ISO/TC20/SC14 efforts. They are assisted by the American Institute of Aeronautics and Astronautics (AIAA).

The ANSI/AIAA intent is to have ISO/TC20/SC14 ISO standards relating to Space Vehicles dominate the 21st century, bypassing National Commercial and Government Standards of the 1970’s through the 1990’s.

ISO 14304 has nearly completed the ISO preparatory stage and will progress to the committee stage shortly. It is anticipated that final release will occur in mid-2001. Copies of the full ISO 14304 proposed standard can be downloaded from http://home.att.net/~j.v.gageby/iso14304.html or http://www.aiaa.org/tc/ecs/.

The rest of this paper presents the details of Annex B, which describes the All-Fire /No-fire Test and Analysis Methods.

ANNEX B, All-Fire/No-Fire Test and Analysis Methods

1. Scope. This Appendix offers test and analysis methods for estimating first element functional all-fire and non-functional no-fire input energy ratings. These are applicable to electrical, optical and mechanical first elements. Reliability and safety issues necessitate use of methodologies that can assure consistency in derivation of these estimates. Accepted test and analysis methods for estimating these parameters include the Bruceton method, and other advanced methods such as Langlie and Neyer D-Optimal, all of which are described here. The Bruceton method was developed in the 1950’s and has proven its value. Advanced methods described here have been evaluated through experience in use and may provide improved knowledge of estimates of key parameters while reducing costs to obtain them.

2. Applicable Documents. Product specifications of first elements tested. Detailed descriptions of data acquisition measurement methods and instrumentation components used in tests and, the following technical reports and publications.

Technical Reports

NAVORD Report 2101 - Statistical Methods Appropriate for Evaluation of Fuze Explosive Train Safety and Reliability, U.S. Naval Ordnance Laboratory, White Oak, MD, (1953).

Report U-1792 - A Reliability Test Method for One-Shot Items, Langlie, H. J. (1965), Technical, Aeronautical Division of Ford Motor Company.

Report MLM-3736 - An Analysis of Sensitivity Tests, EG&G Mound Applied Technologies, (1992).

Publications

Journal of the American Statistical Association - A Method for Obtaining and Analyzing Sensitivity Data, Dixon, J. D., and Mood, A. M. (1948), Vol. 43, 109-126

Technometrics - A D-Optimality Based Sensitivity Test, February 1994, Volume 36, Number 1, pages 61-70, Neyer, B. T. (1994)
(See
http://www.neyersoftware.com/Papers/D-Optimal/D-OptimalAbstract.htm)

3. Test and Analysis Methods. The following describes test and analysis techniques most commonly used for estimating and evaluating first element all-fire and no-fire input energy ratings. Although test and analysis are independent functions, each unique test method is historically associated with a unique analysis technique. These methods and techniques are commonly referred to as sensitivity tests and analysis. First elements with outputs that are independent of input magnitude require the specialized test and analysis methods described here.

3.1 Objectives. Objectives of sensitivity test and analysis methodologies used should be assurance that estimates derived are as accurate and precise as possible. The parameter to be estimated is the mean stimulus level at which some fraction of the samples of a specific first element design will always ignite, in the case of an all-fire test, or not ignite, in the case of a no-fire test. There are no methodologies capable of exact determinations of this parameter. There are no non-destructive methods available to obtain the data needed. The tests described here are usually considered as destructive in nature and therefore, the test articles should not be re-used in the end item. The analysis portion of the method computes estimated all-fire or no-fire rating. These estimated ratings are computed at a specific reliability and confidence and are applicable to only the specific first element design tested. These estimated ratings are computed at a specific reliability and confidence and are applicable to only the specific first element design tested.

3.2 Limitations. All methods used are small sample based. Therefore error in the estimates may occur. Care must be exercised when choosing test stimulus levels during the tests. If empirical data on the specific design is not available to assist selection of test levels before start of the test then additional samples should be allocated to perform pre-test evaluations. All of the methods used here assume the distribution of the threshold stimulus levels is normal. It is simple to generalize this assumption and require that some function, such as a logarithm of the threshold levels are normally distributed.

3.3 Reliability and Confidence Levels. Reliability and confidence level values conventionally used in sensitivity tests and analysis are 0.999 and 95%, respectively. This is literally interpreted to mean that 95% of the time no more than 1 in 1000 first elements will fail to function at the estimated all-fire rating. Therefore, the user should assure that the ignition stimulus delivered to the first element in the end item application not be limited to the all-fire rating, as noted in WD/ISO14304, paragraph 4.4.1. Users of these computed values should be made aware that adding margin to estimated all-fire and no-fire values is standard practice. As noted in paragraph 4.4.1 of WD/ISO14304 an ignition system should use input stimulus 1.25 times greater than the estimated all-fire threshold of the interfacing first element. For example, when using an EED having an all-fire estimate of 3.25 amperes, the ignition system should be designed to have a minimum output of 4.06 amperes. Explosive system reliability assessments should therefore use the minimum stimulus values that the ignition system delivers to the first element to assess realistic system level reliability.

3.4 Test Conduct. Tests should be performed in an ambient temperature environment unless conditions anticipated in the end item application dictate a need to do otherwise. Heat sinks used should simulate thermal properties of the end item application, to the extent practical. Once started, the test should continue uninterrupted until completed. Analysis can be performed at any time during or after completion of the test portion of the task. For EED and LID first elements, all-fire tests should use an ignition stimulus pulse duration equivalent to that used in the end item application, but should be no greater than 30 milliseconds. No-fire tests are not required for mechanical first elements.

3.4.1 Bruceton Test. At least forty-five (45) first elements should be allocated for each test. The first sample is pulsed at a defined stimulus level and duration. If that sample fires the next test sample is pulsed at a stimulus level that is reduced by a defined increment, or step, lower than the first. If the first sample had not fired the next sample would have been pulsed with a stimulus increased by the same defined increment. The test continues in this process until at least forty (40) samples are expended. Each sample is pulsed only once during these tests.

The total number of incremental steps of fire and no fire data points should be greater than three (3), but not more than six (6). Tests where the numbers of increment steps are outside this range should be considered invalid. To prevent this, care should be exercised in selecting the magnitude of the initial stimulus used and in the amount of the defined increment, or step between succeeding pulses before starting the test. Experience with similar first element designs and/or pre-test firings can be used to estimate these values. Five (5) samples of the allocated group can be used in initial searches for reasonable starting points. If determined to be valid data, these five may be combined with the total sample tested.

3.4.2 Langlie Test. The main goal in developing the Langlie method was to overcome the dependence of the efficiency of the Bruceton test on the choice of the step size. Analysis and experience had shown that the defined increment, or step size of the Bruceton test had to be correct to within a factor of two for reliable results.

To perform a Langlie test, the experimenter must specify lower and upper stress limits. The first test is conducted at a level midway between these limits. The general rule for obtaining the (n+1)st stress level, having completed n trials, is to work backward in the test sequence, starting at the nth trial, until a previous trial (call it the pth trial) is found such that there are as many successes as failures in the pth through nth trials. The (n+1)st stress level is then obtained by averaging the nth stress level with the pth stress level. If there exists no previous stress level satisfying the requirement stated above, then the (n+1)st stress level is obtained by averaging the nth stress level with the lower or upper stress limits of the test interval according to whether the nth result was a failure or success.

The Langlie test has been shown to be less susceptible to variations in efficiency caused by inaccurate test design. The efficiency of the test is somewhat dependent on the choice of lower and upper stress limits. Most users specify limits that are extremely wide to avoid the situation where the limits are too close together, or do not contain the region of interest. The method is most efficient if the upper and lower limits are 4 standard deviations from the mean.

One problem with the test method, however, is that the method concentrates the test levels too close to the mean, resulting in inefficient determination of the standard deviation of the population.

3.4.3 Neyer D-Optimal Test. This test was designed to extract the maximum amount of statistical information from the test sample. Unlike the other test methods, this method requires detailed computer calculations to determine the test levels. The Neyer D-Optimal test uses the results of all the previous tests to compute the next test level.

There are three parts to this test. The first part is designed to "close-in" on the region of interest, to within a few standard deviations of the mean, as quickly as possible. The second part of the test is designed to determine unique estimates of the parameters efficiently. The third part continuously refines the estimates once unique estimates have been established.

This test requires the user to specify three parameters, i.e., lower and upper limits, and an estimate of the standard deviation. The first two parameters are used only for the first few tests (usually two (2) tests) to obtain at least one fire and one fail to fire. The estimate of the standard deviation is used only until overlap of the data occurs. Thus, the efficiency of the test is essentially independent of the parameters used in the test design.

3.5 Comparison of Test Methods. There is no unambiguous method of ranking the test methods. A good test method should yield estimates of the parameters of the population that are accurate and precise. All of the test methods yield accurate parameters on average. Thus, the best way to characterize the tests is by their precision. The purpose of most sensitivity tests is to determine an all-fire or a no-fire level. These levels are usually defined as that level at which at least 0.999 of the first elements fire (all-fire) or at which at least 0.999 of the first elements fail to fire (no-fire). With the assumption of normality, the all-fire and no-fire levels can be converted into a simple function of the mean, m, and the standard deviation, s, of the population. The 0.999 all-fire level is m +3.09 s, and the 0.001 no-fire level is m -3.09 s. Thus, precise determination of the all-fire or no-fire level requires precise determination of the mean, and especially the standard deviation.

There are several ways to compare the ability of the various test methods to precisely determine estimates of the standard deviation. One method would be to determine the variation of the estimates of the standard deviation as a function of sample size and test method. This variation depends not only on the test method, but also on the selection of the parameters of the population before beginning the test.

The efficiency of the Bruceton test is strongly dependent on the choice of step size. The efficiency of the Langlie test is somewhat dependent on the spacing between the upper and lower test levels. The Neyer D-Optimal test is essentially independent of the choice of parameters.

Figure 1 shows the variation of the estimates of the standard deviation as a function of the sample size for the three test methods under the assumption that the standard deviation is well known before start of testing. If the first elements are well characterized from previous tests, the standard deviation may be known to approximately a factor of two. The figure assumes that the parameters of the test were optimized for the population.

The figure also shows that variation of the estimate of the standard deviation has a strong dependence on the test method chosen. For example, a 20 shot Bruceton test yields a relative variance of 66%, while the Langlie test yields a variance of 28% and the Neyer D-Optimal yields a variance of 20%. The publication by Neyer (1994), noted in section 2, provides more in-depth details of the analysis used to produce this graph. This graph is only valid for the case where the experimenters know the correct test parameters. The variation of the Bruceton test in particular could be much larger if the estimate of the step size is wrong by more than a factor of two.

Figure 1: Comparison of the Variation in Estimates of the Standard Deviation

Another method of judging the utility of the various test methods is to determine the extreme values of the estimates of a parameter. The greatest concern in conducting and analyzing sensitivity tests is the tendency of the method to produce estimates of the parameters that are far removed from the true parameters.

Figure 2 shows the 5% and 95% values of the standard deviation as a function of sample size for the three test methods. Also shown in the figure are the corresponding curves for the F Test whose significance is described in the text below.

Figure 2: 5% and 95%Estimates of the Relative Standard Deviation

The figure illustrates several important points. First, it is extremely difficult to establish the value of the standard deviation to a degree of precision desired with a limited sample size. For example, in a Neyer D-Optimal test with a sample size of 150, 5% of the estimates of the standard deviation will be more that 20% lower than the true value, and 5% of the estimates will be more than 20% higher than the true values. For the same sample size for the Langlie test, 5% would be 50% lower, and 5% would be 45% higher. For the Bruceton test the corresponding results are 100% lower, and 30% higher.

For the typical sample size used in threshold tests, e.g., 20 - 50, it is impossible to estimate the standard deviation, and thus the all-fire and no-fire levels, with great certainty. Thus, in addition to the estimation of the parameters of the population, it is also imperative that the appropriate analysis be performed to estimate the confidence of the estimate of the parameters. Confidence estimation is discussed in the next section.

The F Test curves shown in Figure 2 indicate how much less information is available for sensitivity tests compared to standard statistical tests. The F Test is used in standard statistical testing to calculate the fraction of estimates of the standard deviation that are higher or lower than a given value. If it were possible to measure the exact threshold of individual first elements, then the estimates of the standard deviation would be governed by the F Test. Inspection of the curves shows that a sensitivity test requires a sample size many times greater than the sample size of a classical statistical test to achieve the same range of values for the standard deviation.

The final point illustrated by the figures is that the ability to determine reasonable estimates of the parameters is extremely dependent on the test method chosen to conduct the test. Both Figure 1 and Figure 2 illustrate the importance of choosing an efficient test method when conducting sensitivity tests.

3.6 Analysis Methods. More methods of analyzing the results of sensitivity tests have been proposed than have test methods. The method chosen to analyze the data of the test is at least as important as the test method. While many analysis methods can be used to analyze the results of any test method, other analysis methods are designed to analyze only one test design. All of the analysis methods provide relatively unbiased estimates of the parameters of the population, i.e. the estimate of the mean, M, is close to the true mean, m, and the estimate of the standard deviation, S, is close to the true standard deviation, s. However, the ability of the various methods to compute reliable confidence levels varies greatly.

The variance function method assumes that the variances of M and S can be estimated by simple functions of the sample size and the standard deviation. These functions are generally dependent on the initial conditions, sample size, and the test design, i.e., Bruceton, Langlie, Neyer D-Optimal. Some groups use the T test to compute confidence intervals for the mean and Chi Squared or F tests to compute confidence intervals for the standard deviation. However, these generalized statistical methods should not be used. The assumptions that are used to construct the general statistical tests are violated in the case of sensitivity tests. Figure 2 shows the curves for both the F test as well as curves for the various sensitivity tests. The figures clearly show that the F test can not be used to analyze sensitivity tests.

The simulation method uses test results to determine the variance of the parameters after the test has been completed. This method can provide reliable estimates of the variances as long as the simulation is carried out with parameterization relevant to the population. If simulation is used to estimate the variation of the parameters, the parameters for the simulation must span a wide area around the estimates of the test data. The number of simulation runs must be sufficient (over 1000) to ensure that the results are statistically valid.

The asymptotic method is used by some computer programs, such as ASENT discussed in section 3.6.2, and in the calculations of the variance in the Bruceton method.

Simulation discussed in some of the referenced papers shows that the variance of both M and S scales approximately with s2. Because s2 is not independently known. All of the previously mentioned techniques base their estimates on the maximum likelihood estimate of s, which is S. If the successes and failures do not overlap, S = 0 and these methods fail to produce estimates for confidence regions for both M and S. The likelihood ratio method discussed in section 3.6.3 can produce reliable confidence interval estimates in all cases, including this degenerate case.

Almost all of the analysis methods used to date produce false confidence. That is, what is reported as a 95% confidence level is in actuality more like a 60% confidence level. Thus, there should be agreement between the first element user and the test facility as to which analysis method is used, and the method should be one that has been shown to produce realistic confidence levels.

Records of test data, computations and results should be retained as permanent parts the first element documentation package.

3.6.1 Bruceton Analysis. The Bruceton test was developed before the advent of electronic computers. It was designed so that simple paper and pencil calculations could be used to determine the mean, the standard deviation, as well as estimates of their variance. Today, more advanced analysis methods are available to analyze this data. The traditional Bruceton analysis method can still be used, but only when the number of test levels are between 4 and 6, and the sample size is not less than 40. In all cases it is preferable to use advanced analysis methods, such as the ones described in the following two sections. When the Bruceton Analysis method is applicable, the Asymptotic method of the next section yields the same results.

3.6.2 Asymptotic Analysis. Advanced methods include computer software known as ASENT which is in use at many test facilities. Although this analysis software is usually associated with the Langlie test method, it can analyze the results of tests conducted according to any test method. The analysis method computes the maximum likelihood estimates of the parameters. It computes estimates of the variance of the parameters by computing the curvature of the likelihood function. This analysis method gives the correct results asymptotically. It will not analyze the results of a test where the successes and failures do not overlap. It gives reliable results if the sample size is greater than 200.

3.6.3 Likelihood Ratio. Another advanced method is the likelihood ratio test. This method is used in software known as MuSig, as described in Report MLM-3736 of section 2. This software is in use at many laboratories around the world. Although this is the analysis method usually associated with the Neyer D-Optimal method, it can analyze the results of tests conducted using any test method. The analysis method computes the maximum likelihood estimates of the parameters. It computes estimates of the variance of the parameters by using the likelihood ratio test. This analysis method gives the correct results asymptotically. It will analyze the results of any test, even if the successes and failures do not overlap. It gives reliable results if the sample size is greater than 20.

3.7 Comparison of Analysis Methods. The two most widely used general analysis methods can be compared in a number of ways. The most meaningful way to compare the methods is to determine what fraction of the time the true parameters are outside of the specified confidence region. A properly computed 95% confidence region, for example, should contain the true parameters approximately 95% of the time.

Figure 3 shows the fraction of parameters outside a given confidence region for both the asymptotic analysis used by ASENT and the likelihood ratio analysis used by MuSig. This figure is for a sample size of 30 for the Bruceton, Langlie, and Neyer D-Optimal tests. The solid line in the figure is what a perfect analysis method would produce. For the group of lines using squares to denote plot points the upper line is the Langlie method, the next lower is the Neyer D-Optimal method and the next is the Bruceton. For the group using circles, the upper is Langlie, the next Neyer D-Optimal and the lower Bruceton.

Figure 3: Comparison of Confidence Likelihood Ratio versus ASENT

The figure clearly shows that both of the analysis methods produce false confidence. For example, for a nominal 95% confidence region, the likelihood ratio test has the parameters outside of the confidence region approximately 8% of the time. While this is more than the 5% expected for a true 95% confidence region it is close to the requested confidence. Note that it would be prudent for the user of this information to specify a slightly more restrictive confidence (such as 97%) to achieve the required 95% confidence region.

The asymptotic 95% confidence region however, has the parameters outside of the confidence region approximately 20% of the time. To achieve a true 95% confidence region using this analysis method would require the computation of a confidence region greater than 99%.

First elements that are considered qualified when analyzed according to one analysis method could be unqualified when analyzed according to a more exact analysis method such as the likelihood ratio test. Thus, the end item user should either specify the analysis method or be consulted by the test facility as to options available. If a true 95% confidence region is required, then only analysis methods capable of producing a realistic confidence region should be used.