This paper is intended to show the methodology for extending Probability of Detection (POD) modeling for continuously-valued (â vs a) signal responses to allow for the addition of multiple factors beyond the simple flaw size model, along with higher-level interactions. The statistical methodology for correctly transforming these more complex linear models into probability of detection curves is provided, and the approach is illustrated with a motivating real-world POD study
KEYWORDS: Classification systems, Automatic target recognition, Information science, Defense and security, Sensors, Mathematics, Data processing, System identification, Mathematical modeling, Information fusion
This work examines the scenario of ATR classification in multi-label settings by using the framework of a classification sequence. Classification tasks are often composed of a sequence of identification tasks that together, generate an overall classification. For instance, objects may be sorted and classified as one particular target type and then those targets are further identified. Rather than passing all objects through each classifier, a sequence of classifiers may be used to identify objects without the need to process data through each classifier. Such sequences exist for two-label outcomes (such as target and non-target) and have been called: Believe the Negative, Believe the Positive, and Believe the Extremes. In each of these sequences, the first classification system is able to identify objects such that only a portion of objects must be passed to the second system for identification. However, to extend these sequences to k-labels, a new definition of the ordering on the labels must be generated in order to incorporate all k-labels into the classification sequence. In this work, we develop the mathematical structures that exist for a k-label classification sequence, provides formula for both the optimal performance and operational cost of these sequences, and examines the performance of such sequences under a variety of operating conditions. Conceptually, we will begin and demonstrate these results with a 3-label ATR system. In conclusion, this work will demonstrate the utility of using a sequence to fuse information in a multi-label classification task.
This paper will investigate the fusion of various detection and classification systems. The architecture of combining these systems is the main interest of this work. We assume the detection and classification systems are known and they are legacy systems such that we know their receiver operating characteristic (ROC) functions, or their approximate ROC functions. Given an objective function we seek the optimal architecture that maximizes the objective function. Combining detection systems sequentially has been around for decades, especially in the bio-medical field where tests are preformed sequentially such that the outcome of one test will determine which test will be performed next. In military applications, we often use multiple detection systems in parallel and combine the outputs into a “fusion” center to determine the final answer. We conjecture that there might be a parallel and series mixture that would yield better performance. Part of determining this mixture is determining which systems go "where" in the mix. We investigate this architecture.
This paper presents a method to quantify detection system families (DSFs) based upon the Precision-Recall (PR) curve and variations of the PR curve. The PR curve is related to the Receiver Operating Characteristic (ROC) curve. The ROC curve of a detection system family shows the trade-off between the probabilities of a true positive classification versus the probability of a false positive classification. The conditional probabilities are conditioned on the true outcomes. The PR curve is similar in the sense that the conditional probabilities are conditioned on the outcomes of the detection systems that "say" they are true outcomes. We present the function that produces the PR curve, called the PR function. We produce the (nonlinear) transformation that relates the ROC function to the Precision-Recall function. We discuss variations of the Precision-Recall function that will be useful.
Given two detection system families A and B, for which we know their respective ROC functions, we know the transformation that produces the ROC function of the conjunction of A with B, and the ROC function of the disjunction of A with B. We review these transformations and relate them to the PR functions. In particular, given the PR functions for detection system families A and B, we produce the PR functions for the detection system families A conjoin B and A disjoin B. Examples are given that demonstrate the theory and usefulness of the transformation to predict the performance of the fused systems. The extension to multiple label classification systems will be presented.
Complex ATR tasks are often decomposed into the identification of sub-targets, that is, objects are sorted and identified as one particular target type and then those targets are further identified. For instance, a field of view may be partitioned into natural and man-made objects. After which, the man-made objects are screened to identify a particular object of interest. These tasks combine classifiers which operate in isolation of each other, yet in fact, perform as a classification sequence. This work examines this scenario, building the ATR task as a sequence of target identifications. Two sequences will be highlighted: Believe the Negative (BN) and Believe the Extremes (BE). In a BN sequence, the second classification system only operates if a target is identified from the first classification system. In a BE sequence, the second classification system only operates if there is no identification from the first classification system. Performance of these classification sequences will be compared to classification systems operating separately. Further, sequence augmentation will be examined to demonstrate how the ATR task may still be completed when information is missing on the primary target. This missing information may represent atmospheric blurring, alternate field of view, or other disturbances. An example of the performance of the sequences under simulated, theoretical levels of missing information is examined, and formulas are presented to describe the optimal performance of these systems when augmented and un-augmented. In conclusion, this work demonstrates utility in how these sequences fuse target information in order to complete an ATR task.
A detection system outputs two distinct labels, thus, there are two errors it can make. The Receiver Operating Characteristic (ROC) function quantifies both of these errors as parameters vary within the system. Combining two detection systems typically yields better performance when a combining rule is chosen appropriately. When detection systems are combined the assumption of independence is usually made in order to simplify the math- ematics, so that we need only combine the individual ROC curve from each system into one ROC curve. This paper investigates label fusion of two detection systems drawn from a single Detection System Family (DSF). Given that one knows the ROC function for the DSF, we seek a formula with the resultant ROC function of the fused detection systems as a function (specifically, a transformation) of the ROC function. In this paper, we derive this transformation for the disjunction and conjunction label rules. Examples are given that demonstrates this transformation. Furthermore, another transformation is given to account for the dependencies between the two systems within the family. Examples will be given that demonstrates these ideas and the corresponding transformation acting on the ROC curve.
In a two class label scenario, classi cation systems may be used to assess whether or not an element of interest
belongs to the targetor non-targetclass. The performance of the system is summarized visually as a trade-
o¤ between the proportions of elements correctly labeled as targetplotted against the proportion of elements
incorrectly labeled as target. These proportions are empirical estimates of the true and false positive rates,
and their trade-o¤ plot is known as a receiver operating characteristic (ROC) curve. Classi cation performance
can be increased, however, if the information provided by multiple systems can be fused together to create a
new, combined system. This research focuses on label-fusion as a common method to increase classi cation
performance and quantifying the bias that occurs when misspecifying the partitioning of the underlying event
set. This partitioning will be de ned in terms of what be called within and across label fusion. When incorrect
assumptions are made about the partitioning of the event set, bias will occur and both the ROC curve and its
optimal parameters will be incorrectly quanti ed. In this work, we analyze the e¤ects of individual classi cation
system performance, correlation, and target environment on the magnitude of this performance bias. This work
will then inspire the development of formulas to adjust optimal performance measures to appropriately reect
the fused system performance according to event set partitioning. As such, bias may be appropriately adjusted
without redesigning the fused system, allowing greater use of currently fused systems across multiple platforms
and environments.
Given two legacy exploitation systems, whose performances are known, one might wish to determine if combining these together using some rule would yield a new exploitation system with improved performance. This is the fusion process. Often there are several performance objectives one would consider in this process. We investigate the fusion process based upon multiple performances. This is related to multi-objective optimization, but is different in some aspects. In this paper we pose a multi-performance problem for combining two classifications systems and derive the multi-performance fusion theory. A classification system with M possible output labels will have M(M-1) possible errors. The Receiver Operating Characteristic (ROC) manifold was created to quantify all of these errors. The assumption of independence is usually made to simply the mathematics of combining the individual systems into one system. Boolean rules do not exist for multiple symbols, thus, Boolean-like rules were created that would yield label fusion rules. An M-label system will have M! consistent rules. The formula for the resultant ROC manifold of the fused classification systems which incorporates the individual classification systems previously was derived. For the multi-performance problem we show how the set of permutations of the label set is used to generate all of the consistent rules and how the permutation matrix is incorporated into a single formula for the ROC manifold. Examples will be given that demonstrate how the solution to the multi-performance fusion problem relates to the solution of the single performance fusion problem.
A classification system with M possible output labels (or decisions) will have M(M-1) possible errors. The Receiver Operating Characteristic (ROC) manifold was created to quantify all of these errors. When multiple classification systems are fused, the assumption of independence is usually made in order to combine the individual ROC manifolds for each system into one ROC manifold. This paper will investigate the label fusion (also called decision fusion) of multiple classification system families (CSF) that have the same number of output labels. Boolean rules do not exist for multiple symbols, thus, we will derive Boolean-like rules as well as other rules that will yield label fusion rules. An M-label system will have M! consistent rules. The formula for the resultant ROC manifold of the fused classification system family which incorporates the individual classification system families will be derived. Specifically, given a label rule and two classification system families, the ROC manifold for the fused family is produced. We generate the formula for the Boolean-like AND ruled and give the resultant ROC manifold for the fused CSF. We show how the set of permutations of the label set is used to generate all of the consistent rules and how the permutation matrix is incorporated into a single formula for the ROC manifold. Examples will be given that demonstrate how each formula is used.
A detection system outputs two distinct labels. Thus, there are two errors it can make. The Receiver Operating Characteristic
(ROC) function quantifies both of these errors as parameters vary within the system. Combining two detection systems
typically yields better performance when a combination rule is chosen appropriately. When multiple detection systems
are combined, the assumption of independence is usually made in order to mathematically combine the individual ROC
functions for each system into one ROC function. This paper investigates feature fusion of multiple detection systems.
Given that one knows the ROC function for each individual detection system, we seek a formula with the resultant ROC
function of the fused detection systems as a function (specifically, a transformation) of the respective ROC functions. In
this paper we derive this transformation for a certain class of feature rules. An example will be given that demonstrates
this transformation.
A classification system with N possible output labels (or decisions) will have N(N-1) possible errors. The
Receiver Operating Characteristic (ROC) manifold was created to quantify all of these errors. When multiple
classication systems are fused, the assumption of independence is usually made in order to mathematically
combine the individual ROC manifolds for each system into one ROC manifold. This paper will investigate
the label fusion (also called decision fusion) of multiple classication systems that have the same number
of output labels. Boolean rules do not exist for multiple symbols, thus, we will derive possible Boolean-like
rules as well as other rules that will yield label fusion rules. The formula for the resultant ROC manifold
of the fused classication systems which incorporates the individual classication systems will be derived.
Specically, given a label rule and two classication systems, the ROC manifold for the fused system is
produced. We generate formulas for other non-Boolean-like OR and non-Boolean-like AND rules and give
the resultant ROC manifold for the fused system. Examples will be given that demonstrate how each formula
is used.
Safe exposure limits for directed energy sources are derived from a compilation of known injury thresholds
taken primarily from animal models and simulation data. The summary statistics for these experiments are
given as exposure levels representing a 50% probability of injury, or ED50, and associated variance. We
examine biological variance in focal geometries and thermal properties and the influence each has in singlepulse
ED50 threshold studies for 514-, 694-, and 1064-nanometer laser exposures in the thermal damage
time domain. Damage threshold is defined to be the amount of energy required for a retinal burn on at
least one retinal pigment epithelium (RPE) cell measuring approximately 10 microns in diameter. Better
understanding of experimental variance will allow for more accurate safety buffers for exposure limits and
improve directed energy research methodology.
A classification system such as an Automatic Target Recognition (ATR) system might yield better performance when fused
sequentially than in parallel. Most fused systems have parallel architecture, but, the medical community often uses sequential
tests due to costs constraints. We define the different types of sequential fusion and investigate their characteristics.
We compare parallel fused systems with sequential fused systems. Another goal of this paper is to compare competing sequential
fused systems to arrive at an optimal architecture design given the systems at hand. These systems may be legacy
systems whose performances are well known. If these systems have known Receiver Operating Characteristic (ROC)
curves/manifolds then we derive a formula that yields the ROC curve/manifold for the resultant sequentially fused system,
thus, enabling one to make these comparisons. This formula is distribution free. We give an example to demonstrate the
utility of our method, and show that one can play "what if" scenarios.
A Classification system such as an Automatic Target Recognition (ATR) system with N possible output labels (or decisions)
will have N(N-1) possible errors. The Receiver Operating Characteristic (ROC) manifold was created to quantify all
of these errors. Finite truth data will produce an approximation to a ROC manifold. How well does the approximate
ROC manifold approximate the TRUE ROC manifold? Several metrics exist that quantify the approximation ability, but
researchers really wish to quantify the confidence in the approximate ROC manifold. This paper will review different
confidence definitions for ROC curves and will derive an expression for confidence of a ROC manifold. The foundation of
the confidence expression is based upon the Chebychev inequality..
The Receiver Operating Characteristic (ROC) curve is typically used to quantify the performance of Automatic Target Recognition (ATR) systems. When multiple systems are to be fused, assumptions are made in order to mathematically combine the individual ROC curves for each of these ATR systems in order to form the ROC curve of the fused system. Often, one of these assumptions is independence between the systems. However, correlation may exist between the classifiers, processors, sensors and the outcomes used to generate each ROC curve. This paper will demonstrate a method for creating a ROC curve of the fused systems which incorporates the correlation that exists between the individual systems. Specifically, we will use the derived covariance matrix between multiple systems to compute the existing correlation and level of dependence between pairs of systems. The ROC curve for the fused system is then produced, adjusting for this level of dependency, using a given fusion rule. We generate the formula for the Boolean OR and AND rules, giving the exact ROC curve for the fused system, that is, not a bound.
Receiver operating characteristic (ROC) curves provide a means to
evaluate the performance of ATR systems. Of specific interest, is
the ability to evaluate the performance of fused ATR systems in
order to gain information on how well the combined system performs
with respect to, for instance, single systems, other fusion
methods, or pre-specified performance criteria. Although various
ROC curves for fused systems have been demonstrated by many
researchers, information regarding the bounds for these curves has
not been examined thoroughly. This paper seeks to describe several
bounds that exist on ROC curves from fused, correlated ATR
systems. These bounds include the lower bound for systems fused
using Boolean rules and bounds based on a measure of the variance
of the ATR system. Examples using simulated ROC curves generated
from correlated and uncorrelated ATR systems will be given as well
as a discussion of how correlation affects these bounds. Examining
such bounds for a set of candidate fusion rules a priori can focus
efforts towards those fused systems that better meet specified ATR
system performance criteria.
n Automatic Target Recognition (ATR) system with N possible output labels (or decisions) will have N(N − 1) possible
errors. The Receiver Operating Characteristic (ROC) manifold was created to quantify all of these errors. When multiple
ATR systems are fused, the assumption of independence is usually made in order to mathematically combine the individual
ROC manifolds for each system into one ROC manifold. This paper will investigate the label fusion (also called decision
fusion) of multiple classification systems that have the same number of output labels. Boolean rules do not exist for multiple
symbols, thus, we will derive possible Boolean-like rules as well as other rules that will yield label fusion rules. The
formula for the resultant ROC manifold of the fused classification systems which incorporates the individual classification
systems will be derived. Specifically, given a fusion rule and two classification systems, the ROC manifold for the fused
system is produced. We generate formulas for the Boolean-like OR rule, Boolean-like AND rule, and other rules and give
the resultant ROC manifold for the fused system. Examples will be given that demonstrate how each formula is used.
The Receiver Operating Characteristic (ROC) curve can be used to quantify the performance of Automatic Target Recognition
(ATR) systems. When multiple classification systems are fused, the assumption of independence is usually made
in order to mathematically combine the individual ROC curves for each of these classification systems into one fused
ROC curve. However, correlation may exist between the classification systems and the outcomes used to generate each
ROC curve. This paper will demonstrate a method for creating a ROC curve of the fused classification systems which
incorporates the correlation that exists between the individual classification systems. Specifically, we will use the derived
covariance between multiple classification systems to compute the existing correlation and thus the level of dependence
between pairs of classification systems. Then, given a fusion rule, two systems, and the correlation between them, the ROC
curve for the fused system is produced. We generate the formula for the Boolean OR and AND rules, giving the resultant
ROC curve for the fused system. This paper extends our previous work in which bounds for the ROC curve of the fused,
correlated classification systems were presented.
Significant advances in the performance of ATR systems can be made when fusing individual classification systems into a single combined classification system. Often, these individual systems are dependent, or correlated, with one another. Additionally, these systems typically assume that two outcome labels, (for instance "target" and "non-target") exist. Little is known about the performance of fused classification systems when multiple outcome labels are used. In this paper, we propose a methodology for quantifying the performance of the fused classifier system using multiple labels. Specifically, a performance measure for a fused classification system using two classifiers and multiple labels will be developed. The performance measure developed is based on the Receiver Operating Characteristic (ROC) curve. The ROC curve in a two-label system has been well defined and used extensively, in not only ATR applications, but also other engineering and biomedical applications. A ROC manifold is defined and use in order to incorporate the multiple labels. An example of this performance measure for a given fusion rule and multiple labels is given.
The Receiver Operating Characteristic (ROC) curve is typically used to quantify the performance of Automatic Target Recognition (ATR) systems. When multiple classifiers are to be fused, assumptions must be made in order to mathematically combine the individual ROC curves for each of these classifiers in order to form one fused ROC curve. Often, one of these assumptions is independence between the classifiers. However, correlation may exist between the classifiers, processors, sensors and the outcomes used to generate each ROC curve. This paper will demonstrate a method for creating a ROC curve of the fused classifiers which incorporates the correlation that exists between the individual ROC curves. Specifically, we will use the derived covariance matrix between multiple classifiers to compute the existing correlation and level of dependence between pairs of classifiers. The ROC curve for the fused system is then produced, adjusting for this level of dependency, using a given fusion rule.
Typically, when considering multiple classifiers, researchers assume that they are independent. Under this assumption estimates for the performance of the fused classifiers are easier to obtain and quantify mathematically. But, in fact, classifiers may be correlated, thus, the performance of the fused classifiers will be over-estimated. This paper will address the issue of the dependence between the classifiers to be fused. Specifically, we will assume a level of dependence between two classifiers for a given fusion rule and produce a formula to quantify the performance of this newly fused classifier. The performance of the fused classifiers will then be evaluated via the Receiver Operating Characteristic (ROC) curve. A classifier typically relies on parameters that may vary over a given range. Thus, the probability of true and false positives can be computed over this range of values. The graph of these probabilities over this range then produces the ROC curve. The probability of true positive and false positive from the fused classifiers are developed according to various decision rules. Examples of dependent fused classifiers will be given for various levels of dependency and multiple decision rules.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.