|
Claims of deep learning’s superhuman performance on medical tasks are typically predicated on evaluations in which the training data and test data sets are exchangeably (statistically identical), differentiated only by the random partitioning of a larger population from which both are sampled. However, this assumption is always violated in practice, sometimes with disastrous effects. Subtle differences in how images are captured or represented can bring an otherwise superhuman system down to unacceptably poor levels of accuracy. In this talk I will discuss the fundamental hardness of these problems—absent further assumptions on the nature of the shifts in distribution, no principled technique can exist for estimating the accuracy of models, let alone for adapting them to new environments. I will also discuss the vast landscape of (i) principled methods (and corresponding assumptions) for developing robust and adaptive predictive models and (ii) heuristic methods that have shown some promise on benchmarks but lack theory to guide their application.
|
