While ML-based AI systems are increasingly deployed in safety-critical settings, they continue to remain unreliable under adverse conditions that violate underlying statistical assumptions. In my work, I aim to (i) understand the conditions under which a lack of reliability can occur and (ii) reason rigorously about the limits of robustness, during both training and test phases.

In the first part of the talk, I demonstrate the existence of strong but stealthy training-time attacks on federated learning, a recent paradigm in distributed learning. I show how a small number of compromised agents can modify model parameters via optimized updates to ensure desired data is misclassified by the global model, while bypassing custom detection methods. Experimentally, this model poisoning attack leads to a lack of reliable prediction on standard datasets.

Test-time attacks via adversarial examples, i.e. imperceptible perturbations to test inputs, have sparked an attack-defense arms race. In the second part of the talk, I step away from this arms race to provide model-agnostic fundamental limits on the loss under adversarial input perturbations. The robust loss is shown to be lower bounded by the optimal transport cost between class-wise distributions using an appropriate adversarial point-wise cost, the latter of which can be efficiently computed via a linear program for empirical distributions of interest.

To conclude, I will discuss my ongoing efforts and future vision towards building continuously reliable and accessible ML systems by accounting for novel attack vectors and new ML paradigms such as generative AI, as well as developing algorithmic tools to improve performance in data-scarce regimes.