Autonomous systems are increasingly seen in safety-critical domains, such as self-driving vehicles and autonomous aircraft. But methods developed for ultra-reliable software generally depend on measures of structural coverage that do not apply well to AI and machine learning. The performance and safety of these systems depend on the data used in training models. How can we measure whether training data is an adequate representation of inputs that will be seen in operational environments? Combinatorial methods provide sound measures beyond conventional approaches. This talk introduces combinatorial data coverage, combination coverage differences, and combination frequency differences that can be used in measuring and visualizing important properties of data used in AI and ML systems.
IEEE Fellow
Virginia Tech / Former National Institute of Standards and Technology
Rick Kuhn is a computer scientist with an extensive background in cybersecurity and software verification and testing. His current interest is in applications of combinatorial methods to testing and simulation, with a focus on autonomous systems assurance. Prior to working with Virginia Tech, he led research in cybersecurity, software failure, and combinatorial test methods at the National Institute of Standards and Technology (NIST). He co-developed the role-based access control (RBAC) model that is the dominant form of access control today. His work in combinatorial testing includes both theory and empirical results in vulnerability and fault detection, and contributions to the ACTS combinatorial test tools used worldwide.
Kuhn is a Fellow of the Institute of Electrical and Electronics Engineers (IEEE), the Washington Academy of Sciences, and the American Association for the Advancement of Science (AAAS). He has authored three books and more than 200 conference or journal publications on cybersecurity, software failure, and software verification and testing. He received an MS in computer science from the University of Maryland, College Park, and an MBA from William & Mary. Before joining NIST, he worked as a software developer with NCR Corporation and the Johns Hopkins University Applied Physics Laboratory.
