(Georgia Institute of Technology, 2020-12-07)
Grimm, David A.P.
A resilient team would be proficient at overcoming sudden, unexpected changes by displaying a rapid, adaptive response to maintain effectiveness. To quantify resilience, I analyzed data from two different experiments examining performance of human-autonomy teams (HATs) operating in a remotely piloted aircraft system (RPAS). Across both experiments, the HATs experienced a variety of automation and autonomy failure perturbations using a Wizard of Oz paradigm. Team performance was measured by the successful completion of simulated reconnaissance missions, a mission level team performance score, a coordination-based target processing efficiency (TPE) score to quantify team efficiency, and a ground truth resilience score (GTRS) to measure how teams performed during and following a failure. Different layers, composed of vehicle, operator controls, communication, and overall system layers, of sociotechnical elements of the system were measured across RPAS missions. To measure resilience, I used entropy and a root mean squared error (RMSE) metric across all system layers. I used these measures to examine the time taken to achieve extreme values of reorganization during a failure and the novelty of the reorganization, respectively, to quantify resilience. I hypothesized that faster times to achieve extreme values of reorganization during a failure would be correlated with all performance measures. Across both experiments, I found negative correlations of time taken to achieve extreme values of reorganization and novelty of reorganization with team performance measured using TPE, and positive correlations while using GTRS. Additionally, I found that teams displayed more reorganization in response to failures, but this was not pronounced for effective teams. In Experiment 2, I also found differential effects of training in the communication and control layers. These results can help inform the measurement and training of resilience in HATs through targeted team training, feedback, and real-time analysis applications.