Simultaneous advances in neural interfacing and data science have created a remarkable opportunity to reshape our understanding of the brain in health and disease. Unfortunately, the most powerful advances in modern machine learning sometimes provide limited scientific insight due to the opaqueness of these "black box" techniques. To illustrate, I will describe our recent efforts collecting local field potential (LFP) data from participants undergoing subcallosal cingulate cortex (SCC) deep brain stimulation (DBS) as an experimental therapy for treatment-resistant depression. While SCC DBS can be effective, scaling this technique to an approved therapy requires objective characterization of patient-specific disease trajectories toward stable recovery. I will describe our development of a novel "explainable AI" technique for generating causal post-hoc explanations of black-box classifiers based on a learned low-dimensional representation of unsupervised data. Using this data-driven approach, we identify unexpected changes in SCC LFP dynamics that correspond to individual recovery trajectories, behavioral changes (reflected in facial expressions), and white matter structural integrity. In addition to illuminating changes to neural dynamics with chronic SCC DBS, our results demonstrate potential biomarkers for future clinical trials and closed-loop neuromodulation systems.