Organizational Unit:

School of Computational Science and Engineering

Permanent Link

https://hdl.handle.net/1853/70780

Parent Organization

Organizational Unit

College of Computing

ArchiveSpace Name Record

https://finding-aids.library.gatech.edu/agents/corporate_entities/1111

Full item page

Publication Search Results

Now showing 1 - 10 of 96

Federated approaches for the visualization and analysis of neuroimaging data

(Georgia Institute of Technology, 2023-12-13) Saha, Debbrata Kumar

In the neuroimaging domain, the data collection process is expensive, and attempting to pool data from multiple imaging sites faces numerous challenges, including variations in data acquisition protocols from site to site. There is also concern associated with revealing the identities of rare disease subjects. The challenges of data sharing associated with these issues prevent the datasets from being as large as desired, ultimately hindering the benefits of utilizing large-scale datasets in research operations. This dissertation aims to address these potential challenges. First, we develop a federated embedding algorithm to assess the quality control of neuroimaging datasets. Our algorithm has demonstrated superior performance in overcoming challenges that some notable existing algorithms struggle to solve. Subsequently, we introduce a privacy-preserving algorithm tailored to the neuroimaging domain, ensuring formal mathematical privacy guarantees during message passing in federated computation. The integration of this algorithm with the existing software platform for federated neuroimaging has been demonstrated, making our methods readily available as tools for neuroimaging users worldwide. Our third proposed approach emphasizes fast federated communication with more stringent privacy assurances. Lastly, we design a federated algorithm to extract multivariate patterns (covarying networks) from structural magnetic resonance imaging (sMRI) data for the analysis of brain morphometry. These four proposed methods enable neuroimaging users to perform operations in a federated environment where it is not possible to run operations centrally in typical scenarios.
Safe Explanations And Explainable Models For Neuroimaging Data Through A Framework Of Constraints

(Georgia Institute of Technology, 2023-12-11) Lewis, Noah Jerome

Neuroimaging data, which can be highly complex and occasionally inscrutable, requires robust, reproducible, and domain-specific methods. Deep learning and model explainability have become common methods for analyzing neuroimaging data. However, the complex, obscure, and sometimes flawed nature of both deep learning and explainability compound the difficulties in neuroimaging analysis. This dissertation addresses several of these issues with explainability by employing a framework of constraint-based solutions. These constraints span the entire modeling pipeline, including initialization, model parameters and gradients, and the loss functions. To familiarize the readers with the field, this dissertation will begin with a comprehensive investigation into current explainability methods both in general and specific to neuroimaging, then describe the three constraint-based methodologies that comprise this framework. First, we develop an attention-based constraint for recurrent models that resolves vanishing saliency. Vanishing saliency is closely related to vanishing gradients, a common issue for training, in which the gradients lose value during backpropagation. Our second proposed method is a set of initialization constraints that target underspecification and its implications for post-hoc explanations. Our final proposed method leverages inherent neuroimaging-based geometric information in the input to constrain the optimization approach to produce more interpretable models. These three constraint methods amount to a broad framework that provides a robust and reproducible explanatory system appropriate for neuroimaging.
Interactive Scalable Discovery Of Concepts, Evolutions, And Vulnerabilities In Deep Learning

(Georgia Institute of Technology, 2023-12-05) Park, Haekyu

Deep Neural Networks (DNNs) are increasingly prevalent, but deciphering their operations is challenging. Such a lack of clarity undermines trust and problem-solving during deployment, highlighting the urgent need for interpretability. How can we efficiently summarize concepts models learn? How do these concepts evolve during training? When models are at risk from potential threats, how do we explain their vulnerabilities? We address these concerns with a human-centered approach, by developing novel systems to interpret learned concepts, their evolution, and potential vulnerabilities within deep learning. This thesis focuses on three key thrusts: (1) Scalable Automatic Visual Summarization of Concepts. We develop NeuroCartography, an interactive system that scalably summarizes and visualizes concepts learned by a large-scale DNN, such as InceptionV1 trained with 1.2M images. A large-scale human evaluation with 244 participants shows that NeuroCartography discovers coherent, human-meaningful concepts. (2) Insights to Reveal Model Vulnerabilities. We develop scalable interpretation techniques to visualize and identify internal elements in DNNs, which are susceptible to potential harms, aiming to understand how these defects lead to incorrect predictions. We develop first-of-its-kind interactive systems such as Bluff that visually compares the activation pathways for benign and attacked images in DNNs, and SkeletonVis that explains how attacks manipulate human joint detection in human action recognition models. (3) Scalable Discovery of Concept Evolution During Training. Our first-of-its-kind ConceptEvo unified interpretation framework holistically reveals the inception and evolution of learned concepts and their relationships during training. ConceptEvo enables powerful new ways to monitor model training and discover training issues, addressing critical limitations of existing post-training interpretation research. A large-scale human evaluation with 260 participants demonstrates that ConceptEvo identifies concept evolutions that are both meaningful to humans and important for class predictions. This thesis contributes to information visualization, deep learning, and crucially, their intersection. We have developed open-source interactive interfaces, scalable algorithms, and a unified framework for interpreting DNNs across different models. Our work impacts academia, industry, and the government. For example, our work has contributed to the DARPA GARD program (Garanteeing AI Robustness against Deception). Additionally, our work has been recognized through a J.P. Morgan AI PhD Fellowship and 2022 Rising Stars in IEEE EECS. NeuroCartography has been highlighted as a top visualization publication (top 1%) invited to SIGGRAPH.
Learning with Less: Low-rank Dynamics, Communication, and Introspection in Machine Learning

(Georgia Institute of Technology, 2023-10-03) Baker, Bradley Thomas

The enclosed research is a focused empirical and theoretical analysis of the optimization methods in machine learning, and the underlying role that the matrix rank of utilized learning statistics plays in these algorithms. We show that this new perspective on machine learning optimization provides benefits in terms of communication-efficient federated learning algorithms, as well as novel insights in terms of model introspection and theory of learning dynamics. In applications to the complex domain of Neuroimaging data analysis, we show that this rank-focused frame of reference allows for unique insights into how models perform on particular populations.
Fast and compact neural network via Tensor-Train reparameterization

(Georgia Institute of Technology, 2023-08-28) Yin, Chunxing

The exponential growth of data and model size poses a number of challenges for deep learning training. Large neural network layers can be parameterized based on tensor decomposition to compress model size, but at the potential costs of degraded accuracy and more execution time to reconstruct the layer parameters from the tensorized representation. In this dissertation, we explore neural network compression through Tensor Train (TT) reparameterization. We aim to develop efficient algorithms to accelerate training of tensorized networks while minimizing the memory consumption, and to understand the necessary components for Tensor Train format to succeed in model compression. We design efficient algorithms to accelerate the training of tensorized layers in Convolutional Neural Networks (CNNs), Deep Learning Recommendation Models (DLRMs), and in Graph Neural Networks (GNNs). While the use of TT for compression in CNNs has been suggested in the past, the prior art has not demonstrated significant speedups for training or inference. The reason is that conventional implementations of TT-compressed convolutional layers pose several challenges: increases in computational work for reconstructing TT-compressed layers, increases in memory footprint due to weight reconstruction, and limitations to parallel scalability as the effective problem sizes shrink under compression. We address these issues through asymptotic reductions in computation, avoidance of data movement, and an alternative parallelization strategy that significantly improves scalability. In recommendation models, the performance of TT-compressed DLRM (TT-Rec) is further optimized with the batched matrix multiplication and caching strategies for embedding vector lookup operations. In addition, we present mathematically and empirically the effect of weight initialization distribution on DLRM accuracy and propose to initialize the tensor cores of TT-Rec following the sampled Gaussian distribution. In the next part of this dissertation, we study the node embeddings in graph neural networks where both the numerical features and topological graph information need to be preserved. We design training schemes that unify hierarchical tensor decomposition and graph topology to exploit graph homophily, as well as to develop novel parameter initialization algorithms that introduces graph spectrum to improve model convergence and accuracy. Finally, we evaluate our technique on million-node graphs to demonstrate the efficiency and accuracy in real-world graphs, as well as on synthetic graphs to understand the correlation between graph homophily and weight sharing in TT. While the primary focus of this dissertation lies in exploring proof-of-concept algorithms, its outcomes can hold significant implications for systems. For example, by transforming the data-intensive embedding operator to compute-intensive and memory-efficient tensorized embedding, we can potentially reconfigure the allocation of system resources within a heterogeneous data-center with a combination of CPUs and GPUs. Moreover, our compression technique would enable storing large modules on a limited-memory accelerator with data-parallelism, thereby providing opportunities for optimizing communication.
Scalable Algorithms for Hypergraph Analytics using Symmetric Tensor Decompositions

(Georgia Institute of Technology, 2023-08-28) Shivakumar, Shruti

Tensors are higher-dimension generalizations of matrices and are used to represent multi-dimensional data. Tensor-based methods are receiving renewed attention in recent years due to their prevalence in diverse real-world applications. Symmetric tensors are an important class of tensors, arising in diverse fields such as signal processing, machine learning, and hypergraph analytics. Hypergraphs, generalizations of graphs which allow edges to span multiple vertices, have become ubiquitous in understanding real-world networks and multi-entity interactions. Affinity relations in a hypergraph can be represented as a high-order adjacency tensor which is sparse and symmetric. While mathematical research on symmetric tensors is longstanding, emerging massive data in these applications has sparked the demand for scalable, efficient algorithms that utilize advances in numerical linear algebra, numerical optimization, as well as high-performance computing. State-of-the-art tensor libraries incorporate high-performance tensor methods for general sparse tensors; however, they lack specialized algorithms for sparse tensors that are symmetric. This dissertation focuses on scaling hypergraph analytics to real-world datasets by taking advantage of the sparsity and symmetry of the associated adjacency tensors through the development of compact storage formats and efficient serial and parallel algorithms for tensor operations. We present a novel computation-aware compressed storage format - CSS - for sparse symmetric tensors, along with efficient parallel algorithms for symmetric tensor operations that are compute- and memory-intensive due to the high tensor order and the associated factorial explosion in the number of non-zeros. In order to scale to large multi-entity complex networks, we consider the problem of distributed-memory hypergraph analytics. To that end, we present algorithms for parallel distributed-memory line graph construction of hypergraphs and demonstrate their application to large-scale symmetric adjacency tensor decomposition for hypergraph clustering. For hypergraphs with varying edge cardinalities, the CSS format has been extended to the CCSS format, using which we present a new shared-memory parallel algorithm for a key symmetric tensor kernel in the complutation of hypergraph tensor eigenvector centrality. Finally, we present Coupled Symmetric Tensor Completion (CoSTCo), a Riemannian optimization framework for the task of link prediction in non-uniform hypergraphs and analyze its performance with both synthetic and real-world datasets against state-of-the-art general tensor completion algorithms.
Multifidelity Memory System Simulation

(Georgia Institute of Technology, 2023-08-25) Lavin, Patrick

As computer systems grow larger and more complex, it takes more time to simulate them in detail. Researchers interested in simulating large systems must choose between simpler, less-accurate models or simulating smaller portions of their benchmarks, both of which can be highly manual, offline approaches that require time-consuming analysis by experts. Multifidelity simulation aims to lessen this burden by adapting the fidelity of a simulation to the complexity of the behavior being simulated. Multifidelity simulation refers to a simulation that can utilize multiple models for the same phenomena at different levels of fidelity. We borrow the phrase from the simulation of physical systems where scientists may have models with more or fewer terms, or may resolve their models on smaller or larger grid sizes, depending on the nature of the behavior at any point or time in the simulation. We have taken those ideas and applied them to computer architecture simulation. In this dissertation, we will present our novel multifidelity computer architecture simulation algorithm and implement it in two separate models: one for the cache and one for the entire memory system. Our cache model is able to automatically train and choose between low-fidelity models to adapt to the complexity of the modeled behavior online. The second model, the memory system, refines upon the ideas developed to create the first. We use statistical techniques to choose data that is used to create the low-fidelity models and implement this work as reusable components within a widely-used simulator, SST. This model achieves up to 2x speedup with only 1-5% mean error in the instructions per cycle.
Artificial Intelligence for Data-centric Surveillance and Forecasting of Epidemics

(Georgia Institute of Technology, 2023-08-15) Rodriguez Castillo, Alexander D.

Surveillance and forecasting of epidemics are crucial tools for decision making and planning of government officials, businesses, and the general public. In many respects, our understanding of how epidemics spread is still at its infancy, despite multiple advances in understanding how diseases spread in the population. Many of the major challenges stem from other complex dynamics, such as mobility patterns, policy compliance, and even shifts in data collection procedures. As a result of efforts to collect and process data from novel sources, granular data are becoming increasingly available on many of these variables. These datasets, however, are difficult to exploit using traditional methodologies from mathematical epidemiology and agent-based modeling. Alternatively, AI methods in epidemiology are challenged by data sparsity, distributional changes, and disparities in data quality. AI also lacks understanding of epidemic dynamics, which may lead to unrealistic predictions. Several frameworks are proposed in this dissertation to address these challenges and move toward more data-centric methods. Specifically, we utilize multiple examples to showcase that bringing the data-driven expressibility of AI into epidemiology leads to more sensitive and precise surveillance and forecasting of epidemics.
Learning-based search algorithm design

(Georgia Institute of Technology, 2023-07-27) Chen, Binghong

Classical search algorithms, such as A* search, evolutionary search, and Monte Carlo tree search, play a central role in solving many combinatorial optimization problems, including robotic path planning, molecule optimization, and playing Go. Conventionally, domain experts design these algorithms analytically by hand. However, such an algorithm design process is not data-driven and cannot benefit from the ever-increasing volumes of data. In this dissertation, we introduce a series of learning-to-search methods for different types of search space. We show that both search efficiency and effectiveness can be improved by learning search algorithms from historical data. Specifically, we focus on addressing challenges in a number of continuous and discrete search spaces with applications in robotics and drug design. High-dimensional continuous space. To search for solutions in high-dimensional continuous space, we resort to sampling-based methods to avoid explicitly discretizing the space. We propose a neural path planner that learns from prior experience to solve new path planning problems. Compared to classical methods, our learning-to-search approach achieves higher sample efficiency in high dimensions and can benefit from prior experience in similar environments. Discrete graph space. Finding graph structures with desired properties is a challenging problem. We present a learning-based evolutionary search algorithm to optimize molecules for desired properties. The proposed algorithm leverages a graph explainable model and the REINFORCE algorithm to generate better molecules on a multi-property molecule optimization benchmark. Discrete decomposable combinatorial space. We present a framework to search for solutions to recursively decomposable problems, based on the AND-OR tree representation that efficiently describes the search space. For the retrosynthesis planning problem, we introduce a learning-based A*-like search algorithm that finds high-quality synthetic routes for target molecules efficiently. The proposed algorithm builds on top of the AND-OR search tree and provides theoretical guarantees similar to the A* algorithm. Continuous molecular conformational space. We present a framework to search for molecular conformers with low energy, based on an Equivariant Transformer Forcefield. This strategy begins with an initial set of conformers, which are subsequently refined through structural optimization. We demonstrate that our ETF-based optimization significantly improves the quality of the conformers generated by state-of-the-art methods, achieving a 45\% reduction in the distance to the reference conformers.
Deep Generative Models for Drug Design

(Georgia Institute of Technology, 2023-04-21) Fu, Tianfan

Machine learning in drug discovery has drawn significant attention and attracted explosive growth in drug discovery and development research. This dissertation studies the deep generative methods in drug design. Despite the rapid progress of machine learning, especially deep learning in drug discovery, the existing drug design methods remain challenging for real-world applications in both categories of the methods from different aspects, including sample efficiency and data requirement, which are summarized as follows. • Sample efficiency. Existing drug optimization methods rely heavily on brute-force trial-and-error strategy and suffer from poor sample efficiency. A sample-efficient drug design method would save much time and computational resources. • Data efficiency. Acquiring data labels (e.g., drugs’ property) is typically laborious and time-consuming in drug discovery because it involves bioassay based wet-lab experiments, or animal models. This dissertation focuses on addressing these challenges by enhancing/designing the following categories of deep generative models: • Enhancing graph-to-graph neural architecture. Graph-to-graph neural architecture is used in drug design to translate a molecule to another similar molecule with property improvement. We design copy & refine (CORE) strategy [1] and Molecule Reward in deep generative models (MOLER) [2] that leverages policy gradient of reinforcement learning. Graph-to-graph methods are easy to train in end-to-end manner and do not require an online oracle query. However, it suffers from data- and sample- inefficiency. • Self-supervised learning (SSL) for generation. SSL can be pretrained in large unlabeled data, alleviating the high demand for labeled data. During the generation process, SSL masks a subset of the whole drug molecule and samples the masked part based on deep neural network’s prediction. It can be applied on both small-molecule drugs (Multi-objective molecule sampling (MIMOSA) [3]) and biologics design (sampling method for inverse protein folding (SIPF) [4]). The pros are that self-supervised learning based generation can quantify the uncertainty and be data-efficient. However, it suffers from sample inefficiency. • Differentiable programming for generation. The discrete drug molecules are relaxed to differentiable ones on continuous space, so the gradient of neural network can be back-propagated to update the differentiable drug molecules directly. The strategy can be also applied to both small-molecule drug (differentiable scaffolding tree (DST) [5]) and biologics (constrained energy model (CEM) [6]). Differentiable programming is data- and sample-efficient. However, it still requires online oracle query. • Intelligent combinatorial optimization. Traditional combinatorial optimization methods such as genetic algorithms (GA) rely heavily on a random-walk-like exploration, which leads to unstable performance. To address this challenge, we propose a Reinforced Genetic Algorithm (RGA) that uses neural models to prioritize the profitable design steps. Intelligent combinatorial optimization suppresses random-walk behavior and enhance efficiency [7]. However, it still requires online oracle query. In the last chapter, we describe future works to extend the current research. First, we will build some hybrid models to inherit the advantages of multiple categories of generative methods. Second, we will conduct comprehensive experiments to systematically compare these generative methods. [1] Tianfan Fu, Cao Xiao, Jimeng Sun: CORE: Automatic Molecule Optimization Using Copy and Refine Strategy. Association for the Advancement of Artificial Intelligence (AAAI) 2020. [2] Tianfan Fu, Cao Xiao, Lucas Glass, Jimeng Sun: MOLER: Incorporate Molecule-Level Reward to Enhance Deep Generative Model for Molecule Optimization. IEEE Transactions on Knowledge and Data Engineering (TKDE) 2021. [3] Tianfan Fu, Cao Xiao, Xinhao Li, Lucas Glass, Jimeng Sun: MIMOSA: Multi-constraint Molecule Sampling for Molecule Optimization. Association for the Advancement of Artificial Intelligence (AAAI) 2021. [4] Tianfan Fu, Jimeng Sun. SIPF: Sampling Method for Inverse Protein Folding. The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2022). [5] Tianfan Fu*, Wenhao Gao*, Cao Xiao, Jacob Yasonik, Connor W. Coley, Jimeng Sun. Differentiable Scaffolding Tree for Molecular Optimization. International Conference on Learning Representation (ICLR), 2022. [6] Tianfan Fu, Jimeng Sun. Antibody Complementarity Determining Regions (CDRs) design using Constrained Energy Model. The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2022). [7] Tianfan Fu*, Wenhao Gao*, Connor W. Coley, Jimeng Sun. Reinforced genetic algorithm for structurebased drug design, NeurIPS, 2022.