Person:

Dellaert, Frank

Permanent Link

https://hdl.handle.net/1853/71202

Associated Organization(s)

Organizational Unit

School of Interactive Computing

Full item page

Publication Search Results

Now showing 1 - 9 of 9

Joint Semantic Segmentation and 3D Reconstruction from Monocular Video

(Georgia Institute of Technology, 2014-09) Kundu, Abhijit ; Li, Yin ; Dellaert, Frank ; Li, Fuxin ; Rehg, James M.

We present an approach for joint inference of 3D scene structure and semantic labeling for monocular video. Starting with monocular image stream, our framework produces a 3D volumetric semantic + occupancy map, which is much more useful than a series of 2D semantic label images or a sparse point cloud produced by traditional semantic segmentation and Structure from Motion(SfM) pipelines respectively. We derive a Conditional Random Field (CRF) model defined in the 3D space, that jointly infers the semantic category and occupancy for each voxel. Such a joint inference in the 3D CRF paves the way for more informed priors and constraints, which is otherwise not possible if solved separately in their traditional frameworks. We make use of class specific semantic cues that constrain the 3D structure in areas, where multiview constraints are weak. Our model comprises of higher order factors, which helps when the depth is unobservable. We also make use of class specific semantic cues to reduce either the degree of such higher order factors, or to approximately model them with unaries if possible. We demonstrate improved 3D structure and temporally consistent semantic segmentation for diffcult, large scale, forward moving monocular image sequence.
Learning and Inferring Motion Patterns Using Parametric Segmental Switching Linear Dynamic Systems

(Georgia Institute of Technology, 2008) Oh, Sang Min ; Rehg, James M. ; Balch, Tucker ; Dellaert, Frank

Switching Linear Dynamic System (SLDS) models are a popular technique for modeling complex nonlinear dynamic systems. An SLDS provides the possibility to describe complex temporal patterns more concisely and accurately than an HMM by using continuous hidden states. However, the use of SLDS models in practical applications is challenging for several reasons. First, exact inference in SLDS models is computationally intractable. Second, the geometric duration model induced in standard SLDSs limits their representational power. Third, standard SLDSs do not provide a systematic way to robustly interpret systematic variations governed by higher order parameters. The contributions in this paper address all three challenges above. First, we present a data-driven MCMC sampling method for SLDSs as a robust and efficient approximate inference method. Second, we present segmental switching linear dynamic systems (S-SLDS), where the geometric distributions are replaced with arbitrary duration models. Third, we extend the standard model with a parametric model that can capture systematic temporal and spatial variations. The resulting parametric SLDS model (P-SLDS) uses EM to robustly interpret parametrized motions by incorporating additional global parameters that underly systematic variations of the overall motion. The overall development of the proposed inference methods and extensions for SLDSs provide a robust framework to interpret complex motions. The framework is applied to the honey bee dance interpretation task in the context of the on-going BioTracking project at Georgia Institute of Technology. The experimental results suggest that the enhanced models provide an effective framework for a wide range of motion analysis applications.
Parameterized Duration Modeling for Switching Linear Dynamic Systems

(Georgia Institute of Technology, 2006-06) Oh, Sang Min ; Rehg, James M. ; Dellaert, Frank

We introduce an extension of switching linear dynamic systems (SLDS) with parameterized duration modeling capabilities. The proposed model allows arbitrary duration models and overcomes the limitation of a geometric distribution induced in standard SLDSs. By incorporating a duration model which reflects the data more closely, the resulting model provides reliable inference results which are robust against observation noise. Moreover, existing inference algorithms for SLDSs can be adopted with only modest additional effort in most cases where an SLDS model can be applied. In addition, we observe the fact that the duration models would vary across data sequences in certain domains, which complicates learning and inference tasks. Such variability in duration is overcome by introducing parameterized duration models. The experimental results on honeybee dance decoding tasks demonstrate the robust inference capabilities of the proposed model.
On-line Learning of the Traversability of Unstructured Terrain for Outdoor Robot Navigation

(Georgia Institute of Technology, 2006) Oh, Sang Min ; Rehg, James M. ; Dellaert, Frank
Learning and Inferring Motion Patterns using Parametric Segmental Switching Linear Dynamic Systems

(Georgia Institute of Technology, 2006) Oh, Sang Min ; Rehg, James M. ; Balch, Tucker ; Dellaert, Frank

Switching Linear Dynamic System (SLDS) models are a popular technique for modeling complex nonlinear dynamic systems. An SLDS has significantly more descriptive power than an HMM by using continuous hidden states. However, the use of SLDS models in practical applications is challenging for several reasons. First, exact inference in SLDS models is computationally intractable. Second, the geometric duration model induced in standard SLDSs limits their representational power. Third, standard SLDSs do not provide a systematic way to robustly interpret systematic variations governed by higher order parameters. The contributions in this paper address all three challenges above. First, we present a data-driven MCMC sampling method for SLDSs as a robust and efficient approximate inference method. Second, we present segmental switching linear dynamic systems (S-SLDS), where the geometric distributions are replaced with arbitrary duration models. Third, we extend the standard model with a parametric model that can capture systematic temporal and spatial variations. The resulting parametric SLDS model (P-SLDS) uses EM to robustly interpret parametrized motions by incorporating additional global parameters that underly systematic variations of the overall motion. The overall development of the proposed inference methods and extensions for SLDSs provide a robust framework to interpret complex motions. The framework is applied to the honey bee dance interpretation task in the context of the ongoing BioTracking project at Georgia Institute of Technology. The experimental results suggest that the enhanced models provide an effective framework for a wide range of motion analysis applications.
Learning and Inference in Parametric Switching Linear Dynamic Systems

(Georgia Institute of Technology, 2005-10) Oh, Sang Min ; Rehg, James M. ; Balch, Tucker ; Dellaert, Frank

We introduce parametric switching linear dynamic systems (P-SLDS) for learning and interpretation of parametrized motion, i.e., motion that exhibits systematic temporal and spatial variations. Our motivating example is the honeybee dance: bees communicate the orientation and distance to food sources through the dance angles and waggle lengths of their stylized dances. Switching linear dynamic systems (SLDS) are a compelling way to model such complex motions. However, SLDS does not provide a means to quantify systematic variations in the motion. Previously, Wilson & Bobick presented parametric HMMs [21], an extension to HMMs with which they successfully interpreted human gestures. Inspired by their work, we similarly extend the standard SLDS model to obtain parametric SLDS. We introduce additional global parameters that represent systematic variations in the motion, and present general expectation-maximization (EM) methods for learning and inference. In the learning phase, P-SLDS learns canonical SLDS model from data. In the inference phase, P-SLDS simultaneously quantifies the global parameters and labels the data. We apply these methods to the automatic interpretation of honey-bee dances, and present both qualitative and quantitative experimental results on actual bee-tracks collected from noisy video data.
Data-Driven MCMC for Learning and Inference in Switching Linear Dynamic Systems

(Georgia Institute of Technology, 2005-07) Oh, Sang Min ; Rehg, James M. ; Balch, Tucker ; Dellaert, Frank

Switching Linear Dynamic System (SLDS) models are a popular technique for modeling complex nonlinear dynamic systems. An SLDS has significantly more descriptive power than an HMM, but inference in SLDS models is computationally intractable. This paper describes a novel inference algorithm for SLDS models based on the Data- Driven MCMC paradigm. We describe a new proposal distribution which substantially increases the convergence speed. Comparisons to standard deterministic approximation methods demonstrate the improved accuracy of our new approach. We apply our approach to the problem of learning an SLDS model of the bee dance. Honeybees communicate the location and distance to food sources through a dance that takes place within the hive. We learn SLDS model parameters from tracking data which is automatically extracted from video. We then demonstrate the ability to successfully segment novel bee dances into their constituent parts, effectively decoding the dance of the bees.
A Variational inference method for Switching Linear Dynamic Systems

(Georgia Institute of Technology, 2005) Oh, Sang Min ; Ranganathan, Ananth ; Rehg, James M. ; Dellaert, Frank

This paper aims to present a structured variational inference algorithm for switching linear dynamical systems (SLDSs) which was initially introduced by Pavlovic and Rehg. Starting with the need for the variational approach, we proceed to the derivation of the generic (model-independent) variational update formulas which are obtained under the mean field assumption. This leads us to the derivation of an approximate variational inference algorithm for an SLDS. The details of deriving the SLDS-specific variational update equations are presented.
Segmental Switching Linear Dynamic Systems

(Georgia Institute of Technology, 2005) Oh, Sang Min ; Rehg, James M. ; Dellaert, Frank

We introduce Segmental Switching Linear Dynamic Systems (S-SLDS), which improve on standard SLDSs by explicitly incorporating duration modeling capabilities. We show that S-SLDSs can adopt arbitrary finite-sized duration models that describe data more accurately than the geometric distributions induced by standard SLDSs. We also show that we can convert an S-SLDS to an equivalent standard SLDS with sparse structure in the resulting transition matrix. This insight makes it possible to adopt existing inference and learning algorithms for the standard SLDS models to the S-SLDS framework. As a consequence, the more powerful S-SLDS model can be adopted with only modest additional effort in most cases where an SLDS model can be applied. The experimental results on honeybee dance decoding tasks demonstrate the robust inference capabilities of the proposed S-SLDS model.