(Georgia Institute of Technology, 2017)
Eisenstein, Jacob
In lexicon-based classification, documents are assigned labels
by comparing the number of words that appear from two opposed
lexicons, such as positive and negative sentiment. Creating
such words lists is often easier than labeling instances,
and they can be debugged by non-experts if classification performance
is unsatisfactory. However, there is little analysis
or justification of this classification heuristic. This paper describes
a set of assumptions that can be used to derive a probabilistic
justification for lexicon-based classification, as well
as an analysis of its expected accuracy. One key assumption
behind lexicon-based classification is that all words in each
lexicon are equally predictive. This is rarely true in practice,
which is why lexicon-based approaches are usually outperformed
by supervised classifiers that learn distinct weights on
each word from labeled instances. This paper shows that it is
possible to learn such weights without labeled data, by leveraging
co-occurrence statistics across the lexicons. This offers
the best of both worlds: light supervision in the form of lexicons,
and data-driven classification with higher accuracy than
traditional word-counting heuristics.
(Georgia Institute of Technology, 2014-08-06)
Eisenstein, Jacob
Electronic social media offers new opportunities for informal communication in written language, while
at the same time, providing new datasets that allow researchers to document dialect variation from records
of natural communication among millions of individuals. The unprecedented scale of this data enables
the application of quantitative methods to automatically discover the lexical variables that distinguish the
language of geographical areas such as cities. This can be paired with the segmentation of geographical space
into dialect regions, within the context of a single joint statistical model | thus simultaneously identifying
coherent dialect regions and the words that distinguish them. Finally, a diachronic analysis reveals rapid
changes in the geographical distribution of these lexical features, suggesting that statistical analysis of social
media may offer new insights on the diffusion of lexical change.