Training Methods for Improving Annotator Accuracy Perception in Natural Language Processing Projects
Author(s)
Turner, Haydn P.
Advisor(s)
Editor(s)
Collections
Supplementary to:
Permanent Link
Abstract
The volume of biomedical research publications produced every year is growing rapidly, leading to an need for improved biocuration methods that enable non-specialist annotators to contribute to natural language processing workstreams that enable more efficient use of large volumes of text data. In this research, we assess 38 student annotator’s opinions about various training methods, including recursive team-based learning sessions and written annotation guides as well as motivational aids, such as performance awards and the perceived value of the project. We also introduce a novel, hierarchical annotation framework, titled a ‘Relationship tree’, used to aid biomedical text annotation and analyze annotator’s sentiments regarding the usefulness of the tool. Our results indicate that annotators have a strong preference for using the relationship trees to annotate texts, with annotators reporting that the felt the tool made them make fewer errors (p<0.05) and improved accuracy (p<0.01). Our results indicate that relationship trees may be a helpful tool to aid annotator understanding of complex texts and may improve annotation accuracy.
Sponsor
Date
Extent
Resource Type
Text
Resource Subtype
Undergraduate Research Option Thesis