arrowCIRSS Home arrow Publications arrow Publication Detail

A research design for measuring variation in database curators' annotations through prospective randomized controlled studies

Full APA Reference

MacMullen, W. J. (2007). A research design for measuring variation in database curators' annotations through prospective randomized controlled studies. Poster session presented at The 3rd International Digital Curation Conference, Washington DC.

Publication Abstract

This project addresses the need for standardized research methodologies for the investigation of variation in workflows and outcomes used and produced by curators of digital repositories when performing standardized tasks, such as indexing, metadata generation, and ontology term assignment. Research on variation in curators’ outcomes (called here ‘annotations’) is important for several reasons, including: to understand the nature and extent of variation in curators’ annotations; to measure internal consistency of curators’ work, and to develop related quality metrics; and to learn from best practices to assist in the education and training of new curators. Standardized evaluation methodologies may also allow for the creation of benchmarking metrics, enabling cross-resource comparisons of such quality facets as consistency, reliability, specificity, completeness, and validity [1]. The experimental design was previously used to investigate variation in human curators’ Gene Ontology (GO) annotations in model organism databases [2], but is here described such that it is generally applicable to many contexts where multiple curators are performing curation or annotation tasks with documents or other forms of structured data and information. The research design consists of prospective randomized controlled studies, and includes discussions of document corpus construction, task formulation, documentand group assignment, resulting data and analysis, and contextual considerations.

1. MacMullen, W.J.: Facets and measures of Gene Ontology annotation quality in model organism databases. In Proc. of the 69th ASIS&T Annual Meeting, Vol. 43 (2006) 2. MacMullen, W.J.: Contextual Analysis of Variation and Quality in Human-curated Gene Ontology Annotations. Docto