The topic of appropriate standards for the evaluation of scholarly digital outputs has come up in conversation at my institution (the University of Canterbury, New Zealand) recently and I’ve realised I haven’t got a ready or simple answer, usually replying that such standards are extremely important because we need to ensure scholarly digital outputs attain to the same standards as, say, monographs, but that they’re evolving. The conversations normally don’t go much further than that. This post, then, is an attempt to get my thoughts down on paper so I can point colleagues to a handy url summarising my thoughts. Much of it will merely repeat common knowledge for digital humanists, but might be of interest.
Category 1: The scholar has built the output themselves, or been a key driver in the technical design and build of it. The output has been driven and project managed by the scholar, often with external funding, including a high degree of technical input in both the design and build phases. The output is complex and/or wide-ranging (either in terms of project scope or technical complexity) and a highly innovative contribution to the field. It conforms to accepted standards in both the digital humanities and computer science. Significant and robust review milestones have been used during all phases of the project, including international feedback. Usage reports (where relevant or possible) indicate high engagement with the output from an international audience. The output has gained wide-spread recognition in both the scholarly and digital humanities communities, and perhaps broader media. It is sustainable, backed up, and controlled by good data management standards.
Category 2: The scholar has built the output themselves, or been a key driver in the technical design and build of it (in this category, because the outputs tend to be of smaller scope than Cat.1, the expectation is really that the scholar has built it themselves, or been an integral part of the team that did). It either conforms to accepted standards in both the digital humanities and computer science, or provides a conscious and challenging departure from them. The product is of limited scope, but represents an innovative contribution to the field and has gained significant recognition in either the scholarly community, digital humanities community, or the broader media. Usage reports (where relevant or possible) indicate high engagement with the output from an international audience.
Category 3: The output has been built by an external service unit or vendor with no technical input from the scholar, but the scholar has been closely involved in the design and build phases, and contributed high quality content of some form (data or text, perhaps). The product conforms to some standards in either the digital humanities or computer science, but these are loosely applied and/or incompletely implemented.