Starting with submissions to DH2024, a new set of evaluation criteria will be used for reviewing the submissions to the Digital Humanities Conference. These changes come as part of the progressive modifications of how the conference is set up that were described earlier this year in the post The Only Constant is Change published on the ADHO website. After an open consultation period in July and the resulting revisions, the new criteria were published in October 2023 and linked from the Call for Papers for DH2024.
The purpose of this post is to summarize the major changes that were made and explain what we hope to improve or accomplish with these changes. Three areas are primarily concerned: (1) how the criteria are maintained; (2) how they are formulated and presented; (3) which criteria are used.
Regarding how the criteria are maintained, the main change is that ADHO intends to maintain, update and document the criteria going forward, rather than expecting that work from successive programme committees. This is intended to provide greater transparency and predictability over time, for authors and reviewers alike.
There are several ways in which the criteria are formulated and presented differently now. First, in the review form in Conftool, the numerical scores are each accompanied by a short descriptive phrase that is adapted to each question. This is meant to create more consistency in how the scores are interpreted by the members of our global and multidisciplinary community. Second, each criterion has its own comment field in the review form. This is to help reviewers better align their numerical scores with their comments. And third, the questions are phrased to encourage reviewers to write constructive reviews that help improve the research.
Finally, with respect to the criteria themselves, two changes among many are the most notable. The first is the removal of the criterion related to the formal aspects of a submission, such as abstract length and citation style. The former is an eligibility criterion, and the latter will be enforced when accepted abstracts are collected. The second is the inclusion of a new criterion regarding how a submission supports the diversity of the conference. This criterion is intended to ensure that the conference can truly showcase the breadth and depth of the topics, approaches, perspectives and contributions from among the wide range of practices and across the global community we call the digital humanities, in the spirit of the “big tent” metaphor.
This last change is the most visible and the one likely to raise the most questions. Therefore, a few words of contextualization might be useful. First of all, it is worth citing it in full: “Does the submission support diversity, in the sense that it describes work that increases the range of topics, approaches and perspectives presented at the DH conference, and/or does it give adequate recognition to a broad range of relevant scholarly work, including by members of disadvantaged or under-represented groups?”
This criterion is intended to encourage reviewers to read the submissions with a question in mind that, just like the other criteria, is an aspect of the quality of the research presented at the conference. Also, it is worth keeping in mind that the quality of the conference as a whole is greater than the sum of the quality of the individual papers.
In any case, there are many ways in which a submission can promote diversity. To name just a few: a submission may address topics, use materials or employ methods and approaches that are rarely used elsewhere in the digital humanities. It may approach familiar issues or materials from a fresh perspective inspired by disciplines other than the humanities and computer science, or by contexts and concerns other than those of Western-centric academic habits. And it can do so by considering scholarly publications and/or artistic practices that emerge from groups or communities that are overlooked or under-represented in mainstream scholarly work. There are certainly more ways in which as submission can promote diversity, and with this in mind, we invite reviewers to be generous in how they approach this criterion.
We recognize that this criterion may not be easy to navigate at times, as it requires a good knowledge or intuition of what is and is not mainstream, conventional, well-represented materials, approaches and perspectives in the digital humanities community, or what relevant scholarly work has been performed by under-represented groups within our community or elsewhere. A very similar challenge holds true, however, for other well-established criteria, such as those related to the current state of knowledge and best practices, for example. Therefore, we also include two questions regarding reviewer confidence, one about familiarity with the topic of the submission, and another about familiarity with the Digital Humanities conference. These are not used to automatically weigh reviews or scores, but they do provide context to the programme committee where needed.
We hope that the aims associated with the new review criteria will be realized. However, we also intend to conduct an evaluation of the reviewing practices and the review process for DH2024, in order to refine and improve the criteria as needed. All authors and reviewers are invited to send their remarks, comments, questions and concerns regarding the criteria to ADHO at adho@adho.org.