Redefining the concept of “master version” in test item development to address the “Anglo-Saxon legacy” and other translation effects

by Pisana Ferrari – cApStAn Ambassador to the Global Village

In the past two decades, new standards have been defined for the adaptation of assessment instruments into multiple languages. Sophisticated translation designs have become the norm in international large scale assessments (ILSAs). However, in item development, the “Anglo-Saxon legacy” still prevails: the most common practice is to produce a well-crafted English source version, to pilot it, validate it, and only then to adapt it into multiple languages. Evidence of recurring translation issues in ILSAs suggests that this practice may not be optimal. Tests earmarked for translation often contain idioms and colloquialisms and are replete with ambiguities that cannot be rendered in target languages. These are features known to affect psychometric properties of items and, therefore, have an impact on functional equivalence and comparability across languages.

At the recent International International Test Commission Conference (ITC) in Montreal, Steve Dept, cApStAn’s CEO, made the case for redefining the concept of “master version”. The key word here is “upstream”, i.e. to start from a “more translatable” source version so as to mitigate language/item effects further down the line. The suggestion is to follow a standardised adaptation process with a simpler, more international and culturally balanced English source version, reviewed by linguists and subject matter experts from different cultures. These reviews generate a comprehensive corpus of adaptation notes, on which (national) translation teams can rely to produce target versions, including into idiomatic English for use in English-speaking regions.

Looking at the broader picture, beyond merely linguistic issues, upstream work can help to address other, more general, translation problems known to be found in ILSAs. We at cApStAn have a broad experience of what we call “Translatability assessment”, a process that takes place before the master version of test items is finalized. A panel of experienced linguists covering a variety of target language groups reviews the draft source. The linguists draft translations in their respective languages with a view to identifying potential translation and adaptation hurdles. They map each issue to a set of translatability categories that allow for standardized analysis (these include, inter alia, checking for inconsistencies, redundancy, internal logic, agreement within segments, register, double-barrelled questions, potential cultural issues, etc.). In some cases, the linguists may draft a translation note, in others they may propose alternative wording for the source items, in order to circumvent the issues identified. As a next step, a senior cApStAn linguist analyzes and collates the feedback from the panel, producing a Translatability Report for the test creator. A Translatability Assessment can address many different types of issues related to translation as well as question design.

To conclude, we believe that, in ILSAs, a culturally balanced, unambiguous, master version is the foundation for linguistic and semantic equivalence across language versions, which in turn will maximize comparability of test results.