On the interplay between survey translation and survey programming

by Dr Dorothée Behr, GESIS – Leibniz Institute for the Social Sciences

Thank you, Steve, for inviting me to join the group of researchers from all kinds of disciplines to write about what is dear to our (working) heart. Having been asked to contribute a story that involves both cApStAn and myself, the first cycle of PIAAC immediately comes to my mind. This is essentially the story of how and when we met.

Back in 2009, having just submitted my PhD (Behr, 2009), I was thrown into the biggest challenge of my young working life, namely into the Programme for the International Assessment for Adult Assessment, short PIAAC, commissioned by the OECD and managed by an international consortium led by ETS. I joined the project as a consortium member fairly late in the process, namely in January 2009. This was just two and a half months prior to a large national project manager meeting where I was supposed to train translators on the translation requirements for the background questionnaire. Not only did I have to acquaint myself with a hugely complex project in the shortest span of time, but we all in the consortium had to surmount a linguistic challenge that had previously flown under the radar. The background questionnaire included a lot of dynamic text, as we called it – little text segments that change depending on previous answers to a questionnaire (other surveys call these segments fills).

The English source questionnaire, in several sections and for many items, looked like this: “In your ^JobLastjob what proportion of your time ^DoDid you usually spend … cooperating or collaborating with co-workers?” In a nutshell, “^JobLastjob” would automatically be replaced in the survey software by “current job” or “past job”, depending on whether a respondent is currently working or not.

Similarly, “^DoDid” would be replaced by the tiny but powerful tense markers “do” and “did”, respectively. These text elements would be translated separately from the item text and then, in the real survey context, be inserted whenever “^JobLastjob” and “^DoDid” appeared in an item. This was the theory, and it worked beautifully for the English source.

However, it had not been considered that for most languages it would not be enough to translate “do” and “did” separately from an item, insert their translations into the rest of an item and – hurrah – obtain a perfect sentence in past or present tense. Instead, the reality is that most languages require different linguistic structures to express past or present tense. Thus, all of a sudden, we were faced with a linguistic challenge that had serious repercussions on the multilingual survey programming.

Given the extremely tight timelines of such large international projects, we needed to find a quick solution for the survey programming to work in all PIAAC languages and allow translators to translate meaningful text elements. This is when cApStAn created the (in)famous DTRS, superbly showing how they and their verifiers can quickly, creatively, and effectively respond to unexpected challenges. The DTRS, the Dynamic Text Rule Spreadsheet, was filled out by cApStAn verifiers for each language to contain the dynamic text rules that would make sense for a given language. Based on this, the programming could be adapted and customised for each language so that translation teams eventually received meaningful source text that they were able to translate.

For PIAAC Cycle 2, in which GESIS and cApStAn are again working together – once again as a part of an international consortium led by ETS –, dynamic text is no longer an enemy. With foresight and addressed in time, the linguistic entanglements of dynamic text can be tamed (see also Behr, D. forthcoming).  

Today, we know that large-scale international projects at the nexus between language and technology can only succeed if developers, methodologists, programmers, and linguists collaborate early on to contribute insights and knowledge from each discipline – a call that cApStAn has tirelessly repeated and brought to wider attention within the research community  (cApStAn contribution in Behr & Zabal, 2019).

Since our first collaboration in 2009, we have successfully worked together in several projects and met at numerous conferences and workshops with many exciting, lively, and fruitful discussions. In 2014, I even spent seven weeks at cApStAn in Brussels. These seven weeks strengthened our research collaboration (Behr, Dept, & Krajčeva, 2018) beyond concrete project work, and in addition allowed me to enjoy cApStAn’s great hospitality, companionship, and friendship in and out of the office. And as a side note: Le babyfoot will always be remembered fondly, all our lunch matches alongside the official world championship in soccer.

I am grateful to the cApStAn group for the numerous professional and personal meetings, collaborations, and valuable exchanges. At GESIS, we very much look forward to continuing our work with you and welcome the new translation challenges and adventures we may encounter as we continue to shape the future of survey translation.

---

References:

Behr, D. (forthcoming). Computer-assisted migration research: What we can learn from the field of software localization for source questionnaire design and translation. In Pötzschke, S., & Rinken, S. (Eds.), Migration research in a digitized world: Using innovative technology to tackle methodological challenges, IMISCOE Research Series, Springer.

Behr, D. & Zabal, A. (2019). A Meeting Report: OECD-GESIS Seminar on Translating and Adapting Instruments in Large-Scale Assessments (2018). Measurement Instruments for the Social Sciences (1): 10. doi: http://dx.doi.org/10.1186/s42409-019-0011-y.

Behr, D. (2009). Translationswissenschaft und international vergleichende Umfrageforschung: Qualitätssicherung bei Fragebogenübersetzungen als Gegenstand einer Prozessanalyse. GESIS-Schriftenreihe 2. Bonn: GESIS. http://nbn-resolving.de/urn:nbn:de:0168-ssoar-261259

Behr, D., Dept, S., Krajčeva, E. (2018). Documenting the survey translation and monitoring process. In T. P. Johnson, B.-E. Pennell, I. Stoop, & B. Dorer (Eds.), Advances in comparative survey methods: multinational, multiregional and multicultural contexts (3MC) (pp. 341-356). New York: John Wiley & Sons.

---