How can you create culturally fair content for tests and surveys?
by Marielle Lerner – Localization Specialist
Lost in adaptation
If you are involved with the production of international surveys or assessments, you have likely observed the “lost in translation” phenomenon in action. On the other hand, you may have never heard the expression “lost in adaptation”, something that we at cApStAn are always thinking about (though this Lost in Adaptation is not what we have in mind).
By “adaptation” we mean what is needed, besides language choice, to make a survey or testing instrument understandable, equivalent, and appropriate for the local context where it is going to be administered. This cross-cultural suitability is an important consideration, also when translation is not needed.
Say that a test is prepared for US English, and it will then be used in Ireland, Singapore, Australia, and Nigeria; the test may be administered in English in each of these countries, but the English source must be adapted to produce a version that is suitable for each country. This includes some fairly straightforward considerations such as currency or units of measurement, as well as more complex and subtle matters of cultural suitability and sensitivity.
It is not so simple as using kilometers for an audience who is not familiar with miles; as noted by Padilla & Medina (1996), “[i]f respondents […] are uncomfortable because of cultural or linguistic differences, this may cause a bias in the responses” (Mills 2016).
Cross-cultural suitability is essential not only for the functioning of, but for the fairness of any instrument of measurement. The American Psychological Association ethics code includes a stipulation for cultural sensitivity in assessments: “… assessment instruments are required to be culturally sensitive, and reliability and validity should be established in the group for which the measure is intended (American Psychological Association, 2002)” (Mills 2016).
Adaptations needed for cross-cultural suitability
There are many contexts that require adaptation, some more obvious than others. For example, if a test stimulus describes a greeting between people, an item writer in the US may use a handshake. But in other cultures, the usual practice could be a kiss or bow; this can be adapted for each culture.
The University of Michigan’s Cross-Cultural Survey Guidelines cite as examples of survey components likely to require adaptation “a population’s familiarity with certain visual representations (thermometers, ladders, scales using faces (Kunin, 1955)); and a wide range of culturally anchored conventions related to visual presentation, including color symbolism, representational preferences, and conventions of emphasis” (Mohler et al. 2016).
The interpretation of a particular behavior is also culturally defined. For example, an evaluation asking employees about workplace behavior may measure something different depending on the local (work) culture. As noted by Sarah Mills in an article for the APA publication The Score, “[…] certain behaviors (e.g., speaking softly) can have unique meanings (e.g., timid vs. respectful) in different cultures” (Mills 2016).
Adapting race or ethnicity
Public opinion surveys commonly include demographic questions on race or ethnicity; just as these classifications are debated and defined differently in fields like anthropology and biology, their linguistic complexity should not be underestimated. Cultural sensitivity in this domain involves knowing which categories are relevant or not, as well as which wording is appropriate or not.
Some category names may simply be meaningless when applied to different countries; “African American” for example is specific to the history of the United States. There are other categories that may exist in multiple countries and even be appropriate in multiple countries, but have different definitions; depending on the point of reference, “Indian” may be understood as referring to indigenous peoples of the Americas (though in the US now usually expressed as “American Indian”), or natives or descendants of the country of India.
Race or ethnicity categories may be considered acceptable in one country and offensive in another. For example, “Coloured” is a term currently used in demographic questions in South Africa, but it would be highly inappropriate to use in the United States (Tamir & Budiman 2019).
Just as race terminology within one country evolves and certain terms become inappropriate with time, a particular term may be appropriate in different countries at different times. Simply the inclusion of a question about race is potentially inappropriate in some countries.
Adapting gender identity
In many countries, it is becoming more and more accepted to expand the classification of gender identity beyond the binary categories of “man and woman” or “male and female”. The Pew Research Center reports that “roughly one-in-five U.S. adults know someone who uses a gender neutral pronoun such as “they” instead of “he” or “she,” and 17 states and the District of Columbia have adapted to this evolution in gender identity by adding a nonbinary option to driver’s licenses” (Amaya 2020).
With many new gender labels in flux and gaining traction in language and society, it is a challenge to come up with a demographic gender question even for one target population.
One approach to attain inclusivity is to try to include all possible options; for example, in 2014 Facebook began allowing their users to indicate their gender as “male”, “female”, or “custom”, the latter choice leading to a drop-down menu of 56 options at the time (Oremus 2014).
Another approach, perhaps more practical for questionnaires, is to provide a third “other” option; this approach was taken by Pew Research Center, which revised their standard gender question in 2020 to read “Do you describe yourself as a man, a woman, or in some other way?”, with a text box for the third option (Amaya 2020).
How to ensure cultural sensitivity
The quality assurance process at cApStAn always involves professional linguists who are members of each target population of the survey or assessment being prepared. We guide them to evaluate the suitability of the master or source version for their local economy, education system, social context, cultural appropriateness, and any other relevant parameters. The end result of this QA process includes guidelines for adaptations for each regional version of an instrument, as well as recommendations for modifications to the master version to avoid any culturally inappropriate or insensitive elements.
If you’d like to learn more about how to adapt your survey or assessment to be culturally suitable for multiple populations, just send us your queries or request a free pilot project in the form below, and we’ll get back to you as soon as we can!
References
Oremus, W. (2014, Feb 13). Here Are All the Different Genders You Can Be on Facebook. Slate.