Why you should include DEI-BR filters in your multilingual tests, assessments and surveys — and how cApStAn can support you

by Pisana Ferrari – cApStAn Ambassador to the Global Village

Cultures and societies are dynamic and so is awareness of social justice issues. Recent global movements such as #metoo and #blm have shone a light on many of the inequities in the world and highlighted the need to work towards a more inclusive society. Awareness of diversity, equity and inclusion (DEI) in tests, assessment, polls and surveys is also growing, becoming increasingly indispensable as it both ensures the fairness of tests and the reliability of any data collection instrument. cApStAn LQC is a language company that routinely partners with organizations who administer surveys and assessments in multiple languages and cultures. In a recent live webinar Marielle Lerner, Grace DeLee and Musab Hayatli, from our Philadelphia (US) team, explained how our support on the language and cultural dimensions of these projects has evolved to include DEI-BR filters.  

Defining the terms

As Grace pointed out in her introduction, in todays’ society DEI is a “hot topic” and, as there are many acronyms floating around, it is important to have clear definitions.

• Diversity: The practice or quality of including or involving people from a range of different social and ethnic backgrounds and of different genders or sexual orientation
• Equity: The quality of being fair and impartial
• Inclusion: The practice or policy of providing equal access to opportunities and resources for people who might otherwise be excluded or marginalized
• Bias: Prejudice in favor of, or against, a person or a group.
• Fairness: Impartial and just treatment or behavior without favoritism or discrimination

Key elements of DEI-BR

• Gender: Without a doubt, commendable effort goes into ensuring the availability of all possible options when it comes to questions about gender or race for example. Sometimes, the simplest is great: man, woman, others.
• Sexual orientation: the distinction between the public and the private spheres varies across different cultures. Yet even in those with greatest separation, there are signs of change, noticeable in the terms used in some languages. Be mindful that what one might consider inclusive can lead to a sense of exclusion.
• Race: Race categories equally vary across the globe, and some are practically meaningless in some places. While the term “Asian” in the UK might refer to those from India and Pakistan, in the US it is likely associated more with Far East countries (China, Japan …). Race is more fluid and more nuanced in places like Latin America and South Africa. And while many from North Sudan would likely be classified as black in the US, in the Middle East they are considered the same race as other Arabs.

cApStAn’s experience with DEI-BR

 cApStAn’s initial contribution to DEI-BR started with our Translatability Assessment process (TA), which aims to identify and resolve potential translation and adaptation issues in surveys and assessments well before the translation process begins. TA helps determine some of the instructions necessary for translators to convey the source text and provides the foundation for optimizing equivalence in translations and thus validity and comparability of survey or assessment results. Over time our TA process was progressively adapted to include DEI-BR evaluation by linguists, with input from subject matter experts.

How cApStAn can support you 

A DEI-BR review as part of a TA process can help identify potential issues such as: sensitive content, relevance for all segments of the target audience, portability of constructs, translatability, ambiguity, gender mainstreaming, racialized stereotypes, and potential cultural bias. We achieve this by: 

• Careful analysis of the source
• Thorough consultation with linguists and people on the ground
• Recommendations for a robust process
• Highlighting possible implicit assumptions

cApStAn DEI-BR case studies 

• Partner 1: A study about gender options in an international survey: some question categories were not culturally appropriate or had potential for legal repercussions. Cultural and legal norms were reviewed via respective countries’ census, national office of statistics, and existing national surveys or assessments.
• Partner 2: Analyzing if two survey questions mean what you think they do: a review process indicated that the two concepts asked about were difficult or impossible to distinguish in some languages. Our proposal was to make the distinction clearer by changing one or both questions, or drop one of the questions.
• Partner 3: Fairness, inclusiveness and bias-reduction in certifications: our input was to update and standardize race/ethnicity, gender, and language terms and improve the bias review process by refining guidance for communication between various actors and adding an additional bias review step.
• Partner 4: Testing subject matter, not English proficiency: in a recruitment assessment for entry level engineers there were concerns about the impact of the English language register used to test non-English speakers. Our analysis results helped item writers and reviewers focus their edits on two main factors: word and sentence length, and vocabulary choice