Multilingual Skills Assessment Translation Series, Episode 5: Disentangling Proficiency in required Skills from Proficiency in Language

This is the 5th and final issue in our “Multilingual Skills Assessment Translation series”. The primary objective of this series is to provide a comprehensive illustration of the complexities inherent in test translation. By doing so, we aim to underscore the immense value derived from establishing a robust linguistic quality assurance design. This episode takes a closer look at the need for analysis and adjustment of the language proficiency level to make sure the assessment is well-understood by candidates speaking the same language (but at different proficiency levels) across the globe.

Let us take again the example of our (fictional) test to measure consultants’ competencies in helping companies in damage control situations (which we will call the “Chaos Management Skills Assessment”, CMSA). This is the second assessment cycle, and part of the items can be recycled from the previous cycle. The test is administered in 12 different languages. For some of the target countries, the test is being administered in English only, although English would not be the first language of the test takers.

In this episode we will look at how to disentangle proficiency in required skills from proficiency in language, in order to ensure that the test is fair and valid for all candidates

If the level of language proficiency of a skills assessment is too high, this may hamper the correct measuring of these very skills. In our CMSA example, a skilled damage control consultant may not pass a test due to the high threshold of English proficiency that is required to complete it. A meticulous analysis of the source text allowed us to determine whether any adjustments were needed.

Here is cApStAn’s 3-step language proficiency analysis and adjustment process:

Analysis of source text tasks

The first step consisted in the analysis of the level of reading comprehension in English that was required to take the damage control skills assessment. This step included:

• A “difficulty analysis” of the test tasks was conducted with the aid of Text Inspector, a professional language analysis tool benchmarked to the CEFR, Common European Framework of Reference for Languages levels, see image 1 below.
• A compilation of a list of technical terms that are essential to measure the skills that are being assessed.
• The exclusion of these technical terms from the analysis.

Fixing a reading comprehension threshold

The next step was to determine a reading comprehension threshold that would strike a good balance between accessibility for a global audience and robust test items. Based on the results of step 1 this was fixed at B1, see image 2 below.

Review of the task descriptions

A review of the tasks was conducted to stay as close as possible to the B1 threshold. This step included:

• Replacing difficult words/phrases with easier synonyms. The English Vocabulary Profile Online website is an excellent resource for checking the CEFR level of candidate replacements.
• All replacements were included in a term base, to facilitate systematic replacements also in future assessments.
• Breaking up long sentence into shorter ones, to improve readability (this required some rewording).
• Adding a number of non-technical terms to the list of known words after inclusion of an added introductory explanation at the beginning.

You can check the previous 4 episodes here:

Episode 1 – Unlocking the power of existing translations from a previous test cycle

Episode 2 – Unveiling the Secrets of Pre-Translation Preparations

Episode 3 – Empowering Translators with Translation Technology and Support

Episode 4 – Ensuring fairness, reliability, and validity in multilingual tests after translation

………………………………………………………………………………………………………………………………………………………………………………………………

Want to try this out on your materials? Select some sample items or sample questions, and request a free pilot analysis and adjustment of language proficiency at hermes@capstan.be

And, if you’d like to learn about how we can help you with your test localization projects, do contact us here, and we’ll get back to you as soon as we can.