02.10.2023

“AI and DEI Take Centre Stage at E-ATP 2023: Insights from cApStAn LQC”

Published in:  Artificial intelligence, cApStAn updates, Conferences, DEI-BR, Test translation and localisation, Tests and Assessments, Translation industry

This year the E-ATP 2023 Annual Conference was held in Vienna from September 20 to 22. At cApStAn, we have never missed the E-ATP conferences since 2015 in Dublin, and we are very proud to be one of the Silver Sponsors of this year’s edition. We were represented in Vienna by Founding Partner and Head of Strategic Partnerships Steve Dept, along with Business Development and Marketing Lead Devasmita Ghosh, and Project Manager Nela Manojlovic. The entire ecosystem of test owners, test sponsors, platform providers, test forensics and assessment research is represented at E-ATP: the buyer side, the vendor side, the stakeholders, the innovators, the industry leaders, the start-ups; everyone is there, and everyone has a voice. This is where anyone involved in the testing industry can learn from peers, share experiences, observe trends, and network.

At the conference, it came as no surprise that the two hottest topics were (i) the use of artificial intelligence (AI) in assessment and (ii) diversity, equity and inclusion (DEI) in assessment. As always, cApStAn’s contribution was to share findings and experiences at the intersection of these topics with language and culture.

For his solo presentation, titled “Three Attitudes towards AI in Test Localisation” Steve explained that multilingualism is integral to Europe’s identity and a political priority, and this is reflected in the design and administration of assessments. Many countries have multiple official languages, leading to a need for extensive translation. Fortunately, technology allows for faster localisation without exponential cost increases. He noted that neural machine translation (NMT) output still requires focused human intervention, and that Large Language Models (LLM) can generate content with minimal post-editing  but thorough subject matter expert validation.

Steve outlined three different attitudes towards using AI in test translation:

  • Gradually automate localization tasks with AI and NMT to reduce reliance on human work.
  • Strive to regulate or even ban AI use in specific contexts to protect humanity.
  • Augment human capacity with AI and NMT, identifying points where human expertise adds most value.

Concrete cases in the European context were presented:

  • An HR and Talent Management entry-level screening exam, with maximum automation, as time to market and cost-effectiveness were the top priorities in this mission.
  • A standardised technical communications certification test, where several layers of human quality assurance were added and only tried and tested technology was used, as reliability and validity were the top priorities in this mission.
  • A leadership assessment that also measures non-cognitive skills and generates reports and dashboards, in which productivity was as important as keeping the human expert in the driver’s seat, so the APIs and functionalities were added progressively.

In an interactive session with Isabelle Gonthier, Chief Assessment Officer at PSI Services LLC, and Ada Woo, VP Innovative Learning Sciences at Ascend Learning, titled “How to Articulate the Assessment Life Cycle around Diversity, Equity and Inclusion” Steve examined DEI across the entire life cycle of an assessment, from test development to use cases, from language to sociocultural markers. DEI is not a checklist but shift in mindset, and it plays a crucial role in ensuring fairness in assessments. Potential sources of bias in exam content include the use of content-irrelevant language, cultural nuances, and regional differences. This session explored the various stages of the assessment life cycle to identify opportunities for greater equity in exam creation, covering subject matter expert selection, item writing guidelines, diverse focus groups for item review processes, language, and localization requirements, as well as post-administration analyses.

Photo, clockwise from top left: Paul Edelblut, Vantage Labs, with Steve Dept and Devasmita Ghosh; Isabelle Gonthier – PSI Services, and Steve; Joel Rapp, CEO of NITE, and his spouse, Ronit Rapp, with Steve; Ed Tech Market data; Steve’s solo presentation; Devasmita Ghosh and Nela Manojlovic at the cApStAn booth; Steve’s solo presentation; Steve with Isabelle Gonthier (PSI Services) and Ada Woo (Ascend Learning); Panel discussion on AI in Tests & Assessments with Neil Wilkinson – Pearson Vue, Alina von Davier – Duolingo, John Kleeman – Learnosity / Questionmark, Paul Edelblut – Vantage Labs, Ruben Garcia – Certiverse. Center: Steve with Ulrich Schulze-Althoff, Managing Director of Meister Cody GmbH, which is now part of the Klett Group.