Discussing accessibility, fairness and equity of tests at E-ATP 2021 as we emerge from COVID-19

by Pisana Ferrari – cApStAn Ambassador to the Global Village

This year’s edition of the E-ATP conference was a virtual yet vibrant event, where attendees discussed all the facets of disruption in the testing industry due to COVID-19. We attended E-ATP in Dublin (2015), Lisbon (2016), Noordwijk (2017), Athens (2018) and Madrid (2019). cApStAn’s involvement in this forum comes naturally: since 2000, cApStAn has contributed to shape, streamline, establish and promote good practices in translation and adaptation of assessment instruments. We were delighted to be a Silver Sponsor of yet another excellent edition. Discussions over the three-day event focussed mainly on the challenges and opportunities to the testing industry brought on by the pandemic. These include the shift to technology-based testing and remote proctoring; catering to new audiences; adapting technology and test questions; safeguarding data privacy; and providing flexibility in delivery. Accessibility, fairness and equity of tests were at the forefront of many sessions, including that of our own Steve Dept on item bias in assessment content (see below). Many speakers pointed out the need to find a consensus on how these terms are being used and the importance of sharing measures taken, and best practices, across the industry. We have picked out some salient topics for this summary.

Accessibility

Ensuring equal access to all candidates, whoever they are, and wherever they are, is one of the key aspects of fairness and equity. Ensuring accessibility is an extremely complex task, made all the more difficult by the increase in technology-based testing. It involves adaptations in the test questions, in the technology, and in delivery mode. One needs to take into account the diversity of test takers and devices used: screen colour and contrast, to allow for low vision or colour-blind candidates; keyboard usability issues; screen reader support; transcripts, captions and alternative video for audio content, for the hearing impaired; and the need for responsive design, to cater for delivery across different devices. It was interesting to hear from Marc Vanhasselt (Prometric), how, over the past ten years, the concept of accessibility has evolved to accommodate not only the disabled but also persons in a  stage of their life or career where they could be at a disadvantage when taking a standardized exam, e. g., pregnant or nursing persons, people with ADHD and similar conditions, and cognitive disabilities that are easily categorized. “These trends”, he said, “have important implications for how we act in terms of the design and execution of our assessment programs. From designing new registration flows and operational procedures to how we design, build and present content.”

Fairness and equity

“Accessibility” is an important part of “fairness and equity” but there is more to it than that. A “fair” and “equitable” test should not advantage or disadvantage individuals because of characteristics that are irrelevant to the intended construct. The language used in assessments may contain hidden social markers that can put some socio-demographic groups at a disadvantage and thus introduce test item bias. Our own Steve Dept addressed this issue at a “coffee conversation” on Day 3. In this lively and well attended session he explained how tests contain far more stereotypes than we think. The examples he gave showed how much it takes to scrutinise the language of assessments for bias, and from how many different perspectives. More diversity in test writer pools and SME panels, with representatives of different communities and cultural backgrounds, would ensure that (at least part of the) problematic expressions are flagged and the necessary adaptations made – or not. Whether an assessment is monolingual or multilingual, it is important to identify places where the representation of what is written can vary across different demographics, he said (see our report of Steve’s talk at this link). The National Board of Medical Examiners, NBME, shared the steps it is taking to ensure fairness and equity in their tests. Top on its list, it recommends keeping “fairness” in mind right upfront, when drafting and editing test items. The importance of fairness and equity was raised also in a session with John Kleeman (Learnosity), which focussed on getting the language of assessments right and recommends: the use of gender-neutral terms (sales agent instead of salesman), of standard formal language, avoiding any references to race and ethnicity, idioms and colloquialisms, obscure language and ambiguous acronyms.

Are fairness and equity affected by remote delivery?

The European Personnel Selection Office, EPSO, is responsible for selecting staff to work at the institutions and agencies of the European Union. One of their competitions was interrupted by COVID-19 in March 2020. After much deliberation it was decided to resume the test remotely. This provided a real-life opportunity to compare the performance of candidates on site vs remote in the same test. Results indicate that the equality of treatment between remote and presential assessments was preserved. This was a small set of students, said the presenter, Stephan Vanderkeven (EPSO), so it is not possible to extrapolate the data on a more global scale, but it is an important aspect that needs to be explored further.

The importance of creating an inclusive company culture

DEI in assessments starts with creating an inclusive company culture. How is this done? During a debate on “Fair and Equitable Assessments for All” the presenters shared measures taken to improve company culture. These include embedding conversations about equity and bias in all aspects; hiring practices that attract an array of backgrounds; putting in place inclusive programs and employee experiences so that all employees feel supported; and requiring ongoing training for all employees. At the same time, as rightly stated by Ali Shalfrooshan, Head of International Assessment R&D at PSI, it is important to understand an individual’s capacity to behave inclusively and respectfully with all people, regardless of how similar or different they are to them. One of his slides read “Diversity is being invited to a party. Inclusion is being invited to dance. Yes.

Why company assessment literacy matters

As a non-expert myself I found the ATP’s suggestion to introduce an “Assessment Literacy Course” for non-psychometricians inside assessment companies, e.g., in marketing, sales, admin, etc., particularly interesting. The approach could be modular, with different levels of difficulty, and allow for the possibility of introducing badges and certificates. Its objective is that participants achieve “a conversational understanding of measurement”. I believe this would be extremely useful not only for the knowledge itself but also in terms on increased interest and engagement in the company’s core business, as well as in facilitating conversations with colleagues (people tend to work in silos).

Future trends

Assessments are continuously evolving and the pandemic has accelerated the process. Challenges to the testing industry include the need to prepare for a future where hybrid administration is the new normal, where test questions go beyond mere recall (as facts can be checked on Google) and where candidates are increasingly geographically dispersed, said the presenters at the session on “Equal Access to Testing NOW” (Questionmark).

Future challenges

In a sobering closing keynote, on Day 3, Marten Roorda, Chief Measurement and Learning Officer at Riiid Labs, warned that the industry will need to address threats from Big Tech, increasingly entering the education field, as well as public opinion hostility towards assessments. Some institutions are dropping admission tests altogether (SAT, ACT….) and the language certification business is also being disrupted (TOEFL, IELTS…): over 3,000 academic institutions now accept a €49 online Duolingo language test that one can take at home. Alternative pathways to validate skills, assess or hire people are coming our way. Roorda’s “Ten Commandments” for the industry are: integrate/measurement/learning/navigation, automate repetitive tasks and lower costs, take a holistic approach, create a great experience for candidates by making tests attractive and engaging, aim for interoperability (plug-and-play), live by values: equity, efficacy and transparency, try “testless testing” (infer from other information you have in the company), innovate, and, “if you can’t beat, them join them” (build partnerships). Interesting testing times lie ahead.

Cover photo

Clockwise from top left: Steve Dept, cApStAn, E-ATP Sponsors, PSI slide on diversity, John Kleeman tips on avoiding bias/stereotypes; Pisana Ferrari, cApStan, John Kleeman slide on inclusivity, Gavin Cooney on multilingual assessments, Devasmita Ghosh, cApStAn, John Kleeman on proctoring flexibility and fair tests, opening keynote speaker Priya Lakhani, John Kleeman on future trends.