A Tale of Joint Resolution of Seemingly Insuperable Challenges

by Dr Paulína Koršňáková in collaboration with Elica Krajceva from cApStAn

This specific international project involved a questionnaire survey, designed to collect valuable data from teachers and school principals. In order to serve the survey’s aims (informing policy review and development in the participating countries), comparability across participating educational systems was a necessity.

This implied that translated versions were expected:

(i) to convey the meaning of the international source version of the questionnaires;

(ii) to maintain the link to questionnaires formerly used at a national level;

(iii) to maintain any links between the teachers’ and the school principals’ questionnaires; and (iv) as a bonus challenge, the questionnaires had to take into account differences among the multiple levels in which schools are organized.

Note that, while IEA saw to it that a healthy portion of the questions were the same across all administered teacher questionnaires, a non-negligible number of adaptations had to be implemented, too. Neither the extent of the questionnaires (in number of words), nor the scope of required changes were anticipated or budgeted. It should come as no surprise that the quality assurance procedures were conceived as a check point focussing on key aspects of the instruments, aligning the different versions of questionnaires across school levels and across language versions.

Usually, one needs to pay attention to the size of the instruments, so respondents can complete the questionnaire swiftly and do not drop out. Thankfully, we found out that teachers are very kind respondents who like to share their opinion, so they coped well with the inordinate length of the questionnaires.

The less time you have, the more important is to get everything right the first time. Well, last minute additions to the instruments as well as late discoveries of ambiguous wording made it very complicated.

In the end, every target was attained. This was only possible because we all played with open cards, communicating problems as they surfaced, discussing the most suitable way out.

While IEA was trying to get organised with the post-last-minute notices and changes to the international source versions of the questionnaires all the way to layout verification stage, the verification team spared no effort to disentangle the wondrous knot of links we had in this project: links between the principal and teacher questionnaires, between the different school level options, between the past survey, the field test and the main data collection instruments.

The challenge was also high for the verifiers: cApStAn team went the proverbial extra mile to provide seemingly simple instructions for a highly complex task:

– The necessary background information shared with linguists was reduced to what was required to get the gist of the exercise.

– All checks that project managers could perform after the verifiers’ work were removed from the instruction document.

– And verifiers stretched their capacity to work their mind around all the possible effects a tiny change would trigger. Almost like the butterfly effect. Their work was heroic and often beyond the call of duty. The lists of checks that were in place would make any connoisseur of this type of operation dizzy!

To add delight to the exercise, IEA and cApStAn were exchanging pdf documents in all scripts and directions: at the time there was no online system that could help us to deal with the instruments in one go (or at least to garner real-time information on the state of art). We were sentenced to deal with a flurry of pdf, rtf and/or Word files that did not look alike on all operating systems.

When looked at from the outside and with the distance of time, rather than insuperable challenges we see the overall quality and consistency achieved in spite of the mission impossible. Maybe it also takes a grain of folly, but good old stories deserve some. Right?

P.S. Of course, there were some lessons learned and there would be some changes to the approach used at the time. The major one, and actually the only one that we could be in charge of, would be to make two or three waves of checks rather than to attempt repeating the tsunami gently outlined above.