Using linguistic quality evaluation to drive revenue

by Pisana Ferrari – cApStAn Ambassador to the Global Village

Webinar hosted by CITLoB with Steve Dept and Sandeep Nulkar, 7.1.2021

“Quality is the most used word in our industry but it is one thing to say you provide quality translation, it’s another to say you can actually measure it”.

These opening words by Sandeep Nulkar, President of CITLoB, the Confederation of Interpreting Translation and Localisation Businesses in India, set the tone for a recent webinar on translation quality evaluation (LQE), featuring cApStAn’s CEO Steve Dept as guest speaker and Sudheen M, President of Crystal Hues, as moderator. In this information-packed and well attended one-hour session Steve Dept answered Sandeep’s pointed and very focussed questions, which made for a lively conversation. The audience was also very receptive, and the Q&A session was excellent too.

How do you define translation quality? How can it be measured? What does it take to put in place an LQE methodology? What is the added value of LQE? How much does it cost? How can it drive revenue? Can it be integrated in a workflow that is automated and on the Cloud? What kind of organisations/companies already use LQE? Is the market, and in particular the Indian market, ready for such sophisticated services? These are some of the issues that were discussed during the webinar, where Steve Dept shared his 20+ year experience with cApStAn in linguistic quality assurance, working with some of the world’s most prestigious organisations and companies.

Steve is confident that the shift in mindset is within reach and that factoring robust translation quality evaluation processes into localisation workflows is neither reserved to large LSPs nor a huge investment. If we create awareness, we raise the bar, he says, and higher standards will help differentiate the most demanding among us. The Indian language industry has enormous potential, and the last stretch towards measurable excellence is within reach of many LSPs. Sandeep agrees that defining quality and putting the methodology in place is where the conversation now needs to be moving.

SN. What exactly is linguistic quality evaluation?

SD. There is a lot of leeway in describing linguistic quality evaluation. One may have heard one’s clients (or critics) say things like “It doesn’t sound right to me”, “it’s a little awkward”, or “you don’t really say that in my language”. That is not a quality assessment. You need to have a quality assessment methodology, you want to be able to measure and you want to present a score card, a metric, to your client. In order to do this, you need your client’s help and that is the most important starting point. Measuring quality is measuring to what extent the translation provided complies with the client specifications.

If that includes style guides, a term base, you already have one objective way, i.e., this translation complies with the term base, it complies with the style guides. And if it doesn’t, one point less here, two points less there, and you have a metric. That is objective, no one will challenge that.

Quality evaluation will consist in defining what you need to look at to give an assessment of the quality of the translation and you are going to give a score, weight and have a sort of score card.

In translation standards, e.g., the ISO 17100:2015, revision is a given. If someone translates, another person revises, and then you deliver to the client. Revision includes a number of factors, and your revisor must have at least the same competencies as the translator, and more.

Then comes the market: yes, it’s all very nice but my costs, and my time to market? What do I do about it? The first thing really is communication with the client; if you ask him to help you define the specifications you can make sure the translation meets all the requirements that you have requested. But then the requirements can be a bit vague, e.g. I have a test and I want it to be just as easy or just as difficult in Hindi. To assess that, you need to be a teacher, you need to get Hindi students to take the test, you need to look at their answers to see if they have understood it the same way, and even the best translator in the world cannot guarantee that the test is going to work the same way. The way to go is to define the specifications and find the best way to meet those specifications.

If there is not enough time or money to do a full LQE then at least propose spot checks, look together at the source text and pick out, say, fifteen things that are crucial in the source text and then, in the target, verify if those fifteen are correctly translated. If they are, we stop there; if they are not you need to go further, and propose a methodology. A methodology will give you more credibility with your client and will drive revenue, too.

If you study the legacy translations, if you correct and implement the style guides that the client provides you or if you make a style guide for your client because he doesn’t have one, and you enter that in your software, in your workflows, in your automated quality assurance tools, that has a cost, but you can carry it over to your client. You can say to the client that it is going to take you five days to put all that in place, but with those five days and that cost, you can have a score card that measures the extent to which the translation meets the client’s requirements. That adds to the credibility and the work is paid by the client. Not all clients are ready to pay that yet but it is really coming on, even in India.

Ultimately translation quality evaluation is about moving away from subjectivity as much as possible and having a framework that allows the evaluation to be replicated, so that it no longer depends on your evaluator.

Because of this one needs to train evaluators to only check compliance with requirements, not to make improvements. It’s hard for evaluators, it is a difficult concept for them to integrate. A translation can always be improved, made differently, fine-tuned, of course, but if that is not in the requirements of the client that is not what we mean today by quality evaluation.

SN. Can you give us an example of who has used linguistic quality evaluation and how did they benefit?

SD. The first examples that would come to mind are large international institutions, like the EU and the World Bank, or NATO. They have very sensitive documents, they would typically outsource a translation quality audit. If large institutions publish materials that are going to inform policies and these are poorly translated, they will lose credibility. They would find it important to go that extra mile.

There are also the pollsters, who collect data in various countries and in various languages and want to be sure that the same questions are being asked, that the data is reliable and that the results are comparable. They will also go the extra step.

The pharmaceutical industry needs quality assessment for translations of clinical surveys and informed consents as these have legal as well as health issues connected to them. For example, you may not report depression in the same way in India as you do in America. You can even have different ways of describing symptoms, so that the questions really need to be adapted. The pharmaceutical industry will use procedures like multiple back translations or multiple reviews. They typically foresee LQE it in their workflows and in their project management from the very start.

Who doesn’t need quality assessment? User guides for vacuum cleaners, for example, are usually poorly translated, but the vacuum cleaner manufacturer knows he is not going to sell more vacuum cleaners with a perfectly translated user guide. So, he will choose not to invest in LQE.

Ultimately it is a matter of functional equivalence: What is QA going to be used for? What sort of impact should it have? Is it linked to credibility?

SN. Would you say that linguistic quality evaluation is directly related to revenue generation? Or is it more of an indirect or collateral benefit?

SD. Excellent question. It is more direct than the majority of Indian LSPs I have talked to seem to think, and I can try to explain how. First of all, if you can demonstrate to your clients, present and future, that you have a methodology in place you will be able to ask for a higher price, even marginally higher, but if you have that process in place and you are used to implementing it, that marginal increase is revenue for you.

Secondly, there is the part that is “on demand”. If you have a procedure in place and the client needs to have an external revision you don’t need to have internal resources for that, all you need is a good freelance linguist for each of the languages that you work in. You can get the client to pay for that additional work.

More importantly, in terms of direct revenue, I see that in India there are companies who understand the value of LQE, and they may even be in the language service industry, but don’t have the time to evaluate their freelance pool themselves. They may want to send a sample of translation or translation memories to an organisation that can rapidly send back a score card and an evaluation. That is direct revenue as well. It’s another service that an LSP can offer.

At cApStAn we started with LQE 20 years ago and profiled ourselves as a linguistic quality agency in Europe. Some clients would ask us to translate, too, given we had the style guides, the term bases and the expertise. Of course, we would not refuse this but it is a different pool of linguists and a different workflow. Often clients use their established vendors for the translation and then come to us to verify the quality of a selection of translations, and that is a steady flow, every month we have something coming in, we verify and give back a score card.

SN. How would you differentiate between LQE and term that anyone can understand which is “bilingual review”?

SD. Let’s have a look at the three main standard definitions.

• Revision is the word we at cApStAn use to talk about “bilingual review”, comparing source and target segment by segment, for accuracy, for localisation errors, stylistic errors, etc., but mainly it is about comparing source and target.
• Proofreading: this term refers to the monolingual activity of cleaning up a translated text by removing residual grammar errors, spelling errors, or punctuation – but you are not allowed to make any content changes.
• Editing is also a monolingual activity: you edit a text that is in a given language to increase its impact, it’s local colour, maybe, but you are not comparing segments against the source.

The simple bilingual revision is checking source versus target for accuracy. Traditionally this was done simply through back translation but people are now starting to see that this is totally obsolete. If you took a text in English translated into Malayalam and translated it back into English and it did not sound right it would be considered bad; on the other hand a literal translation would probably score well! And we all know that literal translations don’t work well, they are not good. So, a revision done in this way is not a very good measuring instrument.

It is much better to have a second person looking at source and target, making edits if it is not accurate, and making suggestions for changes, but there is no evaluation there. There is just someone who—you hope—is going to improve the quality of the translation. If that person has a strong feeling about how things should be said s/he may make a lot of unnecessary edits and these don’t necessarily improve translation quality. Additionally, if this person is, for example, only revising the translation of one part of a set of other documents, there is the risk that he or she may introduce inconsistencies. Revision—in our view—is a somewhat restricted notion, because it does not include all the tools, all the knowledge and, above all, the holistic view.

An evaluation’s aim to improve the quality of the entire project, so you need to train the evaluator for the specific set of criteria that you have agreed with your client. That is what you don’t have in a revision per se.

SN. Is there any way to integrate LQE in a workflow which is automated and on the Cloud?

SD. Yes, that is in fact the idea. You can work on API’s, use regular expressions, or whatever your preference is. Once you have a style guide, once you have a term base, you can encode rules that allow for automatic checking of the compliance with the style guides and adherence to the term base. You can go further, too: we have programmed dozens of language-specific or project-specific rules that automatically flag non-compliance to certain conventions.

For example, the question mark in Greek is a semi colon so you can introduce a rule that every question mark in the source has to be translated as a semi colon in the Greek target. This is of course a very simple example, but with a few more hours’ time you can have any criterion-based translation quality evaluation implemented in automated workflows, if you have good translation technologists (and in India you have some of the best in the world!)

SN. What about the costs of a professional LQE service? Are you going to end up spending more than for the translation project itself?

SD. Cost is indeed an issue and there are different ways of approaching it. For example, for some clients we take asample-based approach. Together with the client we select what is—in their eyes—crucial. This could be checking the professional terminology or else ensuring that, if there are legacy materials, the well translated ones are kept and the others, revised. LQE services can be tailored to specific client needs. Such services can add anything from 5% to 120% to the translation cost. A full LQE on all segments with metrics would on average add about 40% more to the translation cost. And there can indeed be cases where the client pays more for LQE than for the actual translation. You need to probe how far the client is prepared to go. Some may only be willing to add 10% to have some scores. The more you automate, the more processes you can adapt and recycle, the less it is going to cost.

SN. How can interested freelancers and LSPs integrate linguistic quality evaluation? Can you give a step-by-step overview of what it would entail?

SD. It’s about striking a balance between customization and standardisation. For example, you can determine that, regardless of what your customers ask, you keep five main categories of errors: accuracy, terminology, localisation and locale errors, stylistic errors, and linguistic correctness. You define them as such and see to it that they are monitored in some way in the workflow, in project management.

The first step is to determine what you are going to build into the workflow as a standard. Once it’s built in, it really does not add much to the cost to your production. It’s methodology, it’s process-based. The second step is for each project, for each client, define an “evaluation corridor”. You are going to review within this corridor, which is a combination of requirements by the clients and what has been agreed internally to do anyway. The hardest part about getting to a standardised methodology is getting the evaluators to stay within the corridor and stay away from the rest.

SN. The increased cost of LQE can of course be sold as a “market differentiator”, nonetheless this remains a very price-sensitive market. How do you see India as a market for LQE? Is it ready?

SD. Ten years ago no one would have looked into marketing organic food in India and now there is a market for it. Those who started ten years ago are now in a very good position. In the language market, what I see today, for example, is that the pharmaceutical industry needs clinical surveys and informed consents all the time. What they say is that there are excellent linguists and excellent technology in India, but there is a lack of methodology. They need and are looking for translation quality assurance for their surveys and informed consents and this is right now, not in the future.

There are also other products where I think that very quickly those who profile themselves now as having LQE in place will find that in 2-3-5 years the proportion of the players on the market who are educated to the need for LQE will have increased. They will be in pole position for these emerging markets. Even in the current market I see signs of demand that goes faster than the offer.

SN. Is LQE a full-fledged service or is it a form of re-packaging of proofreading? Does it really add value or is it the illusion of value that one is selling when selling LQE?

SD. If you have a good methodology in place, adding score cards to your service offer is maybe relatively close to what you refer to as “repackaging” because you are not fundamentally changing your process. However, revision and proofreading always come with huge elements of subjectivity.

Here is an example of a horizontally structured score sheet where you have different levels (minor, major, critical), weighted accordingly, and the different categories (this is not a cApStAn score sheet, it is from a theory textbook).

Below we have an example of a vertically structured score sheet, where you have the severity (minor, major, critical, fatal) and the error categories vertically. In this case “fatal” means “fail” and the translation should not be delivered (this would have been agreed with the client before, of course).

A score sheet like these will add a lot to an LSP’s credibility with clients. And, ultimately, it’s more about a mindset and methodology than a total change. Perhaps in the first years you may need some help in setting it up, there are a lot of consultants who can provide assistance and it could be a good topic for CITLob to organise a webinar to train people in setting up a methodology.

SN. To conclude, defining quality and putting the methodology in place is where the conversation now needs to be moving

SD. I am confident that the shift in mindset is within reach. Factoring robust translation quality evaluation processes into localisation workflows is neither reserved to large LSPs nor a huge investment. If we create awareness, we raise the bar, and higher standards will help differentiate the most demanding among us. The Indian language industry has enormous potential, and the last stretch towards measurable excellence is within reach of many LSPs. Onward!

21st century skills Artificial intelligence cApStAn updates Conferences DEI-BR Education Evolution of language Films Fun with languages Healthcare Language advocacy Languages/Linguistic curiosities Linguistic research Literary translation Neural Machine Translation Others Professional certifications Surveys and Opinion Polls Survey translation and localisation Talent Management Tests and Assessments Test translation and localisation Translation industry Translation profession Translation technology