Monday January 9, 2006

Slate on MT

There's an interesting article in Slate today that describes various kinds of machine translation software.  It's written by a translator who heard a rumor at his company that he and his fellow humans were about to have their work "supplemented" by software, and so decided to look into the quality of various MT packages.  Many of the packages produce crap, but the one statistical MT package, Language Weaver, seems to surprise him with its accuracy.

The author, Jesse Browner, tried running a sentence in several languages about Pope Benedict XVI, "His youthful years were not easy", through several programs.  While the non-statistical ones produce the sort of humorous results we've all come to enjoy from automatic translation (visit the web site Lost in Translation if this is your idea of fun), the statistical one does much better:

The one that stood out from the pack was Language Weaver. Not only did it recognize the subject as a human being—"The period of his youth was not easy"—but it translated the rest of the paragraph with only one minor error. Intrigued, I began to put the software through its paces. A headline from El Pais: "A wave of attacks left more than 100 dead in several cities in Iraq." So far, so good. A speech from the United Nations: "The problem is to maintain the level of international attention and ensure the implementation of the commitments." Perfect. The first line of Don Quixote: "In a place of the Channel, whose name do not want to remember, has not much Time living a Hidalgo the spearheaded in shipyard, adarga Antigua, Rocín weak and galgo corridor." Clearly, in the world of machine translation, everything has its limits.

Browner seems to show that the SMT package does pretty well with simple sentences, but falls down on something more literate.  I'm suspicious.  Unfortuantely, Browner doesn't provide the original sentences for some of his examples, but we can look up the first sentence of Don Quixote:

En un lugar de la Mancha, de cuyo nombre no quiero acordar-me, no ha mucho tiempo que vivía un hidalgo de los de lanza en astillero, adarga antigua, rocín flaco y galgo corredor.

As I've mentioned before, my knowledge Spanish is slight, so I can't really evaluate how far the language of the novel (which was published in the early 1600's) differs from the modern Spanish the SMT package was surely trained on, but isn't this like handing a line from Shakespeare to a package trained on modern English and being surprised when it can't translate it cleanly?

After discussing SMT very briefly (compling's own Michael Collins along the way), Browner concludes that he's not likely to be out of work soon:

Language Weaver CEO Bryce Benjamin acknowledges that even the best translation software cannot hope to replace human translators; it is simply one tool "to help them to increase productivity and value." That was all I needed to know. The war on terrorism notwithstanding, my job and those of thousands of professional translators in the arts, sciences, and industry seem relatively safe for now. But then again, horses were pretty damn sure of themselves 100 years ago, and look what happened to them.

I think Browner's right that it'll be a very long time before human translators are obsolete for any kind of work where the results need to be polished and perfect, but I think he's fooling himself if he doesn't expect MT systems to significantly impact the translation business much sooner than a century from now.  With ever more processing power, ever improving statistical models, and ever more data to train on, SMT systems seem likely to continue to improve rapidly.  I suspect that in a decade or two, all but the most high-stakes translations (like treaties, legislation, and perhaps literary works) will be done by first running the text through an MT system, then having a human "polisher" go over the output and fix it up.  Now, this may or may not put most translators out of work—if fewer person-hours are needed per page, then the cost of translation should fall, resulting in more texts being translated and continued demand for human translators as "polishers"—but it's going to change the business of translation significantly.  [However, a disclaimer: I'm probably less familiar with the translation industry than Browner is with MT, so I may be underestimating how crucial human translators are.]

Hmm, at this point in the post I'd love to point you at a more detailed article of how a statistical MT system works, but a couple of minutes of Googling didn't produce good overviews aimed at non-specialists.  (If you know of one, leave a comment.)  This is just the sort of geek-friendly subject you'd expect Wikipedia to have a good article about, but at the time of this writing there's no statistical machine translation article and the machine translation article is pretty brief and labeled "in need of attention from an expert on the subject".  I've just started a seminar on MT, so I'm not an expert, but maybe if there's still nothing there at the end of the quarter, I'll stop taking petty potshots at Wikipedia and actually contribute something.  (It could happen!)

Tangent:  Today we were driving by a cemetery and saw a backhoe, which they must use to dig graves, crossing the street.  Imagine what the old-school grave diggers with the long shovels must have thought about the machines that have replaced most of them, and you'll get some idea why you should take an article about machine translation written by a professional translator with a grain of salt.  Paradoxically, though, there's probably no one better qualified than a translator for evaluating the performance of MT software.  That's some catch.

[Now playing: "Hunter" by Björk]

I am The Tensor, and I approve this post.
05:41 PM in Linguistics | Submit: | Links:


TrackBack URL for this entry:

Listed below are links to weblogs that reference Slate on MT:


A translator's opinion: You mentioned "if fewer person-hours are needed per page". That's an "if" that hasn't played out, so far. At many different translator discussion groups, forums and bulletin boards, the same story is retold: "I was asked to edit a translation. It turned out to be a machine translation." From this point, the story can take two tacks. Either (A) "It was so hard to fix that it would have been easier to translate from scratch. Now I wish I had said 'no.'" or (B) "When I realized how hard it would be to fix it, I demanded to work from the source text instead, because it would be cheaper for the client."

Browner's article gives an illustration of why this might be, although he glosses over the point completely. He ran a sentence from a United Nations speech through Language Weaver: "The problem is to maintain the level of international attention and ensure the implementation of the commitments." Browner's assessment: "Perfect." No, it's not perfect. For comprehension, yes, but not for publication. It's a *perfect* example of what Alan Duff wrote about in his book 'Third Language: Recurrent Problems of Translation into English'. It looks like English, it uses English words, it's even comprehensible to the anglophone reader, but its syntax, vocabulary choice and way the thoughts are shaped into words are 100% source language (probably Spanish or French). The translator given this sentence to edit goes through a two-step process (whether consciously or unconsciously); first mentally recreating the source sentence, then retranslating it to "real" English.

To recap, if the speech containing this sentence had been translated by MT in order to give an English-language audience an idea of what it was about, the translation would be quite adequate. But the language of the translation isn't native English; it's "third-language" English.

Since a large part of the translation industry *does* involve translating texts "for publication" (i.e. to have the same effect on their target-language reader as the original would have had on the source-language reader), not only is this quality not enough, but it's also optimistic to think that MT cuts down on human processing time and cost. Not there yet.

Posted by: Margaret at Jan 10, 2006 11:05:59 AM