Improving Lexical Selection using Morphological Analysis

Edward Kenschaft
LING895 Doctoral Candidacy Research

See also reading group notes.

• Abstract
• Introduction
• Related Work
• PBSMT
• Crosslingual WSD
• WSD & MT
• Unsupervised Morphological Induction (UMI)
• Morphology & WSD
• Morphology & MT
• Factored Language Models
• Corpus Linguistics
• Task Formulation
• Hypothesis
• Resources
• Evaluation
• Agenda
• Observations & Results
• WSD
• Table: WSD results on Vickrey English data
• Morphology
• Machine Translation
• Conclusions
• Appendices
• Future Work
• Terminology
• Additional Resources
• References

Abstract

Word sense disambiguation (WSD) is the process of determining which sense of an ambiguous word is intended in context. The task has received considerable study, typically in a monolingual environment.

Crosslingual WSD treats each possible translation of a word into a target language as a distinct sense. WSD techniques can then be used to aid in the process of translation – specifically, in lexical selection.

Morphological analysis is the process of breaking a word into its component parts. This has also been studied in depth, but rarely in the context of WSD or translation. Unsupervised morphological induction can be used to induce morphological structure in any language where written text is available, regardless of what else is known about the language.

We demonstrate that unsupervised morphological induction can improve crosslingual WSD, which in turn improves lexical selection for machine translation.

Introduction

???

Related Work

Phrase-Based Statistical Machine Translation (PBSMT)

Chiang 2005

The University of Maryland currently boasts the state-of-the-art PBSMT system, Hiero (Chiang 2005).

Previous PBSMT systems (e.g. Koehn 2004; Och & Ney 2004) generated translation candidates at two levels, word-level and phrase-level, where a word is any contiguous sequence of characters delimited by white space (i.e. a token), and a phrase is any contiguous sequence of words. The primary innovation of Hiero is to generate translation candidates from a hierarchy of phrases of arbitrary depth.

One effect of PBSMT in general, and hierarchical PBSMT in particular, is to mitigate the impact of overzealous tokenization. Two "words" which should not have been separated are simply grouped together by the system into functional phrases. Thus, we expect to benefit by using the most aggressive morphological analysis available.

Caveat: Both Pharaoh and Hiero use a configurable maximum number of "words". Breaking words into their component morphemes will increase the length of any given "phrase". The phrase-length limit will therefore need to be increased, which is likely to increase processing time.

Hiero, like Pharaoh before it (Koehn 2004), provides a probabilistic lexical selection mechanism. The preprocessor can specify one or more proposed translations for a word or phrase, each with a feature label (e.g. "WSD") and a specified probability or cost. For any group of identical feature labels, the MERT process (explain???) then determines empirically how much weight to assign to that feature. The result is a better-informed system, where the worst that can happen is that the weight of this new feature is reduced to 0.

Crosslingual WSD

Do we need a separate section for WSD???

Crosslingual word sense disambiguation (WSD) uses words of the target language as the sense inventory for words of the source language. For instance, if the source language word bank can translate to either banque or encaisser in the target language, then [banque, encaisser] are considered to be possible senses for bank. This nicely solves theoretical questions in WSD such as what level of granularity to use for word senses. If it leads to a different translation, it's a different sense; if it doesn't, it's not.

Vickrey et al. 2005

Vickrey et al. (2005) describe crosslingual WSD results for data taken from the English→French Europarl parallel corpus (Koehn 2002). Their baseline always guesses the most common sense for a given word, yielding an accuracy of 51.1%. Their logistic regression model yields an accuracy of 62.0%, a considerable improvement.

Kenschaft 2005

Experiments suggest that one of the major obstacles to widespread implementation of crosslingual WSD is the morphological complexity of either or both languages (Kenschaft 2005). A morphologically complex source language leads to data sparseness by exploding the lexicon. A morphologically complex target language leads to data sparseness by exploding the sense inventory. For instance, looking at English-French translation, a naïve system considers the English student to have four senses [étudiant, étudiante, étudiants, étudiantes], even though a human observer can easily see there is only one.

We anticipate the greatest benefit to WSD will result from breaking down both source language and target language into morphemes and throwing away inflectional affixes, looking only at the stems. Thus student has only one sense [étudiant] and the French affixes -e and -s are ignored.

WSD & MT

Carpuat & Wu 2005

Carpuat & Wu (2005) concluded, "Even state-of-the-art WSD does not help BLEU score." Their experiments used WSD to constrain the possible translations, in contrast to our approach of providing probabilistic lexical selection recommendations.

Cabezas & Resnik 2005

Cabezas & Resnik (2005) attempted to use crosslingual WSD to improve Spanish→English MT. They found no significant effect. They performed no lemmatization or morphological analysis of any sort; nor did they limit their attempts to content words.

Unsupervised Morphological Induction (UMI)

Morphological analysis is the process of parsing a word into its component morphemes. This can be done manually, which is obviously quite time-consuming, or automatically, using either a rule-based or statistical system. Many of the most commonly studied languages have commercially available rule-based systems which achieve greater than 95% accuracy (???). However, developing such a system for a new language is understandably time-intensive, as is annotating data for use by a supervised statistical system. We will therefore be focusing our attention on unsupervised (or semi-supervised???) induction of morphological structure, an area which has been studied in depth for many years (e.g. Koskenniemi 1983; Kay 1987; Koehn & Knight 2003).

Side note: The name Kenschaft would fit nicely with this list of contributors.

Freitag 2005

Freitag (2005) offers a methodology for inferring morphosyntactic transformation rules, as opposed to morphemes per se. In a nutshell, his method involves the following steps, explained in more detail below???.

1. Group all tokens into cooccurrence classes.
2. Identify Perl-like transformations between members of one cooccurrence class to members of another.
3. Cull transforms to minimal set.

Some advantages of Freitag's system are as follows.

• It provides a motivated distinction between stems and affixes.
• It rates affixes according to frequency, and allows configurable admissibility threshold.
• It can readily be adapted for more complicated transformations, such as reduplication and deletion.

Baldwin 2005

Morphological analysis is related to a task known in the literature as (deep) lexical acquisition (DLA) (Baldwin 2005). Baldwin assumes a seed database, and describes a variety of methods for augmenting that database. His method for identifying morphemes is summarized as follows.

1. Generate all 1- to 6-grams.
2. Filter out n-grams with fewer than (n=3) occurrences.
3. Filter out any n-gram with same frequency as supersequence.
4. Select (k=3900) n-grams with highest saturation (= most complete coverage?).

This method, I suspect, is inferior to Freitag's, since it does not share any of the advantages listed above. However, it should be significantly faster to implement.

Baldwin's non-morphological features depend on deeper semantic knowledge, which I do not presume to have.

Morphology & WSD

To the best of my knowledge, no one has researched the effects of morphological analysis on word sense disambiguation (WSD), either monolingual or crosslingual. As noted above, previous experiments provide anecdotal evidence that it should help dramatically.

Morphology & MT

Nießen & Ney 2000, 2001a, 2001b, 2004

Nießen & Ney (2000, 2001b, 2004) explore morphosyntactic restructuring to improve English-German alignment. Their experiments show significant improvement in translation quality, although the most effective techniques were syntactic rather than morphological. Their techniques are specific to this language pair.

Nießen & Ney (2001a) introduce the idea of a hierarchical lexicon, where a word is represented at various levels of inflectional specificity, starting with the bare stem. The technique yielded a slight reduction in subjective semantic error rate. The technique was further developed by various later researchers

Koehn & Knight 2003

Koehn & Knight (2003) experiment with heuristics for dividing German compounds into their component parts, evaluating the effect on alignment (German-English), word-based MT, and phrase-based MT. The technique produced satisfying improvements (0.026 BLEU) with word-based MT, and impressive improvements (0.039 BLEU) with phrase-based MT. This is explained by the observation that overzealousness in dividing a word into alleged morphemes is no problem for phrase-based MT, which simply regroups morphemes into phrases as necessary.

I expect Hiero to show even greater improvements, particularly with overzealous tokenization.

Improvements in alignment were most dramatic when information from the parallel text was taken into account. Basically, every possible split is checked against the aligned text to see if it corresponds with English words, and discarded if it does not.

Young-Suk Lee 2004

Lee (2004) observes that in Arabic-English translation, one Arabic word freqently aligns with multiple English words, owing to functional affixes in Arabic, e.g. llmEArDp ↔ of the opposition. This 1-many alignment poses technical challenges to PBSMT systems.

Lee addresses this problem with a several-step process:

1. POS tag Arabic and English parallel text.
2. Segment Arabic words into prefix(es)-stem-suffix(es).
3. Align segmented Arabic text with English text.
4. Determine translation probability of each Arabic stem/affix and POS into English POS.
5. Evaluate each Arabic affix.
1. If the affix robustly translates to an English POS, keep it as a token.
2. If the most common translation for the affix is NULL, delete it.
3. Else, merge the affix back into the stem.

He evaluates the effects on BLEU score for corpora ranging from 3.5K to 3.3M words, using either Model 1 or phrase-based alignment. For Model 1, he consistently approximately doubles the BLEU score. For PBSMT, the improvements are more modest, but significant, e.g. 0.36→0.39 for the 3.3M corpus.

The PBSMT scores include a manual deletion of multiple definite determiners in a single Arabic phrase.

Popović & Ney 2004a, 2004b, 2005

Popović & Ney (2004a) build on the concept of the hierarchical lexicon (Nießen & Ney 2001a). They first parse German words into stems and artificial inflectional morphemes, e.g. PRES for present tense. They then use a modified EM alignment algorithm to treat each of these complex "words" as a hierarchy, with alignment possible at any level. They evaluate effects on a variety of baseline systems, producing at least modest improvements in every case.

Popović et al. (2004b, 2005) use knowledge of the specific language pair to remove inflectional morphemes or function words that are not translatable, leading to a significantly reduced lexicon, and reduced AER.

I presume the alignment of function words to NULL has been handled in the mainstream alignment literature, although I haven't read up on it. Removing inflectional morphemes by hand seems (a) ad hoc, and (b) potentially detrimental to translation due to loss of information. Splitting apart but retaining all morphemes should provide the same benefits, without these objections.

Bettina Schrader 2004, 2006

Schrader (2004) performed experiments to determine the impact of morphological analysis (among other things) on German-English alignment & translation. In experiments on three different data sets, she removed inflectional affixes from the German, and aligned/translated only the roots (lemmas). She evaluated on translation candidates generated from tokens of interest in new sentences, using a bag-of-words dictionary generator (Hiemstra 1996).

With all three data sets, recall went up with lemmatization. However, with the exception of one data set (Patente), precision went down. Schrader attributes this to a naïve precision metric, which unduly penalizes the system for generating multiple translation candidates for the same term.

Using only the Patente data set, Schrader tried another experiment where she broke long compounds into component morphemes before alignment, e.g. Dämpfungsscheibenanordnung ('dampening disk assembly') → Dämpfung scheibe anordnung. Recall improved somewhat, but precision did not change significantly.  Schrader noted that the system was severely hindered by its inability to handle alignments involving multi-word units. This suggests that results should be significantly better with a modern PBSMT system.

Schrader (2006) addresses the problem of aligning German noun compounds with English multi-word units. She identifies a compound as any word of 12 or more characters, marked by a POS tagger as a noun. When a compound noun is identified, she invokes a special procedure for identifying potential English multi-word units with which it might align. Overall, the process resulted in 70% recall.

No baseline for comparison exists, confirmed in email from Bettina.

Schrader argues that compounds should not be broken into components, since the meaning of the compound may not be compositional. However, a modern PBSMT system will take this into account, translating the sequence as a unit when possible, and as individual components when not.

Goldwater & McClosky 2005

Goldwater & McClosky (2005) provide a good summary of the issues motivating morphology in MT, and the history of related research. They also set up a variety of experiments in Czech-English MT, resulting in a BLEU score improvement from 0.270 to 0.333.

Their last experimental setup is supposed to use full morphemes. However, they are not what I conceive of as morphemes. First they lemmatize each word and attach pseudowords (cars → car+PL), then they fiddle with the alignment algorithm to treat the word as a complex structure. As discussed above, this has potential advantages over treating morphemes as words (cars → car +s) and using GIZA++ for alignment, but is also more challenging to implement.

A quick glance at their sample Czech data suggests that it is highly fusional and suppletive. This suggests that I would be better off postponing work with Czech until I have made progress with a cleaner agglutinative language, such as German.

Isbihani et al. 2006

Isbihani et al. (2006) compared the impact on MT of various methods for segmenting Arabic source text. They found that the best results came from an unsupervised method using an FSA augmented with a memory of word properties.

Factored Language Models

???

Corpus Linguistics

Babych & Hartley 2003

As part of a paper on ???, Babych & Hartley (2003) introduced the s-score heuristic for identifying content words in a large corpus.

Task Formulation

Hypothesis

This set of experiments is intended to support the following (complex) hypothesis:

1. Unsupervised morphological induction (UMI) can improve the quality of crosslingual WSD.
2. Crosslingual WSD can improve the quality of phrase-based statistical MT.

Resources

Software

For WSD we will use the system described in (Kenschaft 2005).

For MT we will use Hiero for original experiments. The WSD system will provide lexical selection recommendations.

Languages of Interest

Choices of language are driven by several factors:

1. morphological complexity & variation
2. ease of experimentation
1. availability of parallel text
2. readability to native English speaker (me)
3. demand within the broader research community

We mostly limit our experiments to language pairs with English as the target language, because English is relatively simple morphologically. We can therefore hope to get away with doing morphological analysis only on the source side, postponing various challenges introduced by doing similar analysis on the target side.

For source languages, we will include high-resource languages covering all the major language types.

• analytic – e.g. Chinese, Bulgarian
• agglutinative – e.g. German, Turkish, Inuktitut
• fusional – e.g. Spanish, French
• suppletive – e.g. Czech
• templatic – e.g. Arabic

The types are presented in order of the expected degree of challenge to an unsupervised morphological induction system.

Preprocessing Methods

Several different preprocessing heuristics and algorithms will be evaluated. The simplest will be to truncate each word to n characters, with n ranging from 4 to 7, a la (Och ???). Other simple heuristics will include stripping off inflectional endings known to be common, such as -e and -s in French.

A more sophisticated UMI algorithm will be based on (Koehn & Knight 2003). In a nutshell, we will use known root words to posit structure for longer words containing those roots. Koehn & Knight successfully used this approach with German complex nouns. We predict that it will also be of benefit for more inflectional, less agglutinative languages, such as the Romance languages.

If we have time, we will also implement other sophisticated heuristics, such as those described in (Frietag 2005) and (Baldwin 2005).

Evaluation

We will evaluate our approaches against the following state-of-the-art systems.

Vickrey et al. 2005

Vickrey et al. (2005) describe crosslingual WSD results for data taken from the English→French Europarl parallel corpus (Koehn 2002). They graciously sent us a copy of their data so we could replicate their baseline experiment and compare our results to theirs. We will also show results for all other Europarl languages translating into English.

Vickrey et al. also describe an experiment with filling in a missing ambiguous word in a target sentence. This was intended to prove the potential usefulness of WSD in machine translation. We will repeat their experiment using our own WSD system.

Cabezas & Resnik 2005

Cabezas & Resnik (2005) attempted to use crosslingual WSD to improve Spanish→English MT, without significant success. We will use the same data, but using morphological analysis, and limiting our efforts to content words.

Koehn & Knight 2003

Koehn & Knight (2003) used morphological analysis of German compound nouns to improve German→English MT, with significant success. We will attempt to get a copy of their data, or else construct a comparable framework from the Europarl corpus. We may use Pharaoh instead of Hiero for this set of experiments, in order to most closely mimic their environment. We will show results using:

1. just the original text (baseline)
2. morphological analysis alone
3. morphological analysis and lexical selection using crosslingual WSD

Since our focus is on lexical selection rather than morphology alone, we will be less concerned with whether our result (2) improves upon (Koehn & Knight 2003), and more concerned with the improvement from (2) to (3).

If time permits, we will also implement Koehn & Knight's algorithm for identifying crosslingually relevant morphology.

Hiero

The UMD MT team continues to develop and enhance Hiero. We will evaluate using a recent stable version of the system, Europarl language data, and GALE Arabic→English data. We will compare results using:

1. Hiero alone
2. morphological analysis on source text
3. WSD for lexical selection.

Again, we will be more interested in the improvement from (2) to (3).

Isbihani et al. 2006

I am not inclined to pursue any comparisons with this paper, except perhaps as a follow-up exercise.

Agenda

Our experiments will follow roughly the following plan.

The complete long-term agenda can be found under Future Work.

Observations and Results

WSD

Preliminary

The number of possible experiments based on (I) above is obviously huge. We ran several representative experiments on small data sets with various languages, using English translations as a sense inventory. Some results are listed in Table 1, using the following key.

• Source
• Lang : source-target language pair, using standard abbreviations (e.g. de-en for German-English)
• Process: text preprocessing, according to the following options
• none : no preprocessing
• truncN : truncate to N characters
• lc : lowercase all
• diac : eliminate diacritics
• suffix : strip common inflectional endings (-e and -s)
• Evaluation
• Dev : accuracy on dev set
• Statistics
• Types : number of distinct types (i.e. word forms) being disambiguated
• Senses : total number of distinct senses (i.e. translations) for all types
• Examples : total number of training examples for all types
• S/T : average number of senses per type
• E/T : average number of examples per type
• E/S : average number of examples per sense

Experiments are sorted by language in decreasing order of accuracy. Blue is the baseline.

Table 1: Naïve WSD results

Lang	Process	Dev	Types	Senses	Examples	S/T	E/T	E/S
de-en	trunc6	0.575	99	1505	127071	15.20	1283.55	84.43
	trunc5	0.574	96	1961	155221	20.43	1616.89	79.15
	diac	0.559	101	598	45048	5.92	446.02	75.33
	none	0.558	101	587	44330	5.81	438.91	75.52
	lc	0.558	101	587	44330	5.81	438.91	75.52
es-en	trunc6	0.614	99	1006	101280	10.16	1023.03	100.68
	trunc5	0.608	98	1490	136166	15.20	1389.45	91.39
	diac	0.604	104	618	50637	5.94	486.89	81.94
	none	0.598	104	610	46811	5.87	450.11	76.74
	lc	0.598	104	610	46811	5.87	450.11	76.74
	strip	0.575	97	825	93046	8.51	959.24	112.78
fi-en	trunc6	0.560	60	522	37618	8.70	626.97	72.07
	trunc5	0.557	58	755	55885	13.02	963.53	74.02
	lc	0.539	63	347	25411	5.51	403.35	73.23
	none	0.539	63	347	25411	5.51	403.35	73.23
	diac	0.538	63	347	25411	5.51	403.35	73.23
fr-en	trunc6	0.598	75	930	79461	12.40	1059.48	85.44
	trunc5	0.593	74	1338	111140	18.08	1501.89	83.06
	diac	0.586	80	485	36139	6.06	451.74	74.51
	none	0.584	80	478	35857	5.98	448.21	75.01
	lc	0.583	80	478	35857	5.98	448.21	75.01
	strip	0.568	77	650	66794	8.44	867.45	102.76

graphs???

Certain observations jump out across all languages tested.

1. Of these naïve heuristics, only truncation has a significant positive effect. This suggests that truncation alone might help MT, as suggested in (Och ???).
2. Lowercasing and removing diacritics have negligible effect. We held these constant for later experiments.
3. Stripping common inflectional endings has a significant negative effect. This suggests that those endings carry more semantic weight than we may have thought. The final -s, in particular, might carry information of benefit to lexical selection.
4. There is no obvious correlation of accuracy with any of the statistics noted, unless we throw out "strip" experiments as an outlier. In that case, there is a correlation with E/S, the average number of examples per sense. In other words, if we can increase the amount of available data without losing important information or increasing the complexity of the problem, we win.

Morphology

It is difficult to evaluate a system for morphological analysis directly. However, it is possible to produce anecdotal evidence to highlight the potential benefits, and to demonstrate its impact on downstream applications where evaluation metrics are available.

It should also be possible to compare results against rule-based systems for well-studied languages, but I have not tried this.

Rare Events

Schrader (2006) observes that compounding languages such as German have an inordinately high proportion of rare events. Indeed, roughly half of the token types are hapax. Presumably, a statistical system has a zero-percent chance of translating these correctly, since the ones that occur in the test set never occur in training. Likewise, the frequency of rare events (however defined) will adversely affect the quality of translation. While most languages are not as bad as German, Zipf's law suggests that rare events will always have a comparatively high frequency. Morphological analysis can have a potentially profound impact by reducing these impossible or nearly impossible cases to component morphemes which the system recognizes.

The problem may not be as bad as it seems at first. Even if half the token types are hapax, the percentage of singleton tokens will be considerably fewer, often less than one percent of all tokens. On the other hand, these hapax tokens are likely to be critical to the adequacy of translation, where the highest frequency types are most likely to be function words.

In other words, rare events are far more critcal to correct translation than is captured by their relative impact on BLEU score or other automated measures. Note for future work: Develop an evaluation metric for MT which weights the contribution of a word on the overall score based on its s-score.

We performed analysis similar to that in Schrader (2006) for German, English, French, Spanish, and Finnish corpora, both before and after breaking words into component morphemes. For the purpose of analysis, we considered "rare" any word that occurs 10 or fewer times in the corpus. The results are in Table 2.

Table 2: Rare events

		Total		Frequency 1			Frequency 1-10
		# Types	# Tokens	# Tokens	% Types	% Tokens	# Tokens	% Types	% Tokens
English	Words	34,827	10,648,289	9,361	26.88%	0.088%	21,350	61.82%	0.587%
	Morphemes	18,955	10,659,570	3,991	21.06%	0.038%	9,817	51.79%	0.285%
Finnish	Words	301,597	7,835,581	155,994	51.72%	1.990%	259,890	86.17%	7.102%
	Morphemes	38,019	8,085,040	12,238	32.19%	0.151%	24,095	63.38%	0.788%
French	Words	55,985	12,036,764	16,404	29.30%	0.136%	37,045	66.17%	0.875%
	Morphemes	25,880	12,058,591	5,756	22.24%	0.048%	13,524	52.26%	0.334%
German	Words	170,402	13,233,854	81,052	47.57%	0.613%	140,464	82.43%	2.377%
	Morphemes	30,898	13,348,182	9,836	31.83%	0.074%	19,507	63.13%	0.385%
Spanish	Words	77,526	12,495,786	25,772	33.24%	0.201%	54,908	70.83%	1.182%
	Morphemes	32,407	12,530,435	8,102	25.00%	0.065%	18,334	56.57%	0.425%

bar graphs ???

As expected, the results are dramatic for German, a compounding language, where the number of distinct types drops 82% from 170,402 to 30,898, and the number of rare tokens drops 86% from 140,464 to 19,507. The results are even more dramatic for Finnish, a highly inflected language, where the number of distinct types drops 87% from 301,597 to 38,019, and the number of rare tokens drops 91% from 259,890 to 24,095. We expect to see the most impact on translation quality in these languages.

However, the reduction in complexity is considerable for all languages, even English, a mostly isolating language. We can reasonably expect improvements in translation and other downstream applications in all these languages.

Note that the number of tokens necessarily goes up as words are broken into component morphemes, but the amount of increase is not as significant as one might expect. Presumably, this reflects the characteristic of language that the most common words are likely to be morphological simple (or irregular, in the case of common verbs).

To be honest, the amount of increase seems ridiculously low. You would expect the increase in tokens to be at least the same magnitude as the decrease in types, but this is not always the case. I will need to double check the code, in case I missed something in the counting. But I expect the conclusions to remain essentially the same.

Schrader (2006) used a totally different approach, both to simplification and evaluation. It is difficult to compare results.

Morphology & WSD

This set of experiments evaluates the use of morphological analysis to improve WSD results.

The analysis involves the following systems:

1. baseline – always choose the most common sense
2. sgt.unmod – run the WSD system on unmodified data
3. sgt.trunc5 – run the WSD system with all words truncated to 5 characters
4. sgt.trunc6 – run the WSD system with all words truncated to 6 characters
5. sgt.morph – run the WSD system using full morphological analysis
6. best – for each word, evaluate each of the above 5 systems on a dev set, and use the best-performing system to make predictions on the test set
7. best (no morph) – same as (6), but omitting sgt.morph

Vickrey

Table 3: WSD results on Vickrey English data (as of 9/27/06)

	Precision	# Best
baseline	52.94%	595
sgt.unmod	55.59%	540
sgt.trunc5	49.96%	214
sgt.trunc6	51.53%	228
sgt.morph	54.66%	294
best	57.73%	–
Vickrey baseline	52.6%	–
Vickrey best	62.0%

The overall results of applying morphological analysis for WSD on the Vickrey data set were negative. Table 3 shows the overall precision of each system, and the number of base words for which each individual system proved to be the best.

There are several reasons why these results may not be indicative.

1. The training set is unusually small, with sparse data. More than 25% of words have 60 or fewer examples, with the median at 164 examples.
2. The source language is English, which we know to be morphologically poor.
3. The baseline is remarkably high, greater than 50% simply from guessing the most common sense for each word.

Note that our system does not beat Vickrey's best. This could be explained in part because Vickrey uses a supervised component, a POS tagger, while we use exclusively unsupervised approaches. A direct comparison would require us to integrate Vickrey's POS tagger.

The # Best column provides useful insights. Although the unmodified WSD system is the best overall performer, the baseline is the best performer on more words. Each of the other three systems is the best performer on a non-neglible minority. This suggests there might be significant benefits from using a dev set to determine which system performs the best on a given word, and using that system on the actual test data.

Eyeballing a word-by-word breakdown of results (not included) does not shed much light. The ultra-high-frequency words tend to work best with either the baseline or unmodified WSD, which helps explain the high overall scores of those systems. Beyond this, it is not immediately clear how results are correllated. It's not even clear that the words where the morphological system performs best are those with the most morphological variation. For instance, love is one of these words, where no other morphologically similar words occur in the corpus.

Machine Translation

Training and test data are those used for the NAACL 2006 Workshop Shared Task: Exploiting Parallel Texts for Statistical Machine Translation (WMT06) (Koehn & Monz 2006).  Table 4: WMT06 results lists the WMT06 results (test data only) for comparison, including:

• Highest: the highest reported score.
• Lowest: the lowest reported score.
• Mean, all: the mean of all scores.
• Mean, pack: the mean of all scores except the lowest two.

Columns indicate results on the following data sets:

• Dev: WMT06 development test set.
• In: WMT06 test set, in-domain.
• Out: WMT06 test set, out-of-domain.

Table 4: WMT06 results (as of 1/08/2007)

		Fr-En			Es-En			De-En			En-Fr			En-Es			En-De
		Dev	In	Out	Dev	In	Out	Dev	In	Out	Dev	In	Out	Dev	In	Out	Dev	In	Out
WMT06	Highest		31.94	22.50		32.37	28.35		27.30	18.87		33.66	25.26		31.85	27.76		18.85	11.82
	Lowest		21.44	19.42		23.91	19.17		15.86	11.78		25.07	21.44		23.17	16.83		9.84	6.55
	Mean, all		29.33	20.70		29.99	25.77		23.75	16.47		31.04	23.45		29.36	24.47		16.62	10.43
	Mean, pack		30.42	21.23		30.65	26.51		25.10	17.07		31.74	23.74		30.10	25.39		18.87	10.90
Baseline	Hiero/base				25.77	25.67	24.25	21.01		15.69
	Hiero/base (sub)				27.14	27.08	24.14
Truncation	Hiero/trunc5
	Hiero/trunc6
Morphology	Hiero/morph
WSD	Hiero/wsd
	Hiero/morph/wsd

Conclusions

.