As a child, I read a lot and wrote plenty of stories of my own. These days, I no longer make time for either, and language has become more of a means to an end than an end in itself. I often catch myself trying to use a familiar turn of phrase, only to remember half of it and fill in the rest of the sentence with a word that sort of fits, but does not quite sound right.

I do not mean idioms or metaphors, but phrases that occur often enough in everyday language and literature that replacing even one word with a very similar one sounds odd. As it turns out, these phrases are called “collocations.”

What bothers me most are phrases where swapping out a particular word makes no real difference to the meaning, yet only one version truly sounds right. And sometimes, I simply cannot think of it.

For example: to attach weight to something. “The judge decided to attach particular weight to the testimony.” You could just as well say “assign particular importance to it” or “give the testimony special emphasis.” But somehow, it all sounds a little stranger.

I know I am not alone in this. When I spoke to other people my age about it, a friend, who shall remain anonymous here, agreed with me and made a remark about a likely cause of the problem that describes it rather well. The conversation was in German and what he said translates to:

“Writing is a fruitless bread.”

This sounds just as odd in German as it does in English. The phrase was probably a blend of the German version of “a fruitless endeavour” and the german idiom “a breadless art”, both of which are fine examples of collocations themselves. At least, I do not know of any such expression, and I could just as easily have made the same slip.

The role of AI

The arrival of LLMs has certainly not improved matters: when writing prompts, you really do not need to pay attention to your phrasing, and LLMs formulate sentences in the way they occur in their training data, even when that means using a common but still awkward variation of a collocation.

One kind of precursor to today’s clever LLMs is the masked language model (MLM). These models are trained to fill a gap in a sentence with its most likely word. They can also generate several suggestions and assign each one an approximate probability of being correct.

This gave me two ideas at once: first, to find out how well simple MLMs can identify the right word in a collocation; second, to build a simple app in which people can practise using collocations correctly while competing against an MLM.

MLMs and collocations

The MLMs tested

To find out whether a language model would make a suitable opponent and scoring system, I tested 322 English collocations in fill-in-the-blank sentences. Alongside each model’s first suggestion, I also checked whether the target word appeared among its five suggestions in total.

BERT-large was by far the most useful. Still, it gets many collocations wrong without producing completely nonsensical answers: instead of the fixed expression I was looking for, it often suggests a more ordinary or stylistically flatter alternative. For instance, lodge a complaint becomes file a complaint. This is interesting because the game is precisely about the difference between “it sort of fits” and “it really sounds right.”

With BERT-large, the accepted answer appears among the five suggestions in almost half of all cases. In 54 exercises, it simply does not come first.

BERT-large is actually at a slight disadvantage in this task: it is trained to predict individual tokens, whereas a word (or whatever belongs in the blank) can consist of several tokens. Unlike BERT, T5 can fill blanks with multiple tokens, but it performed surprisingly poorly here. FLAN-T5 was asked directly to fill the blank, but usually returned other words from the sentence instead. That is why I use BERT-large for the English version.

The German version uses bert-base-german-cased, a BERT model pre-trained on German texts. It fills the same masked blanks and provides the alternative answers for the German questions.

The collocation pipeline

The collocations and their fill-in-the-blank sentences are stored separately as small JSONL datasets. Each exercise includes the sentence, the position of the blank, a difficulty level, and, where appropriate, several accepted answers—for example, different verb tenses. The pipeline replaces the target word with a mask, has BERT calculate the most likely completions, and saves them along with their scores.

One hundred questions were selected for the game from the English test set. BERT-large gets 69 of those 100 answers right. There are now also 100 German questions; a BERT model trained on German gets 47 out of 100 right there. This produces static data that does not have to be calculated while playing: it contains both the correct answer and the model’s plausible alternatives for each question. The fill-in-the-blank game then uses exactly this data.

The fill-in-the-blank game

Using the data generated beforehand, I built a small website that asks users to fill the blanks in sentences correctly. Each question is worth 100 points for the correct answer, 75 points for the answer the MLM considers most likely, and partial points for answers the MLM also considers possible, in proportion to their probability.

Conclusion

The MLMs I tested were not particularly good at recognizing collocations. However, these were only smaller models that can run on a desktop PC, and with a little more effort the results could probably be improved considerably. I did not spend too much time optimizing them, since I decided to use the AI suggestions only to score players’ answers. A model that was too good would probably be more discouraging anyway. The resulting game is a pleasant little distraction and can be found on my website.

Comments

Feel free to leave your opinion or questions in the comment section below.