What a Readability Score Cannot See

The two readability scores built into Microsoft Word are arithmetic on two counts: how many words your sentences have, and how many syllables your words have. We wrote five short work notices and scored them with two programs. The result that explains the others: a clear notice and the same notice with its words shuffled into nonsense got exactly the same score. A notice that never says who has to do what scored best of all, with a grade level below zero.

So a bad score is worth acting on in one narrow way, and a good score proves very little. The texts, the numbers and the software versions are below so you can repeat the test.

The two formulas Word uses

Microsoft’s support page on readability statistics in Word gives both formulas Word uses. Flesch Reading Ease:

206.835 – (1.015 x ASL) – (84.6 x ASW)

Flesch-Kincaid Grade Level:

(.39 x ASL) + (11.8 x ASW) – 15.59

The page defines ASL as “average sentence length (the number of words divided by the number of sentences)” and ASW as “average number of syllables per word (the number of syllables divided by the number of words)”. Reading Ease runs the other way from the grade: higher is easier. Microsoft says that for most documents you want Reading Ease “between 60 and 70” and a grade “around 7.0 to 8.0”, and that a grade of 8.0 “means that an eighth grader can understand the document”.

Nothing else goes in. The formula never looks at which words you used or what order they come in. The grade-level version goes back to a 1975 US Navy report by Kincaid and colleagues, whose abstract says the formulas were derived from tests of 531 Navy enlisted personnel on 18 passages from training manuals. “Grade 8” is a prediction fitted to those readers and those manuals. It is not a measurement of your customer.

Five notices, written for this test

All five are ours, and none was ever sent to anyone. Each is about 100 words. Open any of them to read it.

A. A clear notice

We will turn off the water to flats 1 to 24 on Tuesday 9 March 2027, from 9am to 1pm. We need to replace the main valve in the basement. Before 9am, please fill a kettle and a bucket so that you have water to drink and to flush the toilet, and do not run the washing machine or the dishwasher during the work because they may be damaged when the supply stops. When the water comes back it may look cloudy. Run the cold tap for two minutes and it will clear. If you have no water after 2pm, call Dana on 020 7946 0155.

B. The same notice, shuffled

A script shuffled the words inside each sentence and left the sentence breaks where they were.

1pm will 1 from the to flats to water on 9 9am 2027 to Tuesday 24 We turn March off. the We in replace to main valve basement need the. work supply during when you the Before because the flush washing to damaged so a kettle please have water the fill and bucket drink they 9am dishwasher and or machine may a toilet not be stops run the the to do and that. look comes water it may When cloudy the back. clear the tap for will minutes it cold Run and two. after water If have on you 0155 7946 no 020 call 2pm Dana.

C. Short words, short sentences, no information

There is a change to how we deal with post. It starts soon. Some of you know this. Some may not. The old way was slow. The new way is not. Forms will go out. They need to come back. Do not send them late. If they are late there will be a cost. We can not help with that. Some staff are at the desk. Ask them if you need to. Not all days are the same. Check first. This is for all who use the site. We hope it is clear. Thank you for your time.

D. Clear instructions that need long words

From Monday 5 April 2027 you will need two-step verification to sign in to the customer portal. Before that date, install an authenticator application on your phone. Then open the portal, choose Security, and select Set up verification. The portal shows a QR code. Scan it with the authenticator application. The application then shows a six-digit code that changes every thirty seconds. Type the code into the portal to finish. If you change your phone, repeat these steps on the new one. If you are locked out, call 020 7946 0155 and we will reset your verification.

E. Notice A with its 43-word sentence split in two. The only change is in the third sentence, which becomes: “Before 9am, please fill a kettle and a bucket so that you have water to drink and to flush the toilet. Do not run the washing machine or the dishwasher during the work because they may be damaged when the supply stops.”

The scores

Computed with textstat 0.7.13, a Python library, on Python 3.12. The figures are as the library returned them, rounded to one decimal place. The columns are words per sentence, syllables per word, Flesch Reading Ease and the Flesch-Kincaid grade, for the clear notice (A), the same notice shuffled (B), the short and empty one (C), the clear one with long words (D) and A with one sentence split (E).

We also ran Gunning Fog and SMOG, two more grade-level formulas that lean harder on long words than Flesch-Kincaid does. Both put the five texts in the same order. D came out at 8.0 on Gunning Fog and 9.4 on SMOG, which matters further down. Treat the SMOG figure as a direction only: the textstat documentation says SMOG results for texts of fewer than 30 sentences are “statistically invalid”, and these have between 6 and 17. Other formulas count letters, or check each word against a list of familiar ones. None of them reads the sentence either.

Word order counts for nothing

A and B match on all four formulas. They have to: shuffling changes neither the number of words in a sentence nor the syllables in a word.

The best score went to the notice you cannot act on

C has a Reading Ease of 116.4, above the top of the 100-point scale, and a grade of -1.5. Every word in it has one syllable. It also never says what is changing, when, which forms, where they go, the deadline or the cost. The formula rewards the short sentences and has no way to notice that the facts are missing.

Long words cost a lot, but not as we expected

We expected D to fail a grade 8 target. It did not: Flesch-Kincaid gives it 7.0 and its Reading Ease of 64.2 is inside Microsoft’s recommended range. Its sentences average under 11 words, and that pulled the grade back down. It is still the worst score in the table. On SMOG it gets 9.4, so whether D “passes” depends on which formula your tool happens to use.

To see what the long words cost, we then wrote a version of D with them swapped for short ones: “check” for “verification”, “code app” for “authenticator application”, “client site” for “customer portal”, “put” for “install”, “Safe” for “Security”.

Read the short-word version of D

From Monday 5 April 2027 you will need a two-step check to sign in to the client site. Before that date, put a code app on your phone. Then open the site, choose Safe, and select Set up check. The site shows a QR code. Scan it with the code app. The app then shows a six-digit code that changes every thirty seconds. Type the code into the site to finish. If you change your phone, repeat these steps on the new one. If you are locked out, call 020 7946 0155 and we will reset your check.

That version scores grade 2.4 and Reading Ease 97.4, a drop of 4.6 grades with the sentences left alone. It is also worse. The reader now has to find a menu called “Safe” and a button called “Set up check” that are not on the screen, and “code app” is not something they can search for in an app store.

Splitting one sentence moved the grade by a full point

E differs from A by one full stop and one deleted “and”. The grade fell from 5.7 to 4.6 and Reading Ease rose from 85.9 to 88.5. In a 100-word notice, one long sentence accounted for nearly a fifth of the grade. This is the one result where the score and the reader agree: the 43-word sentence held two instructions, and two sentences are easier to follow.

Two tools, one text, two scores

We ran the same files through the readability tool on this page, which uses the same two formulas. On A, B and E the two programs gave identical figures. On C, textstat gave a Reading Ease of 116.4 and a grade of -1.5 where our tool gave 116.7 and -1.7. On D it was 64.2 and 7.0 against 62.5 and 7.2.

Three texts agree and two do not, though the formulas are identical. The counting is what differs. A computer cannot hear syllables, so each program estimates them. We read textstat’s source: it looks each English word up in a pronouncing dictionary and, for words that are not in it, falls back on hyphenation patterns. Our tool counts groups of vowel letters and adjusts for silent endings. Both count a figure written in digits as one syllable, which is wrong for both, since “2027” is six syllables aloud. We compared the two word by word. In A, B and E they count the same syllables for every word, so the scores match. In D, textstat counts “two-step” as one syllable and “six-digit” as two, each one short, and that is the whole of the 1.7-point gap. In C the syllable counts match exactly, and the scores differ because textstat ignores any sentence of two words or fewer: it does not count “Check first.” and finds 17 sentences where our tool finds 18. Our tool has its own misses: in the short-word version of D it counts “client” as one syllable. On the 4,000 commonest words in the 79,569 emails we counted for How Long Is a Normal Work Email?, the two programs agree on 3,722. The other 278 are why any two tools will sooner or later give you two numbers for one text.

Layout makes a far bigger gap. We scored a five-line version of the kettle instruction set out as a list, with no full stops on the list items.

Read the list

Before 9am on Tuesday 9 March 2027, please:
fill a kettle and a bucket
turn off the washing machine
turn off the dishwasher
The water will be back by 1pm.

Our tool treats each line as a sentence and gave it grade 2.1. textstat looks only for full stops, question marks and exclamation marks, read the whole list as one 30-word sentence, and gave it grade 11.5. If your tool punishes you for using a list, that is the likely reason.

The example is notice A. The tool reports both scores, the counts they are built from, and highlights any sentence over 25 words, which in A is the 43-word one. Its figures can differ from Word’s or any other program’s, for the reasons above.

What the number is good for

It finds long sentences and long words quickly, in a text too long to check by eye. That is all, and it is useful. A sentence count is the more direct tool for the first job: the GOV.UK guidance on using clear language says “Try to split up sentences that are over 25 words long”, and you can find those without any formula.

It is also fair for comparing two drafts of the same text in the same program.

It cannot tell you whether the words are in an order that makes sense, or whether the text says who must do what and by when. It cannot tell you whether the reader knows the words either: “valve” and “check” are both one syllable, and only the reader’s situation decides which of them is clear.

When someone gives you a target

I would sooner be handed a rule such as “no sentence over 25 words” than a grade to hit. I can check the first with my own eyes. The second I can only check with whichever program the other person happens to use, and this test showed how far two programs can be apart. If the target is fixed, this is the order I’d work in.

Ask which formula and which program. “Grade 8” on Flesch-Kincaid and on SMOG are different tests: D got 7.0 on one and 9.4 on the other. Then use the program the target will be checked with: two programs here disagreed by 0.2 of a grade on notice D and by more than nine grades on a list.

Shorten sentences first. By the formula, each word you take off the average sentence is worth 0.39 of a grade. Find the longest sentence, look for the “and”, “because” or “which” in the middle, and put a full stop there.

Leave the words the reader has to recognise. Syllables weigh heavily: 0.1 more per word adds 1.18 grades, which in a 100-word text is ten syllables, or three or four long words. That makes it tempting to swap them out. Do it where a short word means the same thing (“use” for “utilise”, “help” for “assistance”). Do not do it to a product name, a button label, a legal term or the name of a form. GOV.UK’s guidance on the same page says of specialist terms: “You just need to explain what they mean the first time you use them.”

If the target still fails because of terms you cannot change, say so. Show the score with and without those terms.

Then check what the score cannot. Read the text once as the person receiving it and ask what they must do, by what date, and what happens if they don’t. Notice C passes every formula and fails that check on the first question.

Corrected on 8 October 2026: the first sentence described every readability score as arithmetic on words per sentence and syllables per word. That is true of the two Flesch formulas tested here. Other formulas count letters, or use a list of familiar words. The comparison of the two programs on 4,000 common words was run again with the word list now named: they agree on 3,719, where the page said 3,720.

Corrected on 9 October 2026: the 4,000 words come from the emails counted for another article, and that count was corrected today. On the list drawn from the corrected count the two programs agree on 3,722 words and differ on 278, where this page said 3,719 and 281.