"Stable Genius" - Let's Go to the Data

By Factbase Writer

Posted January 8, 2018 at 2:30am

[Note: The data has been updated as of October 3, 2019. Latest: Not “Stable Genius” Again, Or Please Stop Making Us Run This Analysis]

So, as always. First the headline, then you need to eat your vegetables to get the details.

The headline:

By any metric to measure vocabulary, using more than a half dozen tests with different methodologies, Donald Trump has the most basic, most simplistically constructed, least diverse vocabulary of any President in the last 90 years. This is by a statistically significant margin in each case.

Okay, the headline’s out of the way. On to the vegetables, so you understand why we checked this, and the methodology.

(And with our apologies for the simplistic charts. The Google Sheets plug in is quick and dirty… but the data’s all there for you at the bottom)

[gdoc key=”1HvS5jQxqrbh4u5ynv1TXUNoPesIVHQ4zq4SvPunK5Nc” chart=”Column” query=”select Q, C” title=”Presidential Vocabulary Grade Level”]

Why Are You Blogging on a Sunday Night?

Well, the Golden Globes are on. Also…

I usually try to unplug over the weekend. And by unplug, I mean “catch up on everything I was supposed to do during the week but didn’t because who the hell can get work done during office hours.” You know, by relaxing and stuff.

So the emails that started coming in Saturday morning around 8 a.m. kind of interfered with that plan. I ignored them for all of 20 seconds before seeing what the heck was going on. In general, when something is going on, the emails tend to clump together. The phone wasn’t going to stop vibrating by force of will alone.

Turned out, it was a number of folks asking if I’d seen the “genius” tweet, and if Factba.se had ever run an intelligence test.

Now, when someone emails me at an ungodly hour (and prior to 11 am on a weekend more than qualifies, given my normal bedtime is defined as “Thursday”) to ask about a tweet, I put the darker thoughts out of my mind and did my best not to get upset.

But I was awake. May as well spoil it. The tweet in question (a three-parter, which is more unusual of late since the character limit was upped):

Now that Russian collusion, after one year of intense study, has proven to be a total hoax on the American public, the Democrats and their lapdogs, the Fake News Mainstream Media, are taking out the old Ronald Reagan playbook and screaming mental stability and intelligence…..

— Donald J. Trump (@realDonaldTrump) January 6, 2018

….Actually, throughout my life, my two greatest assets have been mental stability and being, like, really smart. Crooked Hillary Clinton also played these cards very hard and, as everyone knows, went down in flames. I went from VERY successful businessman, to top T.V. Star…..

— Donald J. Trump (@realDonaldTrump) January 6, 2018

….to President of the United States (on my first try). I think that would qualify as not smart, but genius….and a very stable genius at that!

— Donald J. Trump (@realDonaldTrump) January 6, 2018

…spanning 11 minutes. (Sorry about that last one… one of my favorite Road Runners).

The quote that seemed to stick out in everyone’s mind was the last one: “I think that would qualify as not smart, but genius….and a very stable genius at that!”

Okay, I was awake.

Apparently, the intellectual exercise would be to parse the phrase “genius” and could it be proven, or disproven.

Into the Den of Snopes

Measuring intelligence is normally done through a simple method with no agreed upon standard: an IQ test, a loosely-defined standardized test, variations of which have been in use for more than a century. The most common one in modern use is the the Wechsler Adult Intelligence Scale (WAIS) v4, in use since 2008.

However, there is no peer-reviewed method to look at writing / speeches / etc to assess intelligence. The closest is a 2006 study, which used a historiometric method.

Suffice it to say, that method is fine, but it takes a doctorate and an expert. We don’t have presidential scholars at Factba.se. We’re a bunch of data schmoos. Also, this particular study was ripped off and faked enough in the past 15 years that it has multiple snopes pages (here, here, and here) and it rates its own Wikipedia page. Again, the study is fine. Making stuff up around it isn’t.

Supercalifragilisticvocabularydocious

However, the ability to measure the complexity of vocabulary, the diversity and its comprehension level is something we do all the time here in the Fact Cave, courtesy of Margaret, our platform’s AI. In fact, it’s done every time we add a word into the platform, automagically. The most common metric, the Flesch-Kincaid Grade Level, was actually developed for the military in the 1970s as a way to check that training materials were appropriate and could be understood by its personnel. It is used as a measurement in legislation to ensure documents such as insurance policies can be understood.

There are a number of competing algorithms. They use different approaches, but all try to do one of two things:

Grade Level. Establish the grade level at which the text could be understood
Reading Ease. Essentially the same thing, but with a normalized statistical score vs. a U.S.-centric grade level.

At Factba.se, Margaret runs every single bit of text automatically through the following algorithms:

… and about a dozen others, including difficult word count, etc. We’re also testing the Lexile Framework.

As a side benefit, recreationally, we built a database of interviews, speeches and press conferences for previous presidents, leaning heavily on what’s available publicly from presidential libraries, and the wonderful collections at the University of California, Santa Barbara’s American Presidency Project. One of the reasons we did this is to provide a point of contrast. Looking at a single datapoint can tell you everything and nothing. A nice cohort comparison… that’s better.

Importantly, as we’ve blogged earlier, we like to focus on a person’s own words if possible, not speechwriters. The UCSB archive in particular gave us a rich trove of Presidential press conferences back to Herbert Hoover in 1929. So we could look at just what a president said. Unscripted (or as close an approximation as is possible for a president).

Okay. We had the algorithms. We had the text. On to…

Methodology

As mentioned previously, we narrowed our samples from Hoover forward to just press conferences, presidential debates and interviews. Of course, within those, we only use words spoken by the President, nothing else.

This left us with a deep sample for each, but spread out. We ran the analysis two ways:

Complete. Whatever we have, we have. On the low end, it’s 44,705 words for Gerald Ford, up to 1,124,164 words for Bill Clinton. Trump clocked in second at 915,801 words.
Equal Sample. We then ran the same test on 30,000 words, plus or minus 1% (actual range was 30,003 – 30,253 words), where we looked only within the person’s presidency (no pre-election debates) and started from Inauguration Day forward, adding sentences until we hit 30,000, then stopped and analyzed those.

In addition, we’ve been testing the Lexile framework. It’s a free test so we’re limited to 1,000 words. But we took the first 1,000 words (in full sentence format) from the equal sample and tested those.

It’s important to note: for the two presidents where social media existed, this was not included. This was strictly utilizing the responses given by a president in an interview, during a press conference, or in a political debate.

The Result

It statistically made no difference which way we analyzed it, or which method. It affected some scores and some of the ranks, but not the position of Donald Trump on that list. In each case, he ranked last of the past 15 presidents.

By every metric and methodology tested, Donald Trump’s vocabulary and grammatical structure is significantly more simple, and less diverse, than any President since Herbert Hoover, when measuring “off-script” words, that is, words far less likely to have been written in advance for the speaker.

Significant is not editorializing. The gap between Trump and the next closest president (in most indices, Harry Truman, known historically for a folksy, simple pattern of speech), is larger than any other gap using Flesch-Kincaid. Statistically speaking, there is a significant gap.
[gdoc key=”1HvS5jQxqrbh4u5ynv1TXUNoPesIVHQ4zq4SvPunK5Nc” chart=”Column” color=”red” query=”select Q, C” title=”Presidential Vocabulary Grade Level”]

This gap appears both when using the complete corpus available to us for all presidents, and the more limited 30,000 word set to use an equal data set for each. In either data set, Donald Trump consistently clocks in at the bottom of the list. Depending on the scale used, it’s between a 3rd and 7th grade reading level.

Using the same one used by the Department of Defense, the grade level on the equal sample is 4.6. That’s between a fourth and fifth grade level.

The next closest is Truman at 5.9, followed by Bush 41 at 6.7. The top three: Herbert Hoover (11.3), Jimmy Carter (10.7) and Barack Obama (9.7).
[gdoc key=”1HvS5jQxqrbh4u5ynv1TXUNoPesIVHQ4zq4SvPunK5Nc” chart=”Column” query=”select Q, K” color=”green” title=”Presidential Vocabulary Word Complexity”]

In terms of word diversity and structure, Trump averages 1.33 syllables per word, which all others average 1.42 – 1.57 words. In terms of variety of vocabulary, in the 30,000-word sample, Trump was at the bottom, with 2,605 unique words in that sample while all others averaged 3,068 – 3,869. The exception: Bill Clinton, who clocked in at 2,752 words in our unique sample.
[gdoc key=”1HvS5jQxqrbh4u5ynv1TXUNoPesIVHQ4zq4SvPunK5Nc” chart=”Column” query=”select Q, M” color=”purple” title=”Presidential Vocabulary Word Diversity”]

So What?

That’s a fair question. So what? Vocabulary is not a proxy for intelligence. In IQ Tests, vocabulary is a component, but only a component. However, it is used as a proxy for a number of things:

Doctors use it to measure symptoms of degenerative brain diseases (note: as blogged previously, we see no downward trend over 40 years in Trump’s vocabulary. For unscripted, it’s very consistent).
Psychologists use vocabulary as a measure of intellectual curiosity and a person’s reading ability.

But also, it should be pointed out:

Politicians strive to get a clear, concise message in front of the public. That includes keeping it short and simple.

Other than Donald Trump, all presidents in this cohort were either career politicians, or in the case of Eisenhower, a very public figure and military leader for decades before running for president (historians argue whether a general at Eisenhower’s level would already be considered a politician before running for office, due to the need to navigate very political waters at that level).

Back to so what? In answer to those who emailed the equivalent of “is the president a stable genius”, the answer is “we don’t know.” Short of IQ tests, there’s no way to know for sure.

But what we can say is, compared to the 14 presidents who preceded him, by every measure, his use of words when off script are significantly less diverse, and simpler, than all presidents who preceded him back to Herbert Hoover.

As always, feel free to dispute the analysis, but come prepared with data. We don’t need more opinions. But more analysis with supporting data is always welcome.

Here’s the data. Have fun!

[Note: Hmm… thought the plug in would download all the tabs, not just one. Oh well. This is the Google Sheets link…
[gdoc key=”1HvS5jQxqrbh4u5ynv1TXUNoPesIVHQ4zq4SvPunK5Nc” datatables_page_length=”15″]