Menu
Does Bionic Reading actually
work? We timed over 2,000
readers and the results might
surprise you
Daniel Doyon | 19 Jul 2022
tl;dr. Actually no, the results will probably not surprise you. After analyzing
data from 2,074 testers, we found no evidence that Bionic Reading has any
positive effect on reading speed. In fact, participants read 2.6 words per
minute slower on average with Bionic Reading than without. That said, the
difference here is so small (less than 1%), that the real takeaway is Bionic
Reading has no impact on reading speed.
In June 2022, we posted an experiment to Hacker News, Reddit, and
Twitter to test the claim that Bionic Reading enables you to read faster
without any loss of comprehension. Part 1 of this series described in-depth
the motivation, the claim, and the experiment design. In this Part 2, we're
jumping right into the results.
Preliminaries
Participants were asked to read two 1,000 word essays divided into two
halves resulting in articles 1A, 1B, 2A, and 2B. The first font was randomly
selected for each tester (either Literata plain, or Literata with Bionic
Reading applied) and then alternated from there.
Menu
The selected essays were written by the same author (Paul Graham) during
the same time period (circa 2010) to minimize variation between articles.
The matrix setup depicted above was intended to further control for both
inter-article variation (article 1 might be easier/more interesting to read
than article 2, or vice versa) and intra-article variation (the beginning of
each article might be easier/more interesting to read than the end of each
article, or vice versa).
When the cross-matrix reading speeds for each font are averaged, we have
a pair of reading speeds per participant that can be fairly compared against
one other, isolating the font as the independent variable. We also asked
three multiple choice questions at the end of each article half to control for
reading comprehension.
Data preparation
As with anything on the internet, completion rates followed a predictable
funnel pattern. 3,334 participants completed article 1A and this sample
gradually dwindled to 2,074 participants who completed all four.
Menu
There are advanced statistical techniques, such as a random effects model,
that can take advantage of all the data regardless of whether each
participant completed all four halves or not. But for the sake of
interpretability, we've decided to disregard any incomplete entries resulting
in an initial sample size of n=2074 — still more than enough data to
power a robust statistical analysis.
Also as with anything on the internet, there is some junk data which must
be removed.
* First, we removed any multiple entries which came from the same IP
addresses (37 entries).
* Next, we removed what I call "tirekickers" — curious cats who started
the experiment, rapidly scrolled to the bottom, and clicked next to see
what happens. We can identify these participants by reading speeds that
are humanly impossible, e.g., 2 seconds to get through 500 words. For
our purposes here, we used a cutoff of 2,000 words per minute (95
entries).
* Finally, we removed what I call "quitters" — participants who started in
earnest but then gave up near the end and rapidly scrolled to the bottom
to see their results. For our purposes here, we can identify these
participants by reading speeds on 2A or 2B that are 2x greater than their
average previous reading speeds on 1A, 1B, and 2A, respectively (20
entries).
This data preparation resulted in a final dataset with n=1916 .
Menu
We appreciate that outlier removal is a delicate process that risks
introducing bias to the dataset, but we feel confident based on our
observing of participants and our subject matter expertise that these entries
are invalid. Further, we've made publicly available the anonymized raw
data (pre-munging) here in the event you wish to apply your own data
preparation steps to run your own analyses or to check our work.
Average speeds per font
As mentioned above, we can now average each participant's results for each
font to derive a pair of speeds that can be compared apples-to-apples. Let's
quickly explore the summary speed statistics for each font.
Table 1: Summary speed statistics per font (WPM)
Bionic Non-Bionic
Mean 325.3 327.9
Median 294.0 289.6
Standard Deviation 134.6 148.5
Minimum 60.9 100.2
Maximum 1,118 1,293
Count 1,916 1,916
As you can see, the average speeds for both fonts (325.3 and 327.9 words
per minute, respectively) were virtually identical with non-Bionic Reading
actually clocking in approximately 3 words per minute faster. In percentage
terms, however, this is a negligible 0.8% difference.
Average speed differences per Menu
participant
The beauty of this experiment is that we can perform various "paired"
analyses which control not only for reading speed variation across articles
and article position, but also reading speed variation across individuals.
Observed differences, on average, should be caused by the font rather than
any other lurking factor.
The simplest way to look at differences is to subtract each participant's non-
Bionic Reading speed from their Bionic Reading speed. If Bionic Reading
helps people read faster, we would expect to see a mean difference greater
than zero words per minute.
Table 2: Summary speed difference statistics per user (WPM)
Delta
Mean -2.6
Median 1.8
Standard Deviation 58.2
Minimum -439
Maximum 293
Count 1,916
In plain language, this says the opposite: participants in our sample read on
average 2.6 words per minute slower with Bionic Reading than without.
Average speed differences per faster
font
Since posting this experiment, I've received a lot of side comments along
the lines of, "Well, of course I don't expect Bionic Reading to work for
Menu
most people, but for [my subpopulation], it really works." If that were the
case, we might expect to see disproportionate benefits for those participants
who read faster with Bionic Reading than for those who read faster without
Bionic Reading. Let's look at how many participants read faster with each
font and their average speed gains.
Table 3: Summary speed differences per faster font
Count Percent Delta (WPM)
Bionic 998 52% 35
Non-Bionic 918 48% 43
The number of people who read faster with Bionic Reading was slightly
greater (52%) than the number of people who read faster without Bionic
Reading (48%). That said, those who read faster with Bionic Reading only
picked up 35 words per minute on average. In contrast, those who read
faster without Bionic Reading picked up 43 words per minute. It does not
appear that when Bionic Reading works, it really works.
Hypothesis test
Now let's run a paired t-test to test the null hypothesis that the average
speed with Bionic Reading equals the average speed without.
Table 4: Paired two sample t-test results
Bionic Non-Bionic
Mean 325.3 327.9
Variance 18,108 22,055
Observations 1,916 1,916
Bionic Non-Bionic
Menu
Hypothesized Mean Difference 0
df 1915
t Stat -1.9196
P(T<=t) one-tail 0.0275
t Critical one-tail 1.6456
P(T<=t) two-tail 0.0551
t Critical two-tail 1.9612
In statistical terms, the mean speed difference (M = -2.6, SD = 58.2, N =
1,916) was not significantly different than zero (t = -1.92, two-tail p-value =
0.055), meaning that we fail to reject the null hypothesis. A 95% confidence
interval about the mean speed difference is [0.1, -5.2], nearly implying that
Bionic Reading actually has a negative impact on reading speed.
In plain language, we almost got a statistically significant result suggesting
that Bionic Reading is slower. As mentioned in the introduction, however,
we're talking about such small magnitudes here (less than 1% difference)
that you can probably walk away with the conclusion that Bionic Reading
simply has no effect.
Comprehension
Up until this point, we've been testing reading speed without regard for
reading comprehension. Let's quickly explore the summary comprehension
statistics for each font.
Table 5: Summary comprehension statistics per font (% correct)
Bionic Non-Bionic
Mean 88% 88%
Bionic Non-Bionic
Menu
Median 100% 100%
Standard Deviation 16% 15%
Minimum 0% 17%
Maximum 100% 100%
Count 1,916 1,916
Like reading speed, comprehension across fonts was virtually identical. In
fact, these comprehension numbers were so close that I feared I messed
something up. That said, I triple-checked my calculations and they're
correct.
These results correspond with common sense and the existing body of
research showing that reading comprehension is inversely correlated with
reading speed. The faster you read the less you remember and, conversely,
the slower you read the more you remember. Because the reading speeds
for both fonts were nearly the same in this experiment, we'd expect to see
similar comprehension, which is exactly what happened.
Conclusion
Based on the data collected, it's hard to find any evidence whatsoever that
Bionic Reading has any impact on reading speed or reading
comprehension. Instead, our results seem to corroborate prior research that
there's no universal best font, but idiosyncratic configurations might lead to
gains on an individual level.
For more on this, see Accelerating Adult Readers with Typeface: A Study
of Individual Preferences and Effectiveness and Towards Individuated
Reading Experiences: Different Fonts Increase Reading Speed for Different
Individuals (both by researchers at Adobe).
Based on the above, you might be thinking: Well then who cares whether
Bionic Reading truly lives up to its claims so long as some people like it?Menu
Who cares if it's probably just a placebo? It's just one more font. Give the
people what they want!
One thing that might not be obvious is that despite my casual usage of the
word "font" herein, Bionic Reading is definitely not just another typeface
that can be simply licensed and installed. Instead, it's more of a font style
that requires making an API call to the diligent patent/trademark holder
(and remunerating said IP holder) followed by the injection of all kinds of
tags into your markup.
To give you an idea of what this looks like, here's the first sentence of a
Paul Graham article with Bionic Reading applied:
1 <p><b class="b bionic">Th</b>e <b class="b bionic">bes</b>t <b class
Here it is without:
1 <p>The best way to come up with startup ideas is to ask yourselfthe
Menu
As software developers, is this a dealbreaker? Of course not. But it sure is
cumbersome, particularly when you're building an offline app, so we'd
prefer a modicum of evidence that the juice is worth the squeeze before
introducing all this cost and complexity.
Another thing that might not be obvious is that — despite any skeptical
undertones that might come through in these blog posts — we were
sincerely hoping that Bionic Reading would be proven effective. We're in
the business of reading technology, after all, and we genuinely believe that
software has the potential to revolutionize the practice of reading. We're
seeking any advantage digital reading might offer over its analog analog to
persuade people to make the switch from paper to pixels. This is our
mission and the reason we're building reading software in the first place.
For this reason, and because we had so much fun designing and executing
this Bionic Reading experiment, we intend to run some more tests in the
hopes of discovering a screen reading technique that yields material
benefits. We're aware of a couple other technologies that seem interesting
including BeeLine Reader, Spritz, and Sans Forgetica.
Are there any we're overlooking? Let us know on Twitter or Hacker
News!
Otherwise, stay tuned for our next reading speed experiment 🤓
