I love these so called ‘studies’ experts use to try to confuse and mislead users. Dr. Jakob Nielsen of the Nielsen Norman Group (not affiliated with Nielsen, the metrics company), conducted a ‘study’ claiming reading a book is faster than reading the same material from an iPad. The study compared reading times between the iPad and a paper back book.
The Study
The study consisted of 24 users who “liked reading and frequently read books”. That’s stop there for a moment. What does that mean? Does it mean they read an hour a night, or 5? Is it based on the number of books read per year, or is it from asking random people if they read at all? The point is, we have no idea. We have no information on the population, or how it was selected. Were they male, female, a mix? What were the ages? How about education, or geographical location? For all we know, these people were chosen by asking the first 24 people who showed up if they liked to read. This is hardly scientific.
The study compared reading times for the iPad, Kindle 2.0, printed book and PC. Each user read short stories by Earnest Hemingway on all four devices. The study was a within-subjects study, meaning each user read from each device, and the times were recorded for each. One can only assume different stories were read on each device, however the study lacks this information. Each user was given a questionnaire to make sure they read and understood the stories, however this questionnaire and data was absent from the study. The authors state all user’s scores were similar, however we have no data to validate this claim. Multiple-user Simultaneous Testing (MUST) was used, meaning multiple participants were tested at once. This opens a door for flawed data, as it is hard for a researcher to monitor participants at the same time. This is the least of the problems for this study, as the main one is the users within the study.
Users
The test specifies a total of 32 users, with 5 removed for program testing. 3 were discarded due to measurement flaws, so the grand total remaining was 24. With less than 30 participants, it is impossible to determine if a result is significant. In statistics, you need at least 30, or you cannot suggest a significant difference occurred. Thus, no matter what the results stated, you cannot state that reading from an iPad is slower than reading from a book. Without a proper sample size, your study is invalid. This is the true problem with this study. Even if the study had 30, the power level is so low it would be difficult to justify any significance. A sample size of 100 is more appropriate.
Each user was given a REALM literacy test. This helped to determine if each user was at an appropriate reading level. The test asks participants to read words of varying difficulty and scores them on the number of words mispronounced. For this study, two people failed to pronounce one word, suggesting at least a high school literacy level. However, again no data was provided to prove this statement.
Results
The iPad measured at 6.2% lower reading speed when compared to the printed book, where the Kindle measured 10.7% slower. As a result, the findings are not statistically significant, which the study does specify. Thus, the results do NOT state reading on paper is faster then reading on an iPad! If something is not scientifically significant, then you cannot pretend it is. Shame on you, PC World, and all you other media sources, for misleading your readers to think this study is true. Then again, it got people to read it, so who cares if the results are not true?
To Conclude
To end this pathetic example of science, no delimitations were present. A delimitation is where the authors point out flaws in the study, and suggest ways of improving it. I can think of a few ways: increase the same size and disclose your data! I do not expect PC World or any news outlet to give all the facts, but I expect them to report honest research. The flawed study even specifies the data was not significant, yet we have article titles stating reading from paper is faster then reading from the iPad. The sad truth is, I bet every single study we read from news organizations is similar. Next time you read a claim on a website, read the scientific study, and question it. Questions are what science is all about.
To read the study for yourself, click Here.
Article Source: Click Here.
About the author: Psychcomp is owned and maintained by Nathan Driskell, a Licensed Professional Counselor - Intern specializing in Internet Addiction and Asperger's Disorder. Nathan is also a Web Designer and Network Administrator.
Contact Nathan at the following locations:
dami...@psychcomp.com">E-Mail
Twitter
Related Posts
- Get Listed in Google Faster
- Have you Read the Health Care Bill?
- NCE: Review and Impression
- President Obama’s Internet Kill Switch
- Book Review: Lightning by Dean R. Koontz
Tags: iPad, Kindle, Scientific Study
Loading ...
I like the post with such a nice material which is much informative. Thank you for the information you provided.
You are welcome. Anytime I read some fact from a study I like to read the source material. If news organizations reported on the true facts from a study, there would be little news of importance most of the time, which is why they do not.
You’re almost-but-not-quite right about the sample size being an issue. Perhaps you’re keeping it simple, but that simplification also limits your own accuracy.
Power mainly concerns the ability to detect a significant effect when one is there to be found. When no significant effect is found it is worth highlighting low power due to a small sample size. However when a significant effect is found there is little value in criticizing the power of the study – it found an effect.
Another point about power and sample sizes. The type of test performed has a considerable impact on the power given a particular sample size. The test performed was within-subjects, which is typically more powerful than a between-subjects design. Presumably they performed a repeated measures ANOVA. In that case fewer than 100 participants might be just fine. As an example, with an effect size of 0.25, a power level of 0.8, an alpha level of 0.05, 1 group, and 4 repetitions, the required sample size would be 24. Since we don’t know the effect size or the actual power, we can’t say for sure that their sample size was too small.
So basically, lack of power wasn’t necessarily the issue, and to conclude on the basis of the available data that the study lacked power is, well, bad science. Sorry. You’re probably right, I doubt their true power was 0.8 and the effect size was 0.25, but we just don’t know, so we shouldn’t make any definite claims.
Still, more participants would make the results more reliable, though the emphasis there is on the high variability of the data, not on the lack of power. This may be due to some of the things you rightly criticized in the section titled “The Study”. Increasing the sample size won’t necessarily fix those issues.
Also, in your comments regarding the results you said, “The iPad measured at 6.2% lower reading speed when compared to the printed book, where the Kindle measured 10.7% slower. As a result, the findings are not statistically significant, which the study does specify. Thus, the results do NOT state reading on paper is faster then reading on an iPad!”
The relevant lack of statistical significance which the study specified was the lack of significance between the two devices, not the difference between the iPad and paper. The results do state that reading on paper is *marginally* significantly faster than reading on the iPad, at p=.06.
Ultimately, the main problem with the study as shown is that is doesn’t provide enough information for us to evaluate the quality of the study. What we have in the PC World article and Dr Nielsen’s Alertbox article is uncritical reporting and a lack of transparency, not “pathetic” science per se.