Explaining SSD reviews by the Tech Media

The short version is some reviews of Solid State Drives online can be poor and uninformative. They relay benchmark results generated by sythetic tests that provide the end user no or little indication as to what improvement in user experience and responsiveness they would get from a SSD upgrade. While HDD to SSD can be demonstrated relatively easily in practice, SSD to SSD is much harder for a practical demonstration of user benefit. Worse so, the typical consumer expects 'more space, more gigabytes' on their next new PC purchase to store bulk data such as photos, video, music and iPhone backups and would be dismayed if presented with a storage solution (SSD) that is much smaller as it is perceived as a downgrade regardless of the massive speed increase.

Synthetic tests such as CrystalDiskMark, ATTO, AS SSD and Anvil provide metrics applicable to power users and not the general public, even to some IT professionals. What message does drive A doing 540MB/s versus drive B doing 535MB/s say to anyone? the real difference is so small it would not be noticed in real world unless a massive transfer occurs where this difference can save many minutes. 

The PC hardware media has been reviewing client SSDs for almost ten years now, and the first PCs from OEMs to offer SSDs as a build to order option are now almost 10 yrs old, dating from the very late 2000s. Some of these may even be in landfill now. What has been forgotten already that were attempts to market SSDs on the PATA Interface, also known as IDE. We reported on this drives here. At the time the ATTO Disk benchmark was used to demonstrate those drives to us and is still used to this day. ATTO is and was a smaller vendor of RAID controller cards and enclosures and has relied on their free benchmark for continuous publicity of their brand.

ATTO is a synthetic disk benchmark that uses the standard windows file APIs to read/write a test file to an already formatted and mounted disk partition, emulating the behaviour of any typical windows application accessing a disk.

It is one thing for a vendor to select a particular benchmark to demonstrate a drive in person or in a document for marketing purposes, it is another to base the drives specifications on the results achieved by ‘popular’ benchmarks used by media reviewers and end users.

The specifications you see that are listed on the SSD’s box, datasheet, marketing flyer or spec sheet are actually derived from common benchmark programs. This is the case for the vast majority of vendors in the industry. Tools used to determine these scores include the now old PCMark Vantage from Futuremark, ATTO Disk Benchmark, and the open source IO meter created and formerly curated by Intel and FIO tool for Linux which is typically used for enterprise drives.

Now guess what, typical hardware reviewers (including myself, I am happy to admit) will use these tools plus CrystalDiskMark, AS SSD and Anvil Disk Benchmark to benchmark drives they have for review, and what will the result will be especially if they follow a review guide provided by the vendor? The result will replicate the advertised manufacturer specs. The reviewer has now followed an infinite loop of proving a vendor claim without adding extra value to the information he/she is presenting to their readership.

A competent and respected reviewer should be able to add value to the commentary about a product or service they are providing to their readership and the industry as a whole. To answer questions to their readership 'why is this better' or 'how is this better' not 'it just is better'.

If vendor X clearly states in the footnotes of their datasheet they achieved their specs using IO meter and a reviewer replicates this, then so what? That’s not much different to someone checking if a 17” wheel is actually 17 inches. Verifying a vendors claims only goes so far and is only one part or a component of a thorough product review which provides value, direction and a message to an end user.

There are many tech review websites with hundreds of thousands of followers on their socials and native readership and many of them are very cluey in their chosen field, so why do don’t they do SSDs differently, well threes several reasons:

• Technical knowledge - not all ‘hardware reviewers’ have professional IT or electronic engineering knowledge.
• Variability in testing – depending on scope of review outlet, may vary between a simple mention in a roundup to a very detailed technical analysis
• Time – turnaround time for reviews is constantly getting shorter. It is often excepted a product review is done within a matter of a few days and it’s not uncommon for only a few hours to be spent on something.
• Interest – such reporters who have a specialization or interest in Business IT may not care about commodity client hardware

Depending on the reviewer and outlet, these caveats may be acceptable as different media outlets have different areas of focus and specialization. For instance outlets like CNet, ZDnet and Gizmodo have a generalist approach to consumer or business tech and would present an overview as to why an SSD is beneficial over a hard drive and what brands are available.

Whereas sites like Anandtech, PC Perspective, Toms Hardware, The Tech Report, The SSD Review and number of specialist European sites will go right through down to the component level, talk about how one drive has a small power advantage, use custom hand written software to test drives or even perform yearlong torture tests to determine the true endurance of SSDs versus the manufacture claim.

Speaking for myself. I want to tell the big picture and answer specific questions about the product I am writing about such as Is this device faster, why is it better, how is it different, what engineering has gone into the device to make it different or better to other or older devices. Speeds and Feeds can be obtained from the vendor. Theres not much point for us to copy and paste the vendors spec sheet to pad an article.

For SATA drives, even with custom written software there is only so much story to tell. Using the operating system’s own file functions we will typically see trends between the different benchmarks

What media can do better is come up with different scenarios to test drive endurance as well as demonstrate the overall experience difference between different model drives. How does a drive compare in game loading workloads, what about Adobe Creative Suite/Professional Content Creation, disk imaging or benchmarks like BAPCO’S SYSmark, a professional test which tests a number of typical software scenarios using real copies of these typical software in a simulated environment and gives a reference score. Now SYsmark does have some criticism in the industry but so does everything, mainly from vendors who do not want to pay membership fees to be involved in the development of such software.

PCMark also provides such trace based testing, but manufacturers rarely spec their drives with the newer versions as they no longer solely test storage.

One question I hear alot in the community is the affect of an SSD on game load times.

There is also a role for manual file copy tests. This is simply a timed test manually using a watch or helper utility to assert the total time taken to copy files to/from the SSD or to even compress files via ZIP/RAR. The issue with this type of testing is the wide variety and variability of windows systems, with the specific environment and hardware configuration affecting the result. Whereas with a specific benchmark utility there is more control over the end result especially with the tools from Bapco and Futuremark. The copy test answers a very important question, how well does my SSD copy files of varying sizes in the real world. Entry level SSDs such as Crucial’s BX200 and OCZ’s Trion and several others which used Planar TLC NAND and a weaker controller and cache setup were caught out by such tests, where certain large files such as a typical several gigabyte video file overwhelmed the SSD. Even on the other end of the spectrum, Samsung’s enthusiast/performance oriented 960 EVO ‘only’ has a 48 GB cache for the 1TB model, meaning extremely large files will be transferred at lower than advertised speeds.

In their recent tests, Anandtech have resorted to measuring power consumption for some specific file I/O workloads. I am not sure this is relevant or that people care what the power consumption of a SATA drive is. For a NVME drive in the M.2 or PCI-E Add in Card Format this is more relevant as heat generation is a product of power and a cooler running controller means better performance from less throttling. For a NVME drive this does make significant differences. Smaller, less power consuming silicon can also be cheaper to make and sell. On an engineering curiosity level this information is interesting and useful but the power consumption of SSDs generally is 'resonable' especially given the small PCBs many use, their cost and performance.

Allyn Malventano at PCPER wrote his own custom test software which is going beyond the call of duty to ensure repeatability between drives and to test different weighted workloads. But again this approach has it's limitations. It is proprietary and does not answer questions about experience with real world applications and scenarios.

PC hardware reviews need to strike a balance of being concise, delivering scientific test results and relaying the right message to their audience. The majority of publications get one of more of these right but rarely does one get all of these right. It is all of our responsively to strive to be better but unfortunately some more mainstream household name 'technology' blogs (Not PC hardware) don't take this as seriously as they see their role in the grand scheme of things as news reporters not testers or analysts. The manufacturer is always right (!) and they think it is their job to be completely objective from the product they are writing about. Diving into the depths of a product is not seen as objective to some.

I am constantly researching and testing different methods of testing SSDs and this takes alot of time. I have to present something in the meantime therefore there is a typical mix of benchmarks in this review. More applicable real world use cases not only provide more relevance and meaning to what I am reviewing, but may even be easier to test on a repeatability point of view. 

End users want questions answered such as 'will my game load faster on a SSD', indeed some even swap in a SSD for their consoles. It is the job of a professional review to answer this question or at least to best of ability.  This review would even have been more accurate if I had compared the same capacity for all drives tested but that simply is not possible due to what drives we have on hand or have been sampled by the manufacturers.