When your web browser or a game crashes, you are likely to consider it a bug in that software. In reality, however, you may have faulty or unstable system memory without even knowing it. Unlike HDDs or SSDs, data in RAM is not protected against corruption. And it appears that unstable memory can silently cause computer trouble far more often than we think, potentially even putting your valuable data at risk.
Mozilla Firefox developer Gabriele Svelto has now published quite concerning data on how often can system memory cause issues. He was addressing a problem where crash reports sent by Firefox sometimes show no obvious cause that could explain the error, raising suspicion that the user may simply have had an unstable system (meaning instructions return incorrect results or data in memory changes spontaneously).
To investigate this, Svelto wrote code that runs after a Firefox crash and quickly tests whether data stored in system memory is being corrupted spontaneously—i.e., whether values occasionally change (so-called bit flips), which can break program code and lead to crashes or other incorrect behavior. A bit flip in program code is actually the better-case scenario. If a bit flips in say your photo or another file with your data, which you then save to disk, you may only discover upon reopening it—too late—that the file is corrupted and your data has been partially or completely lost.
The Firefox code only performs a quick test, checking just 1 GB of memory and not running longer than three seconds. However, after deployment it turned out that even this limited approach detects an unexpectedly high number of cases where the browser likely crashed due to unstable memory or another hardware fault (because the root cause does not have to be the memory module itself—it could also be the motherboard or a malfunctioning power supply).
Errors caused by RAM may be behind a large share of application crashes
Svelto states that in one week, the telemetry system collected about 470 thousand Firefox crash reports originating from various causes (note that Firefox sending reports and crash data is opt-in, so this is not all crashes). Of these, around 25 thousand cases were identified by the testing code as involving unstable memory (bit flips were detected), making it the likely cause of the crash. Because detection is only quick and simple, this is almost certainly a heavily underestimated lower bound. According to Svelto, a conservative estimate would be that around 10% or possibly even larger share of all crashes are actually caused by faulty memory.
In reality, the share is likely to be even higher, because modern web browsers often crash due to running out of available RAM when many tabs are open (note: this happens especially if you have manually set page file size, which used to be a commonly recommendation; today this is more likely to cause problems, so it is better to let Windows manage the paging file automatically—it will improve system stability). According to Svelto, after excluding out-of-memory cases, the share of crashes caused by unstable memory likely exceeds 15%.
A much bigger problem than is assumed
In any case, developers expected the share to be much lower before deploying this tool—they did not anticipate that unstable memory would be such a widespread issue. Most affected users are probably unaware that their system is unstable. This may be because they are running an overclocked CPU or memory (which includes using XMP or EXPO profiles that are often presented as “guaranteed to work” but they may not always work with 100% stability across all motherboards and processors).
Sometimes the hardware itself may be defective even without overclocking and should be RMA’d. Unfortunately, commonly available stability testing tools today are far from reliable—Memtest86+ often reports everything is fine even when the system crashes or freezes due to instability or incorrect memory operation. Overclockers and performance tuners likely underestimate how insidious CPU and memory instability can be, running systems with tweaked settings that actually lead to subtle errors.
Svelto also notes that instances of faulty RAM are not detected by this tool only on Windows PCs. A significant number of reports reportedly come from current Apple laptops with their own Arm-based M-series processors, which use memory integrated on-package. This means that even tight integration of CPU and memory in MacBooks does not guarantee stability.
For all laptops with soldered memory, instability is of course a serious issue because the memory cannot be easily replaced. Chips mounted directly on a laptop motherboard can be replaced with proper equipment (desoldered and swapped), although this is not something typically achievable by amateurs. In the case of Apple laptops with the on-package memory, this is likely even more difficult and will probably always require service from an official Apple repair center in practice.
Faulty memory naturally does not affect only Firefox—the user of such a system will experience instability or hidden issues in other programs as well. Even the operating system itself may become unstable. The cases where Firefox crashes due to such an error and the reporting tool actually does catch and report the issue, are merely the tip of the iceberg.
Unfortunately, this is a problem that doesn’t have any easy solution at present. Standard consumer PCs and mobile devices typically use memory without ECC data protection, which is a feature that could detect and partially correct errors if it was available. This is precisely why we do not really know how often these issues occur, and we risk losing our data or results of whatever work we are performing on the computer because of them. The only partial mitigation is to avoid overclocking, avoid overly aggressive performance tuning, and possibly even avoid enthusiast-grade components that might be pushed to extreme clock speeds near stability limits; maintaining good cooling and avoiding extreme temperatures may also help.
Will DDR5 help, or do we need proper server-like ECC?
It is worth asking whether these cases involve mainly older DDR4 memory. DDR5 technology has introduced partial ECC protection, which could help in theory. The problem is that so-called on-die ECC implemented in the DDR5 standard operates only internally within memory chips and does not pass information about errors to the CPU or operating system. Errors may be corrected, but not always—especially if multiple bit errors occur and redundancy mechanisms cannot handle them.
Additionally, data transfers between memory chips and the CPU are not protected by DDR5’s within-package ECC, and this is actually likely to be a major source of errors often occur that silently corrupt transmitted data. These errors introduced in trnsit may even be more likely than bit flips occurring directly within the chip itself, as modern memory interfaces operate at extremely high speeds and are highly sensitive to signal quality and interference.
For these reasons, on-die ECC in DDR5 modules likely has only limited effectiveness and cannot replace full ECC memory as used in servers and workstations. Without it, we may never achieve proper reliability on PCs. Hopefully, ECC will become standard in one of the future generations of RAM technology…
Source: Gabriele Svelto
English translation and edit by Jozef Dudáš
⠀
