Protect the Raspberry Pi 4 SD flashcard.
Last updated: June 19, 2022
Introduction.
In this tutorial, the Raspberry Pi 4 SD flashcard is examined. Over time flash memory will wear out. Ultimately it becomes unstable and unreliable. Fortunately, some techniques can significantly extend the life of flash memory. They will be discussed later, but let us first investigate the cause of this behaviour.
Theory.
At the very heart, a flash memory cell consists of a single MOSFET (metal-oxide-semiconductor field-effect transistor) with an extra isolated gate. This floating gate, made of electron friendly material, works as a container for electrons.
Applying a positive charge to the gate attracts electrons into the body just below the insulator. This way, a conductive tunnel between the source and the drain is created. The state becomes 'on'. Removing the voltage on the gate and the MOSFET turns off again.
Extra electrons placed into the floating gate will give this gate a negative charge. It will shield off the effect of the controlling gate. Now, to turn on the MOSFET, a significantly higher voltage is needed.
With the correct voltage on the gate (Vtest), one can test if there are electrons present in the floating gate.
Without electrons in the floating gate, the MOSFET will conduct, will be 'on' or '1'. If there are electrons, the MOSFET will block, it remains 'off' or '0'.
Once electrons are trapped in the floating gate, they cannot escape due to the insulator. Even if the power goes off, they remain in the floating gate. It is the basic principle of flash memory; electrons trapped on an isolated island inside the chip.
Triple Level Cell.
If the number of electrons in the floating gate varies, the influence is inversely proportional. With the correct amount of electrons injected into the floating gate, one can quantify eight different levels. The bottom graph shows the phenomenon. By sequencing the gate voltage from V0 to V6, the state of the cell is determined. With voltage Va at the gate, the MOSFET conducts always, no matter how much electrons are trapped inside the floating gate. This technique triples the capacity of the flash memory without additional transistors, hence the name.
Reading.
Most flash memory nowadays is organized as NAND. See the schematic picture below.
Here, eight MOSFETs (Q1-Q8) placed in series; they create a logical NAND operation, hence the name of this kind of flash memory.
Each MOSFET is driven separately by a control signal. This signal drives many gates. In the picture, only six are shown for clarity. By controlling many cells simultaneously, a single cell (bit) cannot be read. With many computers being 32 or 64 bits, this is not a problem.
In the picture, all gates have a voltage level of Va, except the Q4 line. A staircase signal is used to determine the state of the MOSFET Q4. When the staircase signal comes above the trigger level, Q4 will start to conduct. Remember, all other MOSFETs in the chain are conducting due to the Va level. The exact time at which Q4 conducts is equal to its state and determine the digital output.
Erasing.
Erasing a flash memory is done by removing the trapped electrons from the isolated gate making the state '1'.
The source, drain and substrate are all connected to a high voltage level. The gate is kept at ground level. The difference in voltages creates such an energy that the electrons inside the isolated gate are pulled out to the substrate by the Fowler Nordheim quantum tunnel effect. You cannot say that the electrons travel through the insulator, they vanish on one side and pops up on the other side as if they travel through a tunnel.
By powering the substrate, all the cells are erased at the same time, hence the name 'flash'. In practice, the flash memory is organized into regions, so it is possible to erase a single block, of say 4 KByte, instead of the entire chip.
Writing.
The writing of an individual cell is done by pulling electrons inside the gate if the state has to be '0'.
By applying a positive voltage on the drain, electrons are travelling through the substrate, just below the gate, from source to drain. If the voltage at the controlling gate is high enough, electrons get such a high energy level that some are pulled inside the isolated gate by the same quantum tunnel effect as during erasing. With a triple architecture, the number of electrons in the insulated gate becomes paramount. After all, they determine the analogue level at which the MOSFET will conduct. Pulsed writing is used to inject the correct amount of electrons. After a short write pulse, the state of the MOSFET is read. When the level is not reached, a new write pulse injects a few more electrons into the gate until it reached the correct level. Needless to say, this takes some time.
Garbage collection.
Storing data in a flash memory happens somewhat strangely, since setting the cells to a '1' can only be done by erasing whole blocks. You always start with an empty block. Since all cells are set to 1, you only need to reset the zero bits as described above. Once the data of a file has been written, all blue cells are assigned in the image below.
When the same file is updated with new data, the only way to write the entire file from scratch is in new empty memory. It leaves the previous memory as waste, the light blue areas. The next new file is placed in the following free cells, marked orange. Again, updating this file will take up new free memory and leave some garbage. If there is not enough free space, the whole block is copied to a new block, and the old one is completely erased, as shown in the last row. As a result, data cannot be found in a fixed place but travels over memory locations. It means proper administration to keep track of all the files. Copying data blocks also slows down the data transfer speed. An SD card can be sold with a capacity of 80 MB/s, but this speed is only achieved with a continuous video stream. When it comes to a Linux operating system with its many different files and random memory calls, throughput can drop dramatically. Sometimes more than 100 times.
Microcontroller.
From the above, it is clear that flash memory requires complex and dedicated processing of timing and memory allocation. Things only a microcontroller can do. In the small housing of the SD micro card, you will always find a pretty heavy microcontroller busy streamlining the data transfers with the outside world, the internal writing, the erasing and block movements. Not to speak of error checking and bit correction routines. With this microcontroller continue working at the background, it's now clear why a sudden power cut can corrupt your SD-card, or even let it crash beyond repair.
Limited write cyclus.
Another cause of flash memory failure is the wear that occurs over time. In theory, all electrons should be in the insulated gate forever; they cannot escape. It is, of course, not the case. SD flashcards have mostly a seven-year warranty. If you have programmed your flash only once, the supplier guarantees that the content is correct during this period. After that period you can use the SD card for years, but over the years the chance of deterioration does increase.
Any writing and erasing action will significantly shorten this period. Most of the flash nowadays is TLC (triple-level cell) and will only be programmed ~1000 - 3000 times. Above this number, flash memory is no longer reliable. Keep also in mind the additional erase actions due to garbage collection.
It has everything to do with the quantum tunnel effect. During the writing or erasing phase, there is such a high voltage level that some electrons are tunnelled accidentally inside the insulator itself and remain trapped there. It changes the threshold level of the MOSFET. The threshold bands are very narrow with the TLC flash. Because of this, only a few electrons can change the threshold level and cause an error.
In theory, these electrons could be removed by inverse tunnelling (erase phase), but the voltages required for this action far exceed the levels MOSFET can withstand. It certainly will destroy the transistor.
SSD.
A few words about SSD (Solid State Drive) memory. The functionality is identical to the SD flashcard; it works with the same MOSFETs and their floating gates. The only difference is the microcontroller. SSD uses a much more complex controller. It has a large RAM buffer to ensure high bandwidth with the outside world. The controller also keeps track of the times a block is cleared. It uses a rotating schedule that uses all blocks evenly, distributing the wear. It can detect faulty blocks and flag them out of use. And the controller has advanced bit correction algorithms, making the SSD a suitable replacement for a hard drive.
Pratice.
How bad is all this?
It can be quite a serious problem. Even if you're not actually saving anything to the SD flash, there are still many background writes. The Linux operating system also likes to keep track of many actions. Or if you are browsing the net, a lot of files are being written (history, caching). There are reports of kiosk applications (an app showing 24/7 an infotainment site on a large screen) whose SD flash crashes within a year. If you use your RPi as a desktop with a good quality SD flash card, it should last for at least two years. Apart from power outages, of course, because that can destroy any memory storage system.
What can you do?
You can accept your loss and hope for the best. Make copies of the SD card regularly and buy a new one if it crashes. They are relatively inexpensive. You'll likely be using two to three cards before the next Raspberry Pi 5 hits the market, and we're all saying goodbye to the Pi 4.
Or you can replace the SD card with a USB SSD drive. As explained above, they have many more write cycles due to distributed wear. And their capacity is much greater; another reason why they last for many years. It costs only at least another € 50 in hardware, the price of 5 SD flashcards. More information on our page here.
The option we often use is to make the SD card read-only. The wear only occurs during writing. Reading the SD card does not affect its lifespan. If no write or erase operations are performed on the flash, it can retain its integrity for many years.
SD read-only.
It is possible to use an overlay. In this way, all reads are still performed from the SD memory. All writes are redirected to RAM. By using the same folder structure as the SD card, everything remains transparent. You don't notice any difference. It's like you're still working with the SD flash. And by making the SD card read-only, sudden power cuts can't destroy your card either. However, as soon as you turn off the Raspberry Pi, all your saved work is gone. After all, it was using RAM. Not so bad for your browsing history, worse when it comes to time consuming labour.
Applications that store large amounts of data on disk can cause problems due to the limited RAM on a Raspberry Pi. A web browser that caches entire pages will consume all RAM within an hour. Think carefully if the SD read-only is best in your situation.
Some applications have settings files that need to be changed from time to time. Think of a kiosk app that another site needs to show. Or another lens in a vision app that requires a different DPI setting. An extra partition will be created on the SD card for this purpose. This small partition will be the only part with write access. Use it only as a directory where settings files can be kept, not for storage of other (daily) work. Use a USB stick for that matter.
Before creating this read-only overlay, there are three things to keep in mind.
- Keep in mind that after your overlay works, changes to your app are no longer possible. Otherwise, you have to undo the entire mechanism before any changes can take effect. By the way, it also applies to the operating system; it will no longer receive updates.
- If you want to use an extra partition, create it according to the instructions below before working on your overlay.
- By using the read-only overlay, the swapping software is forced to use the limited RAM now. You have to decide what to do with the swapping mechanism. We sometimes use ZRAM, which creates extra space in RAM through compression techniques.
New partition.
An extra partition on the SD card can be useful to keep specific files like settings. Raspi-Config doesn't let you change or shrink partitions. It requires special software and a machine with an SD slot, to alter a specific partition on the SD card. GParted is the tool in place here. Install GParted on Linux with a simple command ($ sudo apt-get install gparted). On Windows machines, it needs an external USB stick where GParted is installed together with a little Debian distro. After carefully follow the instructions of method B on their site, your Windows PC boots as Debian machine and runs GParted.
Before modifying the partitions on your SD card, it may be a good idea to make an image of the card first. If something went wrong, you have at least a backup. On this page, you can find instructions on how to make a copy of your SD card.
Now, insert the SD card in the machine where Gparted is running and follow the next steps to create a new partition on your SD card.
- Once GParted finds your SD card, it will look something like this. Here you will see a 32 GByte card with a clean NOOBS 3.3.1 installation. The most important part is the root sector, the partition we are going to use. Of the 27 GByte, only 6.86 Gbyte is occupied. The sector is mounted, indicated by the padlock. Before any changes are allowed, you need to unmount this partition from the context menu (right-click on the padlock icon).
- When the padlock icon isn't shown, the partition can be resized. NOOBS always allocates all memory on your SD card. Hence we have to shrink this sector to make additional space for new partitions. Again, with the context menu, select the resize option.
- With the new dialogue, you can give the size of your new partition. In our application with only a small settings file, 32 MByte was more than sufficient.
- The next step is to format the unallocated 32 MByte. Right-click on the newly created section and choose the New. In this dialogue two things are important. First, select ext4 as the file system. Second, give the partition a proper name. This label will later on shown as the name of the "external" storage device.
- All actions are performed with the "Apply" button.
- Once succeeded, close this wizard.
- As you can see, the new partition was created under the name "Disk". Also note that of the original 32 MByte requested, only 30 MByte could be allocated. There is still unallocated memory, which has to do with the different format schemes. These loose bytes result in a "lost + found" folder on your "external" storage device. You can remove this folder later.
- Insert the SD card in your Pi and boot. This will result with a new "Disk" icon on the Desktop and in the File Manager.
Automatic mounting.
The next step is to mount the new partition automatically when starting up the Raspberry Pi. This way, it is no longer an external device, but a folder in the directory tree. Accidentally unmounting (ejection) the partition is now not possible any more. Please follow the next steps.
- Because the partition isn't mounted yet, it shows on your desktop and in the folders, just like the last picture of the previous slide show.
- First, you need the identification of the new partition. With the command $ sudo blkid all partitions are shown. Here, the third entry shows the new partition labelled "Disk". Note the PARTUUID number. You will need it in the next step.
- Open the fstab file with $ sudo nano /etc/fstab. Append a new line PARTUUID=xxxx-xxxx /mnt/WRdisk ext4 defaults,auto,users,rw,nofail 0 0. The xxxx-xxxx stands for the PARTUUID number found at the previous step. The name of the new folder is /mnt/WRdisk. Change it if you like. Mind the filesystem ext4, like the one given in GParted when formating the partition. Save and close the file <Ctrl>+<X> , <Y>, <Enter>. Reboot.
- Please note the absence of the Disk icons and eject button in the File Manager. Soon, the newly created WRdisk folder will be the only location where files can be saved permanently.
- The last action is allowing everyone accesses to the new storage device. It can be done with the command $ sudo chmod -R a+rwx /mnt/WRdisk. With the $ ls -l command, you can check the access rights of the device; they should be granted to everyone.
Install ZRAM.
With read-only mode, your SD card is no longer available for swapping. It shouldn't be a problem as you usually have to use swapping during software installation, which is no longer possible in read-only mode. If, for some reason, you need swap memory, use ZRAM on your Raspberry Pi 4. ZRAM, as the name implies, compresses less-used memory blocks into smaller pieces, freeing up memory. It only works with the RAM, thereby ignoring the flash memory on the SD card. It is a fast and robust method, which gives you extra space if necessary. On the other hand, it still generates some overhead that can slow down your application a fraction.
To install ZRAM on the Raspberry Pi, follow the commands below.
# the usual start
$ sudo apt-get update
$ sudo apt-get upgrade
# get the ZRAM from the novaspirit GitHub
$ sudo wget -O /usr/bin/zram.sh https://raw.githubusercontent.com/novaspirit/rpi_zram/master/zram.sh
# make the script executable
$ sudo chmod +x /usr/bin/zram.sh
Open the Nano text editor with the following command and add the /usr/bin/zram.sh & line before exit 0 at the end of the file. Note the ampersand at the end of this line. It indicates that the script is running in a separate thread while the startup continues. Don't forget this character. Close the session with the <Ctrl+X> key combination. With <Y> and <Enter> changes are being saved. Now reboot and check your swapping space.
$ sudo nano /etc/rc.local
# Add this line just above exit 0 near the end.
/usr/bin/zram.sh &
# Ctrl+X, Y, Enter to save the session
# Reboot to run ZRAM
$ sudo reboot
Below, three possible swap configurations. Of course, you can always install ZRAM on a Raspberry Pi. Not only when its in read-only mode.
Stopping services.
The last action before making the Raspberry Pi read-only is to stop some services. For example, flash swapping is no longer possible any more. Other services you may want to cancel, because they are no longer useful and consume valuable memory unnecessarily, such as logging. Follow the commands below and select the ones you don't need any more.
# stop the swapping the flash always
$ sudo /etc/init.d/dphys-swapfile stop
$ sudo apt-get remove --purge dphys-swapfile
# free the allocated memory
$ sudo rm /var/swap
# replace the logging manager from flash
# to RAM. Use $ logread to check the logs
$ sudo apt-get install busybox-syslogd
$ sudo apt-get remove --purge rsyslog
$ sudo apt-get remove --purge logrotate
# triggerhappy lets you enter setup by holding down the
# <Shift> key during boot time.
$ sudo apt-get remove --purge triggerhappy
# fake-hwclock stores at intervals the time,
# in case a network connection fails
$ sudo apt-get remove --purge fake-hwclock# cron is a time-based task scheduler
# delete it only if your apps don't depend on it
$ sudo apt-get remove --purge cron# samba-common makes file transfers with
# Windows machines possible. Your choise.
$ sudo apt-get remove --purge samba-common# always clean up afterwards
$ sudo apt-get autoremove