ext3/4 and SD-card wear leveling

[Aydan]

While thinking about how to implement a backup suite for the Raspberry I ran across something that might be a killer for SD cards:
It seems linux is not aware it is running on a flash medium and does not trim freed blocks, which basically kills wear leveling after a while.
When I used fstrim on my 24/7 pi, which has been running since july, it trimmed the whole free space, meaning there was almost no space free for wear leveling.
Solution:
run

Code: Select all

sudo ionice -c 3 fstrim -v /

often. I made it a daily cronjob on my pi.
The first time might take a while, depending on the random access times of your SD card, but afterwards it's quite quick.

Regards
Aydan

[jojopi]

It seems linux is not aware it is running on a flash medium and does not trim freed blocks, which basically kills wear leveling after a while.
When I used fstrim on my 24/7 pi, which has been running since july, it trimmed the whole free space, meaning there was almost no space free for wear leveling.
The first time might take a while, depending on the random access times of your SD card, but afterwards it's quite quick.

The kernel is well aware that SD is flash, as is evidenced by the fact that it flags its swap space as "SS" (solid state). But whether it attempts to trim is controlled by the "discard" mount option, not the underlying block device medium. And whether the trim succeeds, and does anything useful, is dependent on the implementation of all the driver levels, and hardware.

fstrim will always (attempt to) discard all the free space of the filesystem. Not only the first time you run it, but at least the first time you run it in any given boot. That is because the ext4 on-disk bitmaps only record whether blocks are free or used; there is simply no space to record whether the free blocks have been trimmed or not. So, you would probably be better to do just one "fstrim" and then switch to mounting with the "discard" option to get the supposed effect in real time.

I am not convinced there actually is an effect; that the discards make it through the SDHCI and BCM2708_Arasan layers in a format that does anything to the hardware. Do you have any evidence of this?

[Aydan]

I have no evidence yet, but I was planning on going through the partition with dd and check the number of erased blocks.
I sure hope the trim command will at least write all zeros to a block (or just erase it) to tell the SD-card it's not used.
If this is not implemented this will kill SD-cards quicker then necessary for uses with lots of writes.

Regards
Aydan

[stecklars]

By the way:
Sandisk cards have got lifetime warranty, so you're safe when using one of those

[Mortimer]

I sure hope the trim command will at least write all zeros to a block (or just erase it) to tell the SD-card it's not used.
If this is not implemented this will kill SD-cards quicker then necessary for uses with lots of writes.

Writing zeroes to flash will use up a write cycle too wouldn't it. Surely the idea of TRIM is simply allow the operating system to inform the flash device's logic which blocks are no longer in use, and therefore available for garbage collection and reuse.

By the way:
Sandisk cards have got lifetime warranty, so you're safe when using one of those

If the lifetime of the Sandisk device is determined by how long it takes the blocks to wear out after being overwritten too many times, that won't help will it?

[Aydan]

Writing zeroes to flash will use up a write cycle too wouldn't it. Surely the idea of TRIM is simply allow the operating system to inform the flash device's logic which blocks are no longer in use, and therefore available for garbage collection and reuse.

The SD standard does not know TRIM itself. You can erase a block, which will go back into the pool and as far as I know writing all zeros to a block will not allocate a block from the pool.
Reading an unallocated block will return all zeros. So even if you can't explicitly erase or trim a block, because the driver doesn't support it, writing zeros to a block will cause it to be erased and not reallocated.

Regards
Aydan

[Mortimer]

I still can't see that there is any advantage to writing zeros to an unwanted block. Having it marked for deletion should be enough, and use up the block next write cycle count for writing wanted data to it when the time comes. Wanting to write zeros to a block that is no longer wanted will in my mind effectively double the wear rate across the whole device.

[Aydan]

I still can't see that there is any advantage to writing zeros to an unwanted block. Having it marked for deletion should be enough, and use up the block next write cycle count for writing wanted data to it when the time comes. Wanting to write zeros to a block that is no longer wanted will in my mind effectively double the wear rate across the whole device.

As long as the SD card does not know that the block is free, the SD card controller's wear leveling can't use it.
This basically means the file system takes control over which block gets rewritten and this might wear blocks earlier than necessary.
Take the following example:
You've been using your SD-card for months and have done a lot of writes to it. It might even have been almost full at one time, meanig all blocks have some sort of data on them, even if the card is only half full.
Now you have a program that updates a file quite often. This means that for each write of this file, the same flash block gets written again and again, because the wear leveling pool is empty.
I know this is a very extreme case, but I'm trying to make a point here.
Now if you could tell the SD card that you're not using half of it's blocks, it can use a new block for each write and wear many blocks very little instead of wearing one block a lot.
And this is actually a real problem. I mean why would SSDs support a dedicated command (TRIM) for this if it wouldn't matter?

[lb]

The discards do make it to the hardware, that's for sure. The SDHCI driver even includes a hack for SD cards that are especially slow at this.

Please do not use the "discard" mount option. Trimming on demand is often very bad for performance. It's much preferable to run fstrim from time to time, just like Aydan does. However, I am not sure if the simple wear-leveling algorithms in most SD cards will actually take advantage of the trimmed space.

[Aydan]

I can confirm that the unused blocks are zeroed out.
Ran a little python script counting zeroed 1k chunks from /dev/root and it came out a bit more than what df / reports as free.
I can also confirm that fstrim will trim all unused blocks after a reboot, no matter if they are zeroed already.
I have no Idea what the SD card does if you erase an already erased block again.
Might be an idea to "improve" it so that it reads a block first to check if it is empty already before actually erasing.

@lb: does the driver only erase or does it write zeroes as well?

[lb]

The driver just emits an "ERASE" command for the blocks in question.

[mole125]

Of course all you have proved is that if you read an erased/trimmed block the SD card controller return zeros, you haven't proved whether it actually wasted time erasing them, it may just have set a flag somewhere and have code in the format:
if(blockErased) return zeroData else return blockData.

[Aydan]

Of course all you have proved is that if you read an erased/trimmed block the SD card controller return zeros, you haven't proved whether it actually wasted time erasing them, it may just have set a flag somewhere and have code in the format:
if(blockErased) return zeroData else return blockData.

What I proved is that IF the SD card controller uses wear leveling, then it can use the erased blocks for it.
It is known that an SSD/Flash drive, that uses wear leveling, is full, it will wear faster because there's less space for the wear leveling mechanism to work with.
Even if the block is erased on write, it means the SD card has a bigger pool of usable blocks.
Also this thread wasn't about actually erasing SD blocks but about telling the SD card you're not using them anymore.

[mcgyver83]

I looked on google but I cannot understand how TRIM command works: the os says to filesystem that I wand to delete e file, filesystem instead of deleting the corresponding cell send a TRIM command to the SDHC and the filesystem knows that when the os asks to save a new file it can say to the SDHC to use the previuos cell using only one write instead of two (1 delete and 1 write).
Using "discard" in the mount options will enable this, is it right?

[Aydan]

I looked on google but I cannot understand how TRIM command works: the os says to filesystem that I wand to delete e file, filesystem instead of deleting the corresponding cell send a TRIM command to the SDHC and the filesystem knows that when the os asks to save a new file it can say to the SDHC to use the previuos cell using only one write instead of two (1 delete and 1 write).
Using "discard" in the mount options will enable this, is it right?

Not quite.
The OS deletes a file and the blocks where the file resides are marked as available for overwriting in the FAT/MFT/whatever. The harddist or SD-card doesn't know anything at this moment. That's how all filesystems work. TRIM means that the OS also explicitly tells the harddrive/SDHC/whatever, that these blocks are no longer needed. So now it's up to the harddrive/SDHC-driver what is does with them. In case of the SDHC driver of the raspberry, it will issue an eraseblock command to the SD-card, since SD-cards don't understand TRIM. An SSD does understand TRIM and erases unused blocks in it's spare time, with minimal performance impact.
Using discard on an SD-card might impact performance quite a bit. I haven't tried it yet.

Regards
Aydan

[Narf03]

By the way:
Sandisk cards have got lifetime warranty, so you're safe when using one of those

Lifetime warranty isn't your lifetime, it's the product lifetime, means if your sandisk 8gb class 10 spoil and if they still selling that 8gb class 10, then it's under warranty. So the lifetime warranty might not be a good thing.

[mcgyver83]

I looked on google but I cannot understand how TRIM command works: the os says to filesystem that I wand to delete e file, filesystem instead of deleting the corresponding cell send a TRIM command to the SDHC and the filesystem knows that when the os asks to save a new file it can say to the SDHC to use the previuos cell using only one write instead of two (1 delete and 1 write).
Using "discard" in the mount options will enable this, is it right?

Not quite.
The OS deletes a file and the blocks where the file resides are marked as available for overwriting in the FAT/MFT/whatever. The harddist or SD-card doesn't know anything at this moment. That's how all filesystems work. TRIM means that the OS also explicitly tells the harddrive/SDHC/whatever, that these blocks are no longer needed. So now it's up to the harddrive/SDHC-driver what is does with them. In case of the SDHC driver of the raspberry, it will issue an eraseblock command to the SD-card, since SD-cards don't understand TRIM. An SSD does understand TRIM and erases unused blocks in it's spare time, with minimal performance impact.
Using discard on an SD-card might impact performance quite a bit. I haven't tried it yet.

Regards
Aydan

Thanks.
So using TRIM on raspberry is bad because SDHC will execute an erase command instead of on an SSD. So why someone is saying to set TRIM command on the rasp filesystem with "discard" mount option?

[Aydan]

Aydan

Thanks.
So using TRIM on raspberry is bad because SDHC will execute an erase command instead of on an SSD. So why someone is saying to set TRIM command on the rasp filesystem with "discard" mount option?[/quote]
It's not bad at all, but you might needlessly sacrifice performance. The erase command is needed for the SD-card to be able to do wear leveling properly. If not used the waer leveling pool might be depleted and the SD-card will wear faster than necessary.
I myself don't use discard, but have a daily cronjob running fstrim on the root partition.

[rrolsbe]

If forum members have found SDHC cards that actually do something with the trim command, I would like to know which ones do. The Trim command is an ATA command that can be sent to an SSD controller telling it which NAND pages (usually 4K in size) have been deleted in the file system. If the controller can process Trim commands, it will flag the NAND page(s) so the SSD controllers garbage collection algorithm can eventually recover/erase the page. AFAIK, no SDHC card has a controller that responds to the Trim command, I wish they did. The SDHC controller has no idea which pages contain deleted information. I believe some EMMC flash, used in newer smartphones, do support a form of Trim. Bottom line SDHC cards are inexpensive devices and do not have many of the performance features a good SSD does.

Regards

[mcgyver83]

Aydan

Thanks.
So using TRIM on raspberry is bad because SDHC will execute an erase command instead of on an SSD. So why someone is saying to set TRIM command on the rasp filesystem with "discard" mount option?

It's not bad at all, but you might needlessly sacrifice performance. The erase command is needed for the SD-card to be able to do wear leveling properly. If not used the waer leveling pool might be depleted and the SD-card will wear faster than necessary.
I myself don't use discard, but have a daily cronjob running fstrim on the root partition.[/quote]

Ok, I understand. Please can you paste here the command line you run with fstrim command, please?

[kalehrl]

Not even all SSDs support the TRIM command.
4GB SSD in my eee pc surf doesn't support it.

[toysareforboys]

I made it a daily cronjob on my pi.

Can you post a "how-to" for complete Linux noobs? I'm running a Pi as a mission critical server, and it does lots of read/writes (captures live weather data, makes some .JPG images with it, hosts it on it's websites). Would like to make sure wear levelling has the best chance at working possible.

-Jamie M.

[Aydan]

add the following line to your crontab:

Code: Select all

# m     h       dom     mon     dow     command
0       1       *       *       *       ionice -c 3 fstrim -v /

This runs the job every day at 1am
This will only trim the root filesystem.

Regards
Aydan

[dms75]

Hi

To add some information regarding the behaviour of SDHC cards,

all the sandisk SDHC cards I have come accross almost imediately return from a block erase command, after wich the block is reread as zero. Writing Zeroes to that block actually is much slower, so I guess that the erase command only marks the block as erased and does not actually write to it at all. When writing a single block or several consecutive blocks you can save writetime by preerasing blocks, but you have to use a specifig erase write sequence for that, so that the card can erase several blocks at once and save time compared to erasing each block with the standard erase and write block command one at a time.
So even if the SDHC cards don't support the trim command the erase block command of the SDHC card might actually do quite the same, since it's too fast for a real physical erase procedure but it still results in a block being reread as zero.
Note that this might well be device and or vendor specific behaviour, its what i saw while working with sandisc SDHC cards on a dataloggger project with a microcontroller using SPI protocoll for communication.

As far as I know the standard itself does not specify this behaviour, but it seams a cheap way to prolong the lifetime of a sdhc card, especially since they don't support a trim command. So since its easy to implement and the sdhc cards need to do some kind of wear leveling anyhow I guess that most cards will use this approach, but ofc its up to the sdcard manufacturer...

Dirk

[toysareforboys]

add the following line to your crontab:

Ok, I'll try and figure out what/where crontab is

-Jamie M.