Trying to get SSD boot working on pi4

[novaavianna]

@HawaiianPi Gosh - it hadn't even occurred to me to check for TRIM support - I just kind of assumed

So according to the spec on Startech, both adapters support both TRIM and UASP:

USB312SAT3CB (Chipset: ASM1351)
https://www.startech.com/HDD/Adapters/u ... B312SAT3CB

USB3S2SAT3CB (Chipset: ASM1153E)
https://www.startech.com/HDD/Adapters/U ... B3S2SAT3CB

Is it safe to assume that if the hardware supports UASP and TRIM that RPiOS will support it - or does that kind of support vary?

Also, re your kind suggestion of the Eluteng adapter - I've been reading that the hardware now has a different revision and the new 'blue' one isn't as good/doesn't work without quirks mode?

Great news that power draw doesn't seem to be an issue - I was reading the data sheet on the Crucial stuff and that was causing a little concern

So looks like either adapter would be fine? I'm leaning toward the USB3S2SAT3CB as thanks to @ColdFire we know it works and it's a bit cheaper - always a good thing!

[HawaiianPi]

Is it safe to assume that if the hardware supports UASP and TRIM that RPiOS will support it - or does that kind of support vary?

It varies, unfortunately. I have some ASMedia based adapters that required firmware updates to fix TRIM (and one bricked when updating).

Also, re your kind suggestion of the Eluteng adapter - I've been reading that the hardware now has a different revision and the new 'blue' one isn't as good/doesn't work without quirks mode?

The newer "blue" model is the one that works best. The older "black" models needed new firmware to fix TRIM, even though UAS was supported (neither required quirks).

I had 3 of the older adapter cables and an mSATA adapter that all had the same VID:PID and supported UAS under Raspbian, but didn't support TRIM. I tried updating the firmware and was successful on 2 cables and the mSATA adapter. Not really sure why or what went wrong, but one of the older Eluteng adapter cables would not come up after the supposedly successful flash (software reported the firmware flash was okay, but the adapter failed to reset and is now dead).

All 3 of my newer "blue" models work with UAS and TRIM out of the box.

[novaavianna]

How interesting - great that you're having good luck with the blue one. I was basing my opinion on this post:

viewtopic.php?p=1440311

Shame this sort of thing isn't a little more supported. The other post that I found about this sort of thing was:

https://jamesachambers.com/raspberry-pi ... sh-drives/

Sorry to hear about your bricked unit - that sucks

[HawaiianPi]

How interesting - great that you're having good luck with the blue one. I was basing my opinion on this post:

viewtopic.php?p=1440311

That old post was about the Pi3B and doesn't really apply to the 4B (very different hardware and booting system).

Shame this sort of thing isn't a little more supported. The other post that I found about this sort of thing was:

https://jamesachambers.com/raspberry-pi ... sh-drives/

Sorry to hear about your bricked unit - that sucks

Quirks aren't needed with any ASMedia based adapter I've tried, but it's not really a "fix" anyway, at least with SSD. Some adapters without proper UAS support are slow to respond to UAS commands, and this can affect performance (or even prevent booting). Blacklisting UAS with quirks improves performance, but disabling UAS also disables TRIM.

Quirks are usually needed with adapters based on JMicron bridge chips, and some chips are blacklisted in Linux by default (Norelsys). I have a bunch of JMicron adapters as well, and all but one don't work in Raspbian without quirks. With quirks they might be useful for HDD (but even that depends on the type of HDD) and none will support TRIM without UAS.

I have one USB 3.1 Gen2 NVMe enclosure with a JMicron chip that does work with UAS/TRIM in Raspbian, but I really don't recommend NVMe SSD for Raspberry Pi computers.

[novaavianna]

You weren't kidding about the boot system being different!

Thanks so much for your informative post - I now have lots of reading to do

[ColdFire]

The StarTech USB3S2SAT3CB supports UASP:

Code: Select all

pi@raspberrypi:~ $ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 2: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M

I don't know how to check for TRIM support but not sure if it matters much for my use case.

[novaavianna]

TRIM is important for the life of your SSD. To check apparently this is the procedure:

1) Install hdparm
sudo apt install hdparm

2) generate a file with random content
sudo dd if=/dev/urandom of=tempfile count=8 bs=512 oflag=direct

3) find the LBA sector number of the file
sudo hdparm --fibmap ./tempfile

4) Read the file by its LBA address (replace LBA value from step #3, and assume SSD on sdb)
sudo hdparm --read-sector LBA /dev/sdb

5) Delete the file and wait 10 second
sudo rm tempfile && sync && sleep 10

6) Read the file by its LBA address again, If the content are all 0000, the Trim is working.
sudo hdparm --read-sector LBA /dev/sdb


source: viewtopic.php?p=1497265

[ColdFire]

TRIM is important for the life of your SSD. To check apparently this is the procedure:

1) Install hdparm
sudo apt install hdparm

2) generate a file with random content
sudo dd if=/dev/urandom of=tempfile count=8 bs=512 oflag=direct

3) find the LBA sector number of the file
sudo hdparm --fibmap ./tempfile

4) Read the file by its LBA address (replace LBA value from step #3, and assume SSD on sdb)
sudo hdparm --read-sector LBA /dev/sdb

5) Delete the file and wait 10 second
sudo rm tempfile && sync && sleep 10

6) Read the file by its LBA address again, If the content are all 0000, the Trim is working.
sudo hdparm --read-sector LBA /dev/sdb


source: viewtopic.php?p=1497265

The content is not all 0000 so TRIM is not working... but does that mean TRIM is not supported? Does it need to be configured/enabled beforehand?

[novaavianna]

So I've been reading about this this morning

Seems like you might need to firmware update the adapter. They have links on their website. Appears to be windows only firmware updater.exe

According to the manufactures website, the adapter does support TRIM...

[tom.ty89]

It kind of needs to be enabled in Linux because the ASMedia firmware isn't really spec-compliant enough (and kernel devs refuse to ignore that informational flaw).

You don't even need to do zero check like that to confirm TRIM is working (some TRIM implementation doesn't always guarantee (fully) "read zero after TRIM" anyway).

First of all install the package sg3-utils (or sg3_utils in some distros) and use `sg_vpd` to check whether the firmware supports TRIM:

Code: Select all

[tom@archlinux ~]$ sudo sg_vpd -p bl /dev/sdc
Block limits VPD page (SBC):
  Write same non-zero (WSNZ): 0
  Maximum compare and write length: 0 blocks [Command not implemented]
  Optimal transfer length granularity: 1 blocks
  Maximum transfer length: 65535 blocks
  Optimal transfer length: 65535 blocks
  Maximum prefetch transfer length: 65535 blocks
  Maximum unmap LBA count: 4194240
  Maximum unmap block descriptor count: 1
  Optimal unmap granularity: 1 blocks
  Unmap granularity alignment valid: false
  Unmap granularity alignment: 0 [invalid]
  Maximum write same length: 0 blocks [not reported]
  Maximum atomic transfer length: 0 blocks [not reported]
  Atomic alignment: 0 [unaligned atomic writes permitted]
  Atomic transfer length granularity: 0 [no granularity requirement
  Maximum atomic transfer length with atomic boundary: 0 blocks [not reported]
  Maximum atomic boundary size: 0 blocks [can only write atomic 1 block]

Code: Select all

[tom@archlinux ~]$ sudo sg_vpd -p lbpv /dev/sdc
Logical block provisioning VPD page (SBC):
  Unmap command supported (LBPU): 1
  Write same (16) with unmap bit supported (LBPWS): 0
  Write same (10) with unmap bit supported (LBPWS10): 0
  Logical block provisioning read zeros (LBPRZ): 0
  Anchored LBAs supported (ANC_SUP): 0
  Threshold exponent: 0 [threshold sets not supported]
  Descriptor present (DP): 0
  Minimum percentage: 0 [not reported]
  Provisioning type: 0 (not known or fully provisioned)
  Threshold percentage: 0 [percentages not supported]
  

If you see "Maximum unmap LBA count" not being 0 and "Unmap command supported (LBPU)" being one, the firmware has TRIM support built in it.

Code: Select all

[tom@archlinux ~]$ sudo sg_readcap -l /dev/sdc
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last LBA=468862127 (0x1bf244af), Number of logical blocks=468862128
   Logical block length=512 bytes
   Logical blocks per physical block exponent=0
   Lowest aligned LBA=0
Hence:
   Device size: 240057409536 bytes, 228936.6 MiB, 240.06 GB

With sg_readcap -l, you can find out the "Logical block length". Multiply it with "Maximum unmap LBA count" you get the maxium bytes that a single SCSI UNMAP (which is translated to ATA TRIM by the adapter) can cover. Note that the value is hard-coded in the adapter, but the value should be safe even with the oldest SSD that supports TRIM (with the above case it's like a minimum that a TRIM command should be able to support anyway, IIRC).

Also see that lbpme=0 here. This is the flaw I mentioned. If it's 1 you do not need to enable TRIM (or more precisely, unmap / discard, the SCSI / OS level terms) manually. I have yet to see any TRIM supporting adapter with firmware that reports this correctly. So I'm intrigued in whether what @HawaiianPi said (that the "ELUTENG blue adapter supports TRIM on Linux OOTB") is real. Anyway, here's the code:
https://github.com/raspberrypi/linux/bl ... sd.c#L2349
https://github.com/raspberrypi/linux/bl ... sd.c#L2979
https://github.com/raspberrypi/linux/bl ... sd.c#L2884
EDIT: I then saw another of his posts talking about the need of changing provisioning_mode

Btw, unless I remember it wrongly, disabling UAS doesn't prevent discard/UNMAP/TRIM from working or being enabled by default (if the firmware fulfills the "requirements" of the scsi disk driver, that is). I glimpsed scsiglue.c in the usb-storage driver again and didn't notice such code either. (Although without UAS NCQ isn't supported so the performance impact might be heavier; something like that I guess.) But maybe I just forgot something.

Code: Select all

[root@archlinux ~]# find /sys/ -name provisioning_mode -exec grep . {} + | sort
/sys/devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2:1.0/host6/target6:0:0/6:0:0:0/scsi_disk/6:0:0:0/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb3/3-4/3-4:1.0/host7/target7:0:0/7:0:0:0/scsi_disk/7:0:0:0/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb3/3-5/3-5:1.0/host8/target8:0:0/8:0:0:0/scsi_disk/8:0:0:0/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/provisioning_mode:writesame_16

The above sysfs files (which are virtual/volatile) reports whether discard is enabled/allowed for the respective drive and can be used to change the setting as well. You can install and use lsscsi (or maybe even udevadm) to find out which drive each of their paths is referring to:

Code: Select all

[root@archlinux ~]# lsscsi
[0:0:0:0]    disk    ATA      SPCC Solid State 4A0   /dev/sda 
[1:0:0:0]    cd/dvd  PIONEER  BD-RW   BDR-S08  1.10  /dev/sr0 
[6:0:0:0]    disk    ASMT     2115             0     /dev/sdb 
[7:0:0:0]    disk    PNY CS90 0 240GB SSD      0     /dev/sdc 
[8:0:0:0]    disk    SPCC Sol id State Disk    0     /dev/sdd 
[root@archlinux ~]# fstrim /
fstrim: /: the discard operation is not supported
[root@archlinux ~]# findmnt /
TARGET SOURCE    FSTYPE OPTIONS
/      /dev/sdc2 f2fs   rw,relatime,lazytime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,fsync_m
[root@archlinux ~]# echo unmap > /sys/devices/pci0000:00/0000:00:14.0/usb3/3-4/3-4:1.0/host7/target7:0:0/7:0:0:0/scsi_disk/7:0:0:0/provisioning_mode
[root@archlinux ~]# cat /sys/devices/pci0000:00/0000:00:14.0/usb3/3-4/3-4:1.0/host7/target7:0:0/7:0:0:0/scsi_disk/7:0:0:0/provisioning_mode
unmap
[root@archlinux ~]# echo $((4194240*512))
2147450880
[root@archlinux ~]# cat /sys/block/sdc/queue/discard_max_bytes 
4294966784
[root@archlinux ~]# echo $((4194240*512)) > /sys/block/sdc/queue/discard_max_bytes 
[root@archlinux ~]# cat /sys/block/sdc/queue/discard_max_bytes 
2147450880
[root@archlinux ~]# fstrim /
[root@archlinux ~]# 

The settings are volatile so you'll need to write udev rules to make them persistent. I'm not gonna cover it here.

Note that discard-upon-deletion requires you to have the filesystem mounted with the option discard, although some do that by default (e.g. f2fs), while fstrim works as long as you done changing the scsi/block-level settings.

[novaavianna]

Wow @tom.ty89 thanks for that insanely great detailed response - very much appreciated.

Just as a follow up I got the ELUTENG blue adaptor from amazon (uk) and it had the JMicron 578 controller in it and basically didn't seem to work at all. I haven't gone down the road of looking for firmware updates for it.

I then got the USB3S2SAT3CB which has the ASM1153E controller. This worked immediately (which was a relief that I wasn't doing something entirely stupid). I then added the /etc/udev/rules.d:

ACTION=="add|change", ATTRS{idVendor}=="1f28", ATTRS{idProduct}=="f001", SUBSYSTEM=="scsi_disk", ATTR{provisioning_mode}="unmap"

and now if I manually call (fstrim -v / ) it gives me results like: /: 206.3 MiB (216350720 bytes) trimmed

which seem like things are working? I did test the commands @tom.ty89 listed so helpfully and I did indeed see the lbpme=0 error but all of the other results seemed to match correctly?

I was reading here:
https://opensource.com/article/20/2/tri ... rage-linux and viewtopic.php?p=1497265
about discard vs fstrim.timer - and fstrim.timer seems to be the way to go I guess?

So I think I'm in good shape?

[HawaiianPi]

Just as a follow up I got the ELUTENG blue adaptor from amazon (uk) and it had the JMicron 578 controller in it and basically didn't seem to work at all.

Are you sure that was the recommended Eluteng adapter cable?

Specifically this one: https://www.amazon.co.uk/dp/B06XCV1W97

I have purchased several at different times from Amazon (USA) and they all have the ASMedia bridge chip. There are a lot of look-alike adapter cables with the same outside appearance but different chips and firmware (I have a couple of those as well, with VIA or JMicron chips, which don't work as well).

[novaavianna]

Yup - that's the exact one. When I clicked on your link, Amazon even helpfully tells me I just purchased it

If I compare your UK link vs your US link I notice that the two products have differing ASIN, namely:

US - B0716JKJ68
UK - B06XCV1W97

I wonder if they just make identical products that just have differing chipsets on a per country basis. Which would be super irritating!

But the UK version I received had official looking ELUTENG branded packaging and was for sure using the JMicron 578 chipset. The USB3S2SAT3CB thankfully had the chipset it should i.e. the ASM1153E.

[HawaiianPi]

Yup - that's the exact one. When I clicked on your link, Amazon even helpfully tells me I just purchased it
...
I wonder if they just make identical products that just have differing chipsets on a per country basis. Which would be super irritating!

Or they just have multiple versions of the same product with different chipsets, which would be even more irritating, although I'd expect we would have seen that before now.

But the UK version I received had official looking ELUTENG branded packaging and was for sure using the JMicron 578 chipset.

You say yours had Eluteng branded packaging, but was the cable itself also Eluteng branded?

I have a generic (no name) adapter that looks nearly identical to the Eluteng (the SATA end is the same but the USB-A plug is slightly different), and that one has a JMicron chipset.

[novaavianna]

Yup - both the usb-c adapter bit and the blue end are ELUTENG branded. Weird right?

[geerlingguy]

Code: Select all

[root@archlinux ~]# find /sys/ -name provisioning_mode -exec grep . {} + | sort
/sys/devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2:1.0/host6/target6:0:0/6:0:0:0/scsi_disk/6:0:0:0/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb3/3-4/3-4:1.0/host7/target7:0:0/7:0:0:0/scsi_disk/7:0:0:0/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:14.0/usb3/3-5/3-5:1.0/host8/target8:0:0/8:0:0:0/scsi_disk/8:0:0:0/provisioning_mode:full
/sys/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/provisioning_mode:writesame_16

One note here—you need to pass -H to grep to get the full path before the contents returned:

Code: Select all

find /sys/ -name provisioning_mode -exec grep -H . {} + | sort

[geerlingguy]

Following on @tom.ty89's comment earlier in this thread, I decided to write up a more step-by-step summary of all the parts involved in testing for, enabling, and persisting TRIM support in an SSD.

I posted that to my blog here: Enabling TRIM on an external SSD on a Raspberry Pi

Thanks so much for the writeup (and everyone else's follow-up comments). I was scratching my head on a few parts, and the details here put me over the hump so I could finally get TRIM working on a couple of my drives!

[macmarty]

I had similar issues with a Pi running the stable version. The part that made it work for me is going from unpowered cable to a powered cable. As soon as I used an external housing for my SSD it started to work. Not sure if it is not getting enough bus power. I need to measure it to see if that is the case.

[iandk]

Is there a reason why the TRIM support has to be manually enabled?

Edit:
Somehow it always shows the remaining disk space as trimmed.
Even if I run it multiple times. The SSD is running XFS

Code: Select all

root@nas ~ # fstrim -v /mnt/ssd 
/mnt/ssd: 198.1 GiB (212705550336 bytes) trimmed
root@nas ~ # fstrim -v /mnt/ssd 
/mnt/ssd: 198.1 GiB (212705550336 bytes) trimmed
root@nas ~ # fstrim -v /mnt/ssd 
/mnt/ssd: 198.1 GiB (212705550336 bytes) trimmed

[DigiBalm]

I'm booting rpi4 8gb and have UAS and trim working with a cheap jmicron usb 3.1 m.2 adapter (jms583) gen2 and teamgroup 256gb nvme. Everything works at good speed and temperature on the jmicron adapter. Thanks to everyone here especially HawaiianPi and geerlingguy for your help. Here's the basic setup:
Using windows pc flashed latest firmware (2.0.9) to jms583.
Installed 64bit version of raspberry os to SD and updated to latest eeprom using instructions on this thread by HawaiianPi. Using raspberry os tool copied SD to nvme SSD. It took less than 5 minutes. Removed hat (nebra anybeam), SD, and all other usb devices during first boot from nvme. Expanded partition to use full disk using raspberry configuration tool.
Getting trim enabled was a little tricky. My "Maximum unmap LBA count" was "-1" which was useless for calculating blocks but I managed a workaround by combining a few different methods from different sites. Now periodic trim works flawlessly.
If anyone else is interested I can provide more details but for now I'm resting.

[rpdom]

Somehow it always shows the remaining disk space as trimmed.
Even if I run it multiple times.

That's how it works. TRIM just tells the card which blocks are not being used. The card will update its internal tables with that information. Linux can't see those tables, so it doesn't know which blocks have already been trimmed.

[HawaiianPi]

Somehow it always shows the remaining disk space as trimmed.
Even if I run it multiple times.

That's how it works. TRIM just tells the card which blocks are not being used. The card will update its internal tables with that information. Linux can't see those tables, so it doesn't know which blocks have already been trimmed.

Not in my experience.

Immediately after a reboot it will indicate all free space being trimmed, but subsequent fstrim commands only return what was actually trimmed.

Code: Select all

RaspberryPi4B:~ $ sudo fstrim / --verbose
/: 223.6 GiB (240069160960 bytes) trimmed

RaspberryPi4B:~ $ sudo fstrim / --verbose
/: 0 B (0 bytes) trimmed

And after using the system for a little while...

RaspberryPi4B:~ $ sudo fstrim / --verbose
/: 96 MiB (100679680 bytes) trimmed

Of course that gets reset the next time you reboot. Regardless, the only blocks that actually get erased are the ones that need to be trimmed (the SSD controller keeps track of that internally). If you are seeing the same blocks being trimmed again when you run the fstrim command twice, then something's wrong. The OS should get updated after the first fstrim command, and should return zero, or a smaller number of blocks trimmed if you run fstrim again (after the first fstrim, but before you reboot).

[joostdevries]

Hi Guys,
seems there is a lot of knowledge in this topic. Hope someone can help me with this issue.

I've order a sandisk extreme pro true SSD flash drive with 128gb of storage. I can't seem to get USB boot to work.
I also get
"usb-MSD boot requires the latest software. Get the latest software from https://www.raspberrypi.org/downloads
ERROR: 00000044"

Every thing is up to date. I also did rpi-update. Bootloader is the latest version, boot order is 0xf14.

lsusb && lsusb -t gives me this info (so the extreme pro is the Bus 002 Device 002, pluged in to a blue usb 3 port, BTW: during booting I have removed the other sandisk usb on Bus 001, to be sure that one is not in the wat of anything):

Bus 002 Device 002: ID 0781:5588 SanDisk Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 0781:5583 SanDisk Corp. Ultra Fit
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
|__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 480M

BTW: the Sandiks is flashed using Balena etcher with as a source a DD full backup zip image from my current SD card (which i run automatically every week, and yes I've made another backup after doing all the things like update rpi).
Greetings,
Joost

[iandk]

iirc you have to boot with a regular microSD and enable USB boot in the raspi-config, after that you can remove the mSD and boot from your USB

[joostdevries]

I have tried that already. No luck with that method.