carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Context switching breaks SPI/DMA

Fri Mar 12, 2021 3:45 am

I'm having problems with SPI and DMA, when trying to use high speeds in a mult-tasking environment. I've tried to extract a simple test case from my large project, but I can't recreate the problem if I simplify it too much. So, I have posted a complete project, based originally on pico-examples/spi/spi_dma, to https://github.com/carlk3/spi_dma.

The gist of it is that in single-threaded mode, I can run the SPI at a baud rate up to 20,833,333. However, I'm running multiple tasks in FreeRTOS. As soon as I introduce one or more additional tasks, I start to get random, flaky errors. No error indications, just bad data.

In the main program, spi_dma.c, I have defines for the number of tasks and the baud rate. For example, this passes the test:
#define N_TASKS 1
#define BAUD_RATE 20833333

However, this fails:
#define N_TASKS 2
#define BAUD_RATE 5000 * 1000
It eventually yields something like:
T1: Done. Checking...T1: Mismatch at 261/1024: expected 7b, got 67
T1: at /home/carlk/pi/pico/spi_dma/spi_dma.c:283: testTask
(different every time).

If I back the speed way down:
#define N_TASKS 2
#define BAUD_RATE 2000 * 1000
it appears to run OK.

What could be happening here? FreeRTOS uses isr_pendsv for task switching. Why would that mess with the SPI/DMA transfer?

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Sat Mar 13, 2021 6:09 pm

Doing some more digging on this, and it just gets weirder!

When I do the checking after the transfer, it frequently looks as if the *transmit* buffer has changed! I changed the code to hex dump the expected vs. actual transmit buffer and here is a typical run:

Code: Select all

SPI DMA example
Actual frequency: 5681818
T0: Done. Checking...T0: All good
T1: Done. Checking...T1: All good
T0: Done. Checking...
T0: hexdump_8(Expected:, 0x20002DC8, 1024)
a7 f1 d9 2a 82 c8 d8 fe 43 4d 98 55 8c e2 b3 47 17 11 98 54 2f 11 2d 05 58 f5 6b d6 88 07 99 92 
48 33 62 41 f3 0d 23 e5 5f 30 d1 c8 ed 61 0c 4b 02 35 39 81 84 b8 14 a2 9c b4 5a 67 2a ca e5 48 
e9 c5 f1 b0 c4 15 8a e5 9b 4d 39 f6 f7 e8 a1 05 d3 fe ed a5 d5 f3 d9 e4 5b fa 6c c3 51 e2 20 ae 
0c e1 06 98 6d 61 ff 34 a1 1e 19 fd 36 50 e9 b7 81 8f c3 3a 1e 0f c0 2c 44 55 7a c8 ab 50 c9 b2 
de b2 f6 b5 e2 4c 4f dd 9f 88 67 bd ce 1f f2 61 00 8e 78 97 97 0e 34 62 07 d7 5e 47 a1 58 29 8e 
5b a2 f5 62 46 86 9c c4 2e 36 2a 02 73 12 64 e6 06 87 ef 53 09 d1 08 53 4f 51 f8 65 8f b4 f0 80 
b7 cb 19 ee 9a eb d7 18 cc 4f a2 7c 8c 37 df c1 ad a5 d1 33 d1 3a be 03 f0 21 e9 b1 b7 8c cb d8 
2f 7f f2 b3 8c 6d 48 d0 1e 48 1b 2d 4f af 71 71 80 5f d7 f2 d3 9e f4 c4 f1 9b 94 96 e8 1d ab 81 
93 b3 73 7e 1b 27 d9 c4 39 57 16 64 41 b9 35 15 e8 f0 3c 95 d8 e8 ce 1e 18 64 fa ad 68 dd fc 59 
32 13 01 09 39 0b 0f 1f e5 ca 71 68 05 f8 36 2e 98 dc ca ad c8 6a db ed 25 80 1a 9a 9d cf a6 26 
43 19 dd af e8 3a 89 c5 1f 3c 6d 19 9d 38 de 10 e6 60 c3 7b e8 72 c3 f2 b3 16 60 de 8b c9 59 02 
b9 10 32 62 cd b9 41 f7 73 76 f5 d3 db b7 a3 d5 a3 87 79 7f c4 81 9a 03 5e ca 70 4c ed b3 71 10 
ee 7f 20 6b 0c 88 05 aa eb f4 96 3e 7c 47 08 ce 8d 4e 09 23 66 e7 17 92 a8 a3 b2 bb cd ee 32 1b 
3e 15 38 0c 54 1e f0 93 08 88 96 9f 74 57 af e1 85 88 82 6a 41 9d 58 31 1c 17 84 b5 48 4e ec db 
39 3f 6a 0a ca 11 b9 1d f0 86 6b 50 0b 8d ee 50 1f d7 eb 9b ce 09 a1 7d 74 12 4b 46 05 ad fc 07 
77 be d9 81 6d 8d 7e 84 88 54 4a 18 d8 04 5c b3 28 3b 0a 75 2b 88 1b 5f 50 0f ad b5 90 10 e6 3d 
1b 66 4f 8b da 2d bf 33 cb 6b e2 1c 8e b3 ec a9 d9 d5 bf 14 4c 08 e9 57 7e d0 d1 e5 e5 60 87 51 
09 b3 40 98 05 80 47 3d bc 2e 68 9a 3b e8 38 e7 7a 0a 33 48 fe 96 0e c9 bf 81 da 36 f1 86 8c a5 
d2 47 88 fa 4c 0c 77 8b f0 d1 23 14 28 54 95 63 65 16 cf 40 86 1b 3d 73 7f d3 5d bb 59 1c 5b 5d 
25 91 6e b1 d8 61 76 b1 4e 0e 67 d2 d0 39 57 f0 cf 6c 87 83 4b f3 28 54 05 88 36 0b a7 c7 c5 f8 
85 41 63 4f b7 ba de 5f 94 ff 67 1d 1f eb dc bd a1 16 d2 da 77 90 38 ed 76 79 89 6c 29 19 8b 26 
57 b5 8c 50 ea 05 4f 64 4f 41 29 c8 ba 8d 8d 54 4b 72 76 33 dd 40 75 43 98 04 67 96 e0 38 62 6f 
ef 92 37 ce 5b 61 5b c0 86 77 ee 5a bf bd 85 f7 3f 7f 88 68 cb 1b 5f ba 4c 13 09 f1 60 61 aa 13 
38 21 fb e2 a7 58 d2 bb e6 aa 04 0a 94 0d 41 b7 d3 b8 69 ce e9 45 15 0a a4 a4 0e 6f f7 19 ee c2 
4b 26 81 cd 5c e0 6b 50 27 34 36 58 40 66 04 66 56 d5 ef ed 73 15 75 91 89 d6 88 15 dd b9 e5 f8 
d7 fd 53 b6 ec 09 66 16 a7 73 b9 42 1f 67 04 ce d3 6e f4 e4 84 ba 0c 6c 5a 48 55 c7 1c 33 a5 4a 
c8 2b e8 03 e5 cf d1 75 77 9f c4 44 b7 e6 aa 90 01 ee fa be bc 0c f9 97 54 88 7c 7b 0a 27 af dd 
c4 15 f8 a0 2c 5a f1 ef ea 26 ad 1e 5d 92 b1 e2 9a 8f af 5b 21 86 c3 09 4f 4a 13 7b cf aa 65 d7 
b2 74 21 4d b6 4c 86 f3 08 5b 24 93 8e 18 32 fb 31 0a 6f 06 41 81 e2 98 d2 30 62 ab c8 17 ba 17 
30 23 c8 c0 4c 5c 3a 1e cb f4 af 72 37 2b 38 1f f6 98 65 c8 f0 e3 c7 0b 93 1c 45 a7 41 9b 3c 44 
18 42 eb fa cc 3d 07 0a c3 b4 33 cd 12 0b 6e 85 b7 2d ad cf 40 b2 3b 17 3c 34 f6 be 1b 19 01 f6 
62 1f 14 97 b0 85 cf 8e 99 9d 98 6e f8 ff 3a 88 9a 02 38 97 99 83 a8 68 6f 69 e1 0e f9 24 9a 87 

T0: hexdump_8(txbuf, 0x200035C8, 1024)
a7 f1 d9 2a 82 c8 d8 fe 43 4d 98 55 8c e2 b3 47 17 11 98 54 2f 11 2d 05 58 f5 6b d6 88 07 99 92 
48 33 62 41 f3 0d 23 e5 5f 30 d1 c8 ed 61 0c 4b 02 35 39 81 84 b8 14 a2 9c b4 5a 67 2a ca e5 48 
e9 c5 f1 b0 c4 15 8a e5 9b 4d 39 f6 f7 e8 a1 05 d3 fe ed a5 d5 f3 d9 e4 5b fa 6c c3 51 e2 20 ae 
0c e1 06 98 6d 61 ff 34 a1 1e 19 fd 36 50 e9 b7 81 8f c3 3a 1e 0f c0 2c 44 55 7a c8 ab 50 c9 b2 
de b2 f6 b5 e2 4c 4f dd 9f 88 67 bd ce 1f f2 61 00 8e 78 97 97 0e 34 62 07 d7 5e 47 a1 58 29 8e 
5b a2 f5 62 46 86 9c c4 2e 36 2a 02 73 12 64 e6 06 87 ef 53 09 d1 08 53 4f 51 f8 65 8f b4 f0 80 
b7 cb 19 ee 9a eb d7 18 cc 4f a2 7c 8c 37 df c1 ad a5 d1 33 d1 3a be 03 f0 21 e9 b1 b7 8c cb d8 
2f 7f f2 b3 8c 6d 48 d0 1e 48 1b 2d 4f af 71 71 80 5f d7 f2 d3 9e f4 c4 f1 9b 94 96 e8 1d ab 81 
93 b3 73 7e 1b 27 d9 c4 39 57 16 64 41 b9 35 15 e8 f0 3c 95 d8 e8 ce 1e 18 64 fa ad 68 dd fc 59 
32 13 01 09 39 0b 0f 1f e5 ca 71 68 05 f8 36 2e 98 dc ca ad c8 6a db ed 25 80 1a 9a 9d cf a6 26 
43 19 dd af e8 3a 89 c5 1f 3c 6d 19 9d 38 de 10 e6 60 c3 7b e8 72 c3 f2 b3 16 60 de 8b c9 59 02 
b9 10 32 62 cd b9 41 f7 73 76 f5 d3 db b7 a3 d5 a3 87 79 7f c4 81 9a 03 5e ca 70 4c ed b3 71 10 
ee 7f 20 6b 0c 88 05 aa eb f4 96 3e 7c 47 08 ce 8d 4e 09 23 66 e7 17 92 a8 a3 b2 bb cd ee 32 1b 
3e 15 38 0c 54 1e f0 93 08 88 96 9f 74 57 af e1 85 88 82 6a 41 9d 58 31 1c 17 84 b5 48 4e ec db 
39 3f 6a 0a ca 11 b9 1d f0 86 6b 50 0b 8d ee 50 1f d7 eb 9b ce 09 a1 7d 74 12 4b 46 05 ad fc 07 
77 be d9 81 6d 8d 7e 84 88 54 4a 18 d8 04 0e d0 e6 a3 ec 3f 88 a2 a3 dc 71 c1 c4 3c 2b 28 cc 7f 
96 19 c1 b9 3f 5f 00 e5 cd be b6 3d 5b c1 ab 18 35 5d 66 02 d3 76 f4 73 55 1d 80 d1 1f bc 7b a0 
8d 5f c8 d7 c7 7d 8b 27 1f 7a 57 b9 55 42 25 32 91 d1 8e 92 53 5b 43 48 cd ab 52 44 09 bb e6 ae 
e0 5d c2 9c 5d 50 db 89 66 94 27 d9 8e 64 15 79 8e 53 18 3c 3d af 7f 96 07 a8 d0 9a e3 ca 72 7d 
1c 78 74 79 d9 37 47 f1 3d c0 1c 87 fe 6e 7a 48 81 2b 7e fe f3 c9 a4 be 0c 22 2a 22 e1 68 a6 67 
25 69 5a 22 ce 53 6d a8 38 37 e4 71 74 e2 03 08 29 30 f5 45 b0 23 35 ef d4 dd b0 4c 7d 7b 48 1c 
38 31 16 15 32 be be 73 18 95 1e 21 be 8c 23 f8 38 57 77 20 3b 47 9b 95 6a ec 56 37 4d 77 c9 24 
51 57 d7 a1 5a f9 bc 26 2a 9c 27 56 d4 5c d2 09 e4 2e c6 e9 f9 ce 90 7d 31 e5 7e 52 4b f6 6f 05 
cb e8 82 6a 24 bc 4c 5e 5d 8b 46 17 84 cf 71 b1 22 96 f6 2c 50 f6 be d3 3d 76 20 09 5f c3 8d cd 
3d 33 85 92 f1 64 c6 68 2f 88 f0 d8 98 92 42 89 b6 d4 34 ca b4 2a 82 12 de 92 a2 b7 f1 9b d1 27 
4e 62 7e c4 6c 86 e1 46 9a ec a8 16 e3 53 53 5c 8a 5b a2 35 6d 38 f7 05 fb 9a 4f 88 40 7a 97 d7 
10 60 2e 90 33 a0 cd fc e7 f5 78 ab 13 de c1 65 4c fd 75 98 23 3e 8d cf b6 bc b5 8f 9e ae ce 44 
3d 77 15 b4 e3 4f b5 ef 05 38 37 fa 01 c2 6b c8 b8 d9 8b 79 ac 50 e2 19 af ae 37 bd 12 f6 11 ca 
28 83 0a 5b e9 24 90 5d a0 b5 91 08 aa 15 85 f6 b1 e9 9e 30 f2 20 0e 60 b9 d1 00 81 ad 97 7f 05 
07 7f 69 03 29 d2 9b 0f 0c a3 c6 ee 05 23 27 30 8f 60 c4 bd 7e b5 cf c1 45 5e 18 47 83 41 29 09 
91 fa 35 0c c5 26 b6 6d 7c 4c bd 9f e7 a2 40 c9 b4 bf d0 b3 33 ec ee b8 91 81 74 29 b0 89 da 8f 
96 26 84 2b cb 56 83 3f ad ef 5e 71 c4 62 96 db 02 71 13 df e7 51 bd fd d4 5a 74 2c 24 82 88 a8 
T0: Mismatch at 494/1024: expected 5c, got 0e
T0: 	at /home/carlk/pi/pico/spi_dma/spi_dma.c:280: testTask
Even more bizarrely, sometimes checking the same buffer twice yields different results! On a hunch I added a delay after the transfer:

Code: Select all

        spi_lock(&spi);
        spi_transfer(&spi, txbuf, rxbuf, TEST_SIZE);
vTaskDelay(pdMS_TO_TICKS(2));        
        spi_unlock(&spi);
and with that I can up the baud rate to 20833333 and run two or three tasks successfully.

I am puzzled. It's almost like the memory needs time to settle, or something.

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1033
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Context switching breaks SPI/DMA

Sat Mar 13, 2021 11:00 pm

If u sure you aren't blowing off the end of your stack? sounds like memory corruption somewhere (perhaps your locking is broken) - i haven't looked closely

Things don't need "time to settle".

fivdi
Posts: 464
Joined: Sun Sep 23, 2012 8:09 pm
Contact: Website

Re: Context switching breaks SPI/DMA

Sun Mar 14, 2021 9:28 am

At 1536 words or 6144 bytes the stack for testTask is quite big. On the other hand, testTask creates two fairly large buffers called txbuf and rxbuf on the stack, each having 1024 bytes. The Pico has RAM to burn, so you could double or triple the stack size to see if it helps.

If adding the call vTaskDelay(pdMS_TO_TICKS(2)) directly after the call to spi_transfer() resolves the issue and I had to take a wild guess why this resolves the issue, I would say that spi_transfer is returning before the transfer has fully completed and that it needs to wait for something else to happen before it returns.

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1033
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Context switching breaks SPI/DMA

Sun Mar 14, 2021 4:58 pm

fivdi wrote:
Sun Mar 14, 2021 9:28 am
At 1536 words or 6144 bytes the stack for testTask is quite big. On the other hand, testTask creates two fairly large buffers called txbuf and rxbuf on the stack, each having 1024 bytes. The Pico has RAM to burn, so you could double or triple the stack size to see if it helps.
yup it sounds like enough stack - i don't know enough about the RP2040 port to know if there could be a byte/word bug in this code
If adding the call vTaskDelay(pdMS_TO_TICKS(2)) directly after the call to spi_transfer() resolves the issue and I had to take a wild guess why this resolves the issue, I would say that spi_transfer is returning before the transfer has fully completed and that it needs to wait for something else to happen before it returns.
yeah it is always worth checking whether DMA's complete before operations complete (which can be a problem with writing to devices with FIFOs), however triggering on the DMA read complete should be OK.

My best debugging advice is to twiddle some pins at meaningful times and look on a logic probe (or a pico being a logic probe!)

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Sun Mar 14, 2021 5:47 pm

kilograham wrote:
Sat Mar 13, 2021 11:00 pm
If u sure you aren't blowing off the end of your stack?
To make sure, I added a call to vTaskGetInfo and it showed a Stack High Water Mark: 594 (words). (The stack high water mark is the minimum amount of stack space that has ever existed, so the closer the number is to zero the closer the task has come to overflowing its stack.) So, according to FreeRTOS, there is a huge amount of headroom. But, I went ahead and removed the tx and rx buffers from the stacks to static arrays. Now, I see Stack High Water Mark: 1096, but the test still quickly fails without the 2 ms. delay after the spi_transfer().

Code: Select all

SPI DMA example
Actual frequency: 20833333
T0: Done. Checking...T3: Done. Checking...T0: All good
T0: Stack High Water Mark: 584
T3: All good
T3: Stack High Water Mark: 594
T2: Done. Checking...T1: Done. Checking...T0: Done. Checking...T3: Done. Checking...T1: All good
T1: Stack High Water Mark: 594
T3: All good
T2: All good
T2: Stack High Water Mark: 594
T3: Stack High Water Mark: 594
T2: Done. Checking...T1: Done. Checking...T2: All good
T2: Stack High Water Mark: 594
T0: All good
T0: Stack High Water Mark: 584
T3: Done. Checking...T1: All good
T1: Stack High Water Mark: 594
T0: Done. Checking...T1: Done. Checking...T0: All good
T0: Stack High Water Mark: 584
T2: Done. Checking...T1: All good
T1: Stack High Water Mark: 594
T0: Done. Checking...T3: All good
T3: Stack High Water Mark: 594
T1: Done. Checking...T2: All good
T2: Stack High Water Mark: 594
T3: Done. Checking...T1: All good
T1: Stack High Water Mark: 594
T2: Done. Checking...T1: Done. Checking...T2: All good
T2: Stack High Water Mark: 594
T0: All good
T1: All good
T1: Stack High Water Mark: 594
T0: Stack High Water Mark: 584
T1: Done. Checking...T3: All good
T3: Stack High Water Mark: 594
T0: Done. Checking...T2: Done. Checking...T0: All good
T0: Stack High Water Mark: 584
T1: All good
T1: Stack High Water Mark: 594
T3: Done. Checking...T1: Done. Checking...T3: All good
T3: Stack High Water Mark: 594
T2: All good
T1: All good
T1: Stack High Water Mark: 594
T2: Stack High Water Mark: 594
T1: Done. Checking...T0: Done. Checking...T1: All good
T1: Stack High Water Mark: 594
T3: Done. Checking...T0: All good
T0: Stack High Water Mark: 584
T1: Done. Checking...T0: Done. Checking...T1: All good
T1: Stack High Water Mark: 594
T2: Done. Checking...T0: All good
T1: Done. Checking...T3: All good
T0: Stack High Water Mark: 584
T1: All good
T2: Mismatch at 785/1024: expected 05, got d5
T2: 	at /home/carlk/pi/pico/spi_dma/spi_dma.c:297: testTask

SPI DMA example
Actual frequency: 20833333
T0: Done. Checking...T3: Done. Checking...T0: All good
T0: Stack High Water Mark: 1096
T3: All good
T3: Stack High Water Mark: 1106
T2: Done. Checking...T1: Done. Checking...T2: All good
T2: Stack High Water Mark: 1106
T0: Done. Checking...T1: All good
T1: Stack High Water Mark: 1106
T2: Done. Checking...T1: Done. Checking...T2: All good
T3: Done. Checking...T1: All good
T1: Stack High Water Mark: 1106
T2: Stack High Water Mark: 1106
T1: Done. Checking...T0: All good
T2: Done. Checking...T3: All good
T0: Stack High Water Mark: 1096
T2: All good
T1: All good
T3: Stack High Water Mark: 1096
T2: Stack High Water Mark: 1106
T0: Done. Checking...T1: Stack High Water Mark: 1106
T0: All good
T3: Done. Checking...T2: Done. Checking...T3: All good
T3: Stack High Water Mark: 1096
T0: Stack High Water Mark: 1096
T2: All good
T2: Stack High Water Mark: 1106
T3: Done. Checking...T1: Done. Checking...T3: All good
T3: Stack High Water Mark: 1096
T0: Done. Checking...T1: All good
T1: Stack High Water Mark: 1106
T3: Done. Checking...T1: Done. Checking...T3: All good
T3: Stack High Water Mark: 1096
T2: Done. Checking...T1: All good
T3: Done. Checking...T0: All good
T1: Stack High Water Mark: 1106
T3: All good
T2: All good
T0: Stack High Water Mark: 1096
T3: Stack High Water Mark: 1096
T1: Done. Checking...T2: Stack High Water Mark: 1106
T1: All good
T0: Done. Checking...T3: Done. Checking...T0: All good
T0: Stack High Water Mark: 1096
T1: Stack High Water Mark: 1106
T3: All good
T3: Stack High Water Mark: 1096
T0: Done. Checking...T2: Done. Checking...T0: All good
T0: Stack High Water Mark: 1096
T1: Done. Checking...T2: All good
T2: Stack High Water Mark: 1106
T0: Done. Checking...T2: Done. Checking...T0: All good
T3: Done. Checking...T2: All good
T2: Stack High Water Mark: 1106
T0: Stack High Water Mark: 1096
T2: Done. Checking...T1: All good
T0: Done. Checking...T2: All good
T2: Stack High Water Mark: 1106
T3: All good
T0: All good
T0: Stack High Water Mark: 1096
T3: Stack High Water Mark: 1096
T0: Done. Checking...T1: Stack High Water Mark: 1106
T0: All good
T2: Done. Checking...T3: Done. Checking...T2: All good
T2: Stack High Water Mark: 1106
T0: Stack High Water Mark: 1096
T3: All good
T3: Stack High Water Mark: 1096
T2: Done. Checking...T1: Done. Checking...T0: Done. Checking...T1: All good
T3: Done. Checking...T0: All good
T0: Stack High Water Mark: 1096
T1: Stack High Water Mark: 1106
T3: All good
T3: Stack High Water Mark: 1096
T0: Done. Checking...T2: All good
T3: Done. Checking...T1: Done. Checking...T2: Stack High Water Mark: 1106
T1: All good
T3: All good
T0: All good
T0: Stack High Water Mark: 1096
T3: Stack High Water Mark: 1096
T1: Stack High Water Mark: 1106
T3: Done. Checking...T0: Done. Checking...T3: All good
T3: Stack High Water Mark: 1096
T2: Done. Checking...T0: All good
T0: Stack High Water Mark: 1090
T3: Done. Checking...T1: Done. Checking...T2: All good
T2: Stack High Water Mark: 1106
T3: All good
T3: Stack High Water Mark: 1096
T1: All good
T1: Stack High Water Mark: 1106
T2: Done. Checking...T3: Done. Checking...T2: All good
T2: Stack High Water Mark: 1106
T0: Done. Checking...T3: All good
T3: Stack High Water Mark: 1096
T2: Done. Checking...T0: All good
T3: Done. Checking...T1: Done. Checking...T0: Stack High Water Mark: 1090
T1: Mismatch at 278/1024: expected d8, got de
T1: 	at /home/carlk/pi/pico/spi_dma/spi_dma.c:297: testTask
I've pushed the changes to https://github.com/carlk3/spi_dma/blob/master/spi_dma.c.

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Sun Mar 14, 2021 5:56 pm

fivdi wrote:
Sun Mar 14, 2021 9:28 am
...

If adding the call vTaskDelay(pdMS_TO_TICKS(2)) directly after the call to spi_transfer() resolves the issue and I had to take a wild guess why this resolves the issue, I would say that spi_transfer is returning before the transfer has fully completed and that it needs to wait for something else to happen before it returns.
That was my first thought, too. At first, I thought waiting for the DMA to raise IRQ line 0 when the RX channel finishes a block would be sufficient. When that failed, I added

Code: Select all

    dma_channel_wait_for_finish_blocking(this->tx_dma);
    dma_channel_wait_for_finish_blocking(this->rx_dma);
just to be sure. Still, no luck. I sprinkled in some

Code: Select all

    configASSERT(!dma_channel_is_busy(this->tx_dma));
    configASSERT(!dma_channel_is_busy(this->rx_dma));
but those have never triggered.

audryhome
Posts: 9
Joined: Thu Mar 17, 2016 7:17 am

Re: Context switching breaks SPI/DMA

Sun Mar 14, 2021 10:30 pm

I noticed the same phenomenon on simple DMA-SPI transfers with interrupts, no RTOS, just the sdk and C.
DMA interrupt handler is fired before alld data is transmitted, so you need to sync with the spi registers or to add a delay (ugly) if you don't want your dma handler act too soon, IMO this normal, since the DMA engien finishes his job after having posted last data item into the SPI FIFO, doesn't car about SPI registers.

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 4:23 am

audryhome wrote:
Sun Mar 14, 2021 10:30 pm
I noticed the same phenomenon on simple DMA-SPI transfers with interrupts, no RTOS, just the sdk and C.
DMA interrupt handler is fired before alld data is transmitted, so you need to sync with the spi registers or to add a delay (ugly) if you don't want your dma handler act too soon, IMO this normal, since the DMA engien finishes his job after having posted last data item into the SPI FIFO, doesn't car about SPI registers.
That makes good sense, but I don't think it's the problem in this case, because my logic analyzer shows that the ISR is entered after the SPI transfer is complete:
Screenshot from 2021-03-14 22-17-21.png
Screenshot from 2021-03-14 22-17-21.png (63.82 KiB) Viewed 1905 times
Screenshot from 2021-03-14 22-15-06.png
Screenshot from 2021-03-14 22-15-06.png (102.47 KiB) Viewed 1905 times
The line labeled "DMA cmplt" is toggled in the ISR itself.

EDIT: Also, I'm triggering on the receive DMA complete event:

Code: Select all

    
    // Tell the DMA to raise IRQ line 0 when the channel finishes a block
    dma_channel_set_irq0_enabled(this->rx_dma, true);
    irq_set_enabled(DMA_IRQ_0, true);
So, that should happen when the DMA has pulled the last byte out of the RX FIFO and copied it to memory, right? And, since SPI is synchronous, of the receive is complete, then so must be the transmit.

But I will peek at the SPI status register, anyway.
Last edited by carlk3 on Mon Mar 15, 2021 5:42 am, edited 1 time in total.

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 5:13 am

kilograham wrote:
Sun Mar 14, 2021 4:58 pm
...
My best debugging advice is to twiddle some pins at meaningful times and look on a logic probe (or a pico being a logic probe!)
I wasn't too sure what I'd see since I'm using SPI loopback. Looks like I get everything except MISO, which I guess makes sense, since that's an input:
Screenshot from 2021-03-14 22-48-55.png
Screenshot from 2021-03-14 22-48-55.png (124.22 KiB) Viewed 1894 times
I added some GPIOs:
  • * DMA cmplt: Gets toggled in the RX DMA IRQ handler, and in the task code after notification from the IRQ handler.
    * Task 0: Turned on in void vTaskSwitchContext( void ) when T0 is the new task, off otherwise.
    * Task 1: Turned on in void vTaskSwitchContext( void ) when T1 is the new task, off otherwise.
    * Trigger: Turned on when a data mismatch is detected.
In this case, the error was "T1: Done. Checking...T1: /home/carlk/pi/pico/spi_dma/spi_dma.c:291: testTask: Mismatch at 108/1024: expected 76, got 44". That comes from this block of code:

Code: Select all

        // Has the transmit buffer changed?!
        rand_st = seed;
        for (uint i = 0; i < TEST_SIZE; ++i) {
            uint8_t x = rand_r(&rand_st);
            if (txbufs[task_no][i] != x) {
                FAIL("txbuf", txbufs[task_no], TEST_SIZE, seed,
                    "Mismatch at %d/%d: expected %02x, got %02x\n", i, TEST_SIZE,
                    x, txbufs[task_no][i]);
            }
        }
Here is expected data from the pseudo-random number generator:

Code: Select all

Expected:
80 90 e0 3a 44 1c 06 56 d6 f7 31 c2 6a 04 86 c3 d5 80 31 08 b5 d7 cb f7 6c c3 8b 53 35 d9 f4 61 
21 86 5e aa d0 e0 fe ae 9f 7d dc 60 d3 89 50 f1 22 b7 e2 bd 1e 23 2d 15 3e 72 74 87 73 40 c2 28 
fe bf a2 0b 79 89 40 b1 a3 f4 c3 3b f8 06 e1 e2 e9 10 82 d8 ef d0 de 5c ff 71 1d a7 ac 41 4b 6c 
34 eb 22 35 c4 bb a4 16 93 49 e6 84 44 3f fa df c6 f2 cf 6b 14 9b 5b 10 0b a3 00 35 83 5a f8 1e 
17 28 71 d3 13 57 b1 b1 bf 12 cb ec 4b fa 47 92 7d 34 12 c8 a0 3e 3b 7b 90 99 28 d0 90 53 cf 66 
9f 3c b1 1a 03 73 36 12 79 34 10 71 8b 14 09 2e 20 91 e4 ba ac b9 67 f7 d3 9b 56 f4 a0 6e 3c 40 
cc e8 e0 3b 79 d0 b8 69 58 d9 7e 7a 6c a5 c4 c1 8d b4 c9 f0 b8 5d 6f db 66 da de 82 f1 43 bb 08 
db be 55 d7 19 f5 6e 71 45 11 b9 fe 9b b9 a8 c1 70 b8 bb 67 ae 1b 7f 13 d0 ce 00 00 62 dd a8 b5 
d7 d7 9a 6b da ea 8c 0a e1 d0 1b 1b 8a b1 f9 f9 31 22 58 a4 63 64 36 24 18 01 18 fd 1f e8 11 b2 
4c 96 ea 94 fe a0 9d 8d 9c 51 d9 65 d9 60 55 51 07 4c 1d a8 59 0f e1 c3 c6 4b 2a 59 92 94 3e 6e 
e8 c7 de a8 11 43 a7 cb c2 37 a1 af b2 39 3c 31 23 0a af a0 55 56 c1 5d dc 78 3a 78 a9 ba c2 54 
b3 4d 7e 13 d5 ac c5 c2 ea 01 a3 d2 c5 f5 a2 fc 9f ec d8 ba 68 82 14 da 5a 37 4a 60 fe d9 a3 60 
15 80 3f 8a 14 73 e4 5f b7 00 4a 42 08 c5 31 1b b1 50 d4 da 44 a3 e8 2f 57 a7 01 8d c2 63 1c 87 
09 94 5c 26 06 3d 3a c2 39 13 24 ea 59 c9 d7 aa 87 7e 45 00 ed 83 30 be 49 de 56 c4 04 0f 7c 98 
ac ce 4d 10 46 fa ad eb 7c 21 a0 36 e8 f2 1f 61 d8 b9 a1 d5 0e 9a 8e c6 00 16 cc 37 8a d7 30 eb 
8f 7c 83 8a bf e2 77 20 2e 62 3f 5f 56 57 60 84 1d 9c 48 d3 ce 2f 9c 9d 53 95 df 94 06 54 0f f9 
bd 4f f4 82 4b dc bf 86 b8 c2 62 08 38 fd 3b aa b9 df 9a 15 2f 78 fb a0 09 0f c3 94 c2 0d eb af 
b6 a9 5b 49 ef 61 eb d9 31 77 4d 2e c5 3a 33 9e 0a 35 bc 25 0d 1b 66 c3 d5 ee 1f 5b a1 31 0a ba 
1d e5 3e 00 22 55 b6 97 7b 91 e5 79 15 4d 88 ac a5 41 e9 6a 5c aa e6 d0 85 85 93 e2 87 42 f6 9c 
9a 8b 59 da 3f a1 5b b0 19 1d b7 3c 6e fd b8 04 c3 04 e0 cb e9 aa 1d 44 b7 33 7e fb ec ef a7 12 
ff d1 bb f7 d8 68 45 c9 14 f8 e9 93 85 ab 0e 8c c2 02 47 6d c2 56 5e d9 02 98 d5 2c 88 92 f5 1c 
b0 cb 15 03 58 e1 db 09 cb f6 f7 37 40 ea f5 4d dc 42 3e 94 fc 49 a2 c4 59 55 8c 46 4e 55 3f c9 
cb 38 93 40 dd 84 86 42 29 84 1f 0d b1 c6 cc ce f7 e2 41 3e d3 d9 b4 3e 3d da c4 a3 4d a3 5a c3 
78 9e b4 4c e4 75 57 e9 43 1d 12 73 20 d3 80 f7 a3 6d a3 79 c0 8f b5 59 37 0d 62 8c b8 ac 26 60 
ab 2a 87 c9 3e 19 8e 7c ee 3e 2f 67 14 14 07 13 3b b0 f1 b6 b4 d3 a7 87 e8 72 75 f3 e2 95 c9 04 
e6 c2 ee 23 6f cb 31 67 c3 f8 b5 4a df f0 2b 18 df 5c 70 09 5a 70 2e 0b 14 35 8c d6 48 e8 f0 62 
51 f2 70 be 8e b0 b0 d8 8f 89 12 3d da 1e 32 7c d0 57 74 2e 6d 65 b6 89 3f 3d 39 bf 5e f7 a6 92 
0e 8a 7c 34 1e 7b ea 99 10 19 ab 49 b8 4a b7 2f 30 cc b6 93 d5 7f d9 b3 ec 68 e5 af 81 3d d1 d2 
52 66 b0 ad 31 2a 58 92 25 d5 6d a8 12 4d 00 f5 b0 42 ac 21 a5 ea 0f 51 06 3a 30 93 60 d5 01 e5 
47 a4 35 29 41 5c 6b 1d bb 2f 86 09 99 c8 6a b6 6b 25 97 2d 77 cb af bb d0 78 b3 86 b8 cc 5f 6e 
10 0d d5 6a 4a d4 fd 02 f5 18 fb c2 23 dc 4b eb ea d4 dd d2 2c 44 d5 a9 6f 38 75 23 f9 8f b5 6b 
7b aa 43 c4 68 f7 d4 1f f8 cc 10 1f 71 99 14 6b 4d 4f 6b d5 b2 ef 55 0e 80 d6 d0 3a c9 04 2c ed 
Here is the TX buffer:

Code: Select all

hexdump_8(txbuf, 0x20002588, 1024)
80 90 e0 3a 44 1c 06 56 d6 f7 31 c2 6a 04 86 c3 d5 80 31 08 b5 d7 cb f7 6c c3 8b 53 35 d9 f4 61 
21 86 5e aa d0 e0 fe ae 9f 7d dc 60 d3 89 50 f1 22 b7 e2 bd 1e 23 2d 15 3e 72 74 87 73 40 c2 28 
fe bf a2 0b 79 89 40 b1 a3 f4 c3 3b f8 06 e1 e2 e9 10 82 d8 ef d0 de 5c ff 71 1d a7 ac 41 4b 6c 
34 eb 22 35 c4 bb a4 16 93 49 e6 84 44 3f fa df c6 f2 cf 6b 14 9b 5b 10 0b a3 00 35 83 5a f8 1e 
17 28 71 d3 13 57 b1 b1 bf 12 cb ec 4b fa 47 92 7d 34 12 c8 a0 3e 3b 7b 90 99 28 d0 90 53 cf 66 
9f 3c b1 1a 03 73 36 12 79 34 10 71 8b 14 09 2e 20 91 e4 ba ac b9 67 f7 d3 9b 56 f4 a0 6e 3c 40 
cc e8 e0 3b 79 d0 b8 69 58 d9 7e 7a 6c a5 c4 c1 8d b4 c9 f0 b8 5d 6f db 66 da de 82 f1 43 bb 08 
db be 55 d7 19 f5 6e 71 45 11 b9 fe 9b b9 a8 c1 70 b8 bb 67 ae 1b 7f 13 d0 ce 00 00 62 dd a8 b5 
d7 d7 9a 6b da ea 8c 0a e1 d0 1b 1b 8a b1 f9 f9 31 22 58 a4 63 64 36 24 18 01 18 fd 1f e8 11 b2 
4c 96 ea 94 fe a0 9d 8d 9c 51 d9 65 d9 60 55 51 07 4c 1d a8 59 0f e1 c3 c6 4b 2a 59 92 94 3e 6e 
e8 c7 de a8 11 43 a7 cb c2 37 a1 af b2 39 3c 31 23 0a af a0 55 56 c1 5d dc 78 3a 78 a9 ba c2 54 
b3 4d 7e 13 d5 ac c5 c2 ea 01 a3 d2 c5 f5 a2 fc 9f ec d8 ba 68 82 14 da 5a 37 4a 60 fe d9 a3 60 
15 80 3f 8a 14 73 e4 5f b7 00 4a 42 08 c5 31 1b b1 50 d4 da 44 a3 e8 2f 57 a7 01 8d c2 63 1c 87 
09 94 5c 26 06 3d 3a c2 39 13 24 ea 59 c9 d7 aa 87 7e 45 00 ed 83 30 be 49 de 56 c4 04 0f 7c 98 
ac ce 4d 10 46 fa ad eb 7c 21 a0 36 e8 f2 1f 61 d8 b9 a1 d5 0e 9a 8e c6 00 16 cc 37 8a d7 30 eb 
8f 7c 83 8a bf e2 77 20 2e 62 3f 5f 56 57 60 84 1d 9c 48 d3 ce 2f 9c 9d 53 95 df 94 06 54 0f f9 
bd 4f f4 82 4b dc bf 86 b8 c2 62 08 38 fd 3b aa b9 df 9a 15 2f 78 fb a0 09 0f c3 94 c2 0d eb af 
b6 a9 5b 49 ef 61 eb d9 31 77 4d 2e c5 3a 33 9e 0a 35 bc 25 0d 1b 66 c3 d5 ee 1f 5b a1 31 0a ba 
1d e5 3e 00 22 55 b6 97 7b 91 e5 79 15 4d 88 ac a5 41 e9 6a 5c aa e6 d0 85 85 93 e2 87 42 f6 9c 
9a 8b 59 da 3f a1 5b b0 19 1d b7 3c 6e fd b8 04 c3 04 e0 cb e9 aa 1d 44 b7 33 7e fb ec ef a7 12 
ff d1 bb f7 d8 68 45 c9 14 f8 e9 93 85 ab 0e 8c c2 02 47 6d c2 56 5e d9 02 98 d5 2c 88 92 f5 1c 
b0 cb 15 03 58 e1 db 09 cb f6 f7 37 40 ea f5 4d dc 42 3e 94 fc 49 a2 c4 59 55 8c 46 4e 55 3f c9 
cb 38 93 40 dd 84 86 42 29 84 1f 0d b1 c6 cc ce f7 e2 41 3e d3 d9 b4 3e 3d da c4 a3 4d a3 5a c3 
78 9e b4 4c e4 75 57 e9 43 1d 12 73 20 d3 80 f7 a3 6d a3 79 c0 8f b5 59 37 0d 62 8c b8 ac 26 60 
ab 2a 87 c9 3e 19 8e 7c ee 3e 2f 67 14 14 07 13 3b b0 f1 b6 b4 d3 a7 87 e8 72 75 f3 e2 95 c9 04 
e6 c2 ee 23 6f cb 31 67 c3 f8 b5 4a df f0 2b 18 df 5c 70 09 5a 70 2e 0b 14 35 8c d6 48 e8 f0 62 
51 f2 70 be 8e b0 b0 d8 8f 89 12 3d da 1e 32 7c d0 57 74 2e 6d 65 b6 89 3f 3d 39 bf 5e f7 a6 92 
0e 8a 7c 34 1e 7b ea 99 10 19 ab 49 b8 4a b7 2f 30 cc b6 93 d5 7f d9 b3 ec 68 e5 af 81 3d d1 d2 
52 66 b0 ad 31 2a 58 92 25 d5 6d a8 12 4d 00 f5 b0 42 ac 21 a5 ea 0f 51 06 3a 30 93 60 d5 01 e5 
47 a4 35 29 41 5c 6b 1d bb 2f 86 09 99 c8 6a b6 6b 25 97 2d 77 cb af bb d0 78 b3 86 b8 cc 5f 6e 
10 0d d5 6a 4a d4 fd 02 f5 18 fb c2 23 dc 4b eb ea d4 dd d2 2c 44 d5 a9 6f 38 75 23 f9 8f b5 6b 
7b aa 43 c4 68 f7 d4 1f f8 cc 10 1f 71 99 14 6b 4d 4f 6b d5 b2 ef 55 0e 80 d6 d0 3a c9 04 2c ed 
and here is what the logic analyzer saw on MOSI:

Code: Select all

80 90 E0 3A 44 1C 06 56 D6 F7 31 C2 6A 04 86 C3 D5 80 31 08 B5 D7 CB F7 6C C3 8B 53 35 D9 F4 61
21 86 5E AA D0 E0 FE AE 9F 7D DC 60 D3 89 50 F1 22 B7 E2 BD 1E 23 2D 15 3E 72 74 87 73 40 C2 28
FE BF A2 0B 79 89 40 B1 A3 F4 C3 3B F8 06 E1 E2 E9 10 82 D8 EF D0 DE 5C FF 71 1D A7 AC 41 4B 6C
34 EB 22 35 C4 BB A4 16 93 49 E6 84 44 3F FA DF C6 F2 CF 6B 14 9B 5B 10 0B A3 00 35 83 5A F8 1E
17 28 71 D3 13 57 B1 B1 BF 12 CB EC 4B FA 47 92 7D 34 12 C8 A0 3E 3B 7B 90 99 28 D0 90 53 CF 66
9F 3C B1 1A 03 73 36 12 79 34 10 71 8B 14 09 2E 20 91 E4 BA AC B9 67 F7 D3 9B 56 F4 A0 6E 3C 40
CC E8 E0 3B 79 D0 B8 69 58 D9 7E 7A 6C A5 C4 C1 8D B4 C9 F0 B8 5D 6F DB 66 DA DE 82 F1 43 BB 08
DB BE 55 D7 19 F5 6E 71 45 11 B9 FE 9B B9 A8 C1 70 B8 BB 67 AE 1B 7F 13 D0 CE 00 00 62 DD A8 B5
D7 D7 9A 6B DA EA 8C 0A E1 D0 1B 1B 8A B1 F9 F9 31 22 58 A4 63 64 36 24 18 01 18 FD 1F E8 11 B2
4C 96 EA 94 FE A0 9D 8D 9C 51 D9 65 D9 60 55 51 07 4C 1D A8 59 0F E1 C3 C6 4B 2A 59 92 94 3E 6E
E8 C7 DE A8 11 43 A7 CB C2 37 A1 AF B2 39 3C 31 23 0A AF A0 55 56 C1 5D DC 78 3A 78 A9 BA C2 54
B3 4D 7E 13 D5 AC C5 C2 EA 01 A3 D2 C5 F5 A2 FC 9F EC D8 BA 68 82 14 DA 5A 37 4A 60 FE D9 A3 60
15 80 3F 8A 14 73 E4 5F B7 00 4A 42 08 C5 31 1B B1 50 D4 DA 44 A3 E8 2F 57 A7 01 8D C2 63 1C 87
09 94 5C 26 06 3D 3A C2 39 13 24 EA 59 C9 D7 AA 87 7E 45 00 ED 83 30 BE 49 DE 56 C4 04 0F 7C 98
AC CE 4D 10 46 FA AD EB 7C 21 A0 36 E8 F2 1F 61 D8 B9 A1 D5 0E 9A 8E C6 00 16 CC 37 8A D7 30 EB
8F 7C 83 8A BF E2 77 20 2E 62 3F 5F 56 57 60 84 1D 9C 48 D3 CE 2F 9C 9D 53 95 DF 94 06 54 0F F9
BD 4F F4 82 4B DC BF 86 B8 C2 62 08 38 FD 3B AA B9 DF 9A 15 2F 78 FB A0 09 0F C3 94 C2 0D EB AF
B6 A9 5B 49 EF 61 EB D9 31 77 4D 2E C5 3A 33 9E 0A 35 BC 25 0D 1B 66 C3 D5 EE 1F 5B A1 31 0A BA
1D E5 3E 00 22 55 B6 97 7B 91 E5 79 15 4D 88 AC A5 41 E9 6A 5C AA E6 D0 85 85 93 E2 87 42 F6 9C
9A 8B 59 DA 3F A1 5B B0 19 1D B7 3C 6E FD B8 04 C3 04 E0 CB E9 AA 1D 44 B7 33 7E FB EC EF A7 12
FF D1 BB F7 D8 68 45 C9 14 F8 E9 93 85 AB 0E 8C C2 02 47 6D C2 56 5E D9 02 98 D5 2C 88 92 F5 1C
B0 CB 15 03 58 E1 DB 09 CB F6 F7 37 40 EA F5 4D DC 42 3E 94 FC 49 A2 C4 59 55 8C 46 4E 55 3F C9
CB 38 93 40 DD 84 86 42 29 84 1F 0D B1 C6 CC CE F7 E2 41 3E D3 D9 B4 3E 3D DA C4 A3 4D A3 5A C3
78 9E B4 4C E4 75 57 E9 43 1D 12 73 20 D3 80 F7 A3 6D A3 79 C0 8F B5 59 37 0D 62 8C B8 AC 26 60
AB 2A 87 C9 3E 19 8E 7C EE 3E 2F 67 14 14 07 13 3B B0 F1 B6 B4 D3 A7 87 E8 72 75 F3 E2 95 C9 04
E6 C2 EE 23 6F CB 31 67 C3 F8 B5 4A DF F0 2B 18 DF 5C 70 09 5A 70 2E 0B 14 35 8C D6 48 E8 F0 62
51 F2 70 BE 8E B0 B0 D8 8F 89 12 3D DA 1E 32 7C D0 57 74 2E 6D 65 B6 89 3F 3D 39 BF 5E F7 A6 92
0E 8A 7C 34 1E 7B EA 99 10 19 AB 49 B8 4A B7 2F 30 CC B6 93 D5 7F D9 B3 EC 68 E5 AF 81 3D D1 D2
52 66 B0 AD 31 2A 58 92 25 D5 6D A8 12 4D 00 F5 B0 42 AC 21 A5 EA 0F 51 06 3A 30 93 60 D5 01 E5
47 A4 35 29 41 5C 6B 1D BB 2F 86 09 99 C8 6A B6 6B 25 97 2D 77 CB AF BB D0 78 B3 86 B8 CC 5F 6E
10 0D D5 6A 4A D4 FD 02 F5 18 FB C2 23 DC 4B EB EA D4 DD D2 2C 44 D5 A9 6F 38 75 23 F9 8F B5 6B
7B AA 43 C4 68 F7 D4 1F F8 CC 10 1F 71 99 14 6B 4D 4F 6B D5 B2 EF 55 0E 80 D6 D0 3A C9 04 2C ED
If I diff the expected data, the tx buffer, and the logic analyzer capture, they all match. How bizarre!

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1033
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 1:03 pm

oh! i had assumed you were comparing sent with received, not with rand_r regenerated data.

rand_r is not thread safe

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 4:34 pm

kilograham wrote:
Mon Mar 15, 2021 1:03 pm
oh! i had assumed you were comparing sent with received, not with rand_r regenerated data.
I'm doing that, too. It compares the tx buffer with the generated sequence, the rx buffer with the generated sequence, and tx vs. rx. Any of the three can fail. I was really surprised to see the tx buffer change. Even more surprised to see it change and then change back.
kilograham wrote:
Mon Mar 15, 2021 1:03 pm
rand_r is not thread safe
I thought rand_r() is the reentrant version of the rand() function, for use with multithreaded applications. In what way is it not thread safe?

Plainsman
Posts: 15
Joined: Tue Jul 12, 2016 2:31 am
Location: Oregon, USA

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 5:06 pm

generic hardware debugging suggestions -

turn off optimization for *all* code generated: add -O0 to the end of CFLAGS or CFLAGS_EXTRA (whichever exists in your build environment). If your problems change significantly, you may be missing a volatile on some hardware related addressing decls -- even down deep into the libraries.


use sequential data for one TX buffer counting up, the same for the other but counting down. may help reduce the sizeof the universe of things you're looking at, make single bit errors more noticeable.

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 5:08 pm

audryhome wrote:
Sun Mar 14, 2021 10:30 pm
I noticed the same phenomenon on simple DMA-SPI transfers with interrupts, no RTOS, just the sdk and C.
DMA interrupt handler is fired before alld data is transmitted, so you need to sync with the spi registers or to add a delay (ugly) if you don't want your dma handler act too soon, IMO this normal, since the DMA engien finishes his job after having posted last data item into the SPI FIFO, doesn't car about SPI registers.
I have added some checks for SPI Transmit FIFO empty, SPI Receive FIFO empty, and SPI not busy. Now I have all these superfluous actions after the RX DMA complete IRQ has fired:

Code: Select all

    dma_channel_wait_for_finish_blocking(this->tx_dma);
    dma_channel_wait_for_finish_blocking(this->rx_dma);
    configASSERT(!dma_channel_is_busy(this->tx_dma));
    configASSERT(!dma_channel_is_busy(this->rx_dma));
    dma_channel_abort(this->tx_dma);
    dma_channel_abort(this->rx_dma);

    configASSERT(spi_get_hw(this->hw_inst)->sr &
                 SPI_SSPSR_TFE_BITS);  // SPI Transmit FIFO should be empty
    configASSERT(!(spi_get_hw(this->hw_inst)->sr &
                   SPI_SSPSR_RNE_BITS));  // SPI Receive FIFO should be empty
    configASSERT(!(spi_get_hw(this->hw_inst)->sr &
                   SPI_SSPSR_BSY_BITS));  // SPI should not be busy
                   
but none of these made a difference; it still fails data checks.

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1033
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 6:16 pm

carlk3 wrote:
Mon Mar 15, 2021 4:34 pm
I thought rand_r() is the reentrant version of the rand() function, for use with multithreaded applications. In what way is it not thread safe?
quite right!

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1033
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 6:23 pm

are you sure the task_no are making it thru correctly, and not all 0?

fivdi
Posts: 464
Joined: Sun Sep 23, 2012 8:09 pm
Contact: Website

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 8:54 pm

kilograham wrote:
Mon Mar 15, 2021 6:16 pm
carlk3 wrote:
Mon Mar 15, 2021 4:34 pm
I thought rand_r() is the reentrant version of the rand() function, for use with multithreaded applications. In what way is it not thread safe?
quite right!

Assuming newlib version 3.1.0 is being used, the source for rand_r can be found at https://github.com/bminor/newlib/blob/n ... b/rand_r.c (or at https://sourceware.org/git/?p=newlib-cy ... 92509c8a5d).

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 10:13 pm

Major discovery: I removed SPI completely and I can still recreate the problem. This greatly simplifies the test case: https://github.com/carlk3/spi_dma/blob/no-spi/spi_dma.c

Output from a typical run:

Code: Select all

example
T0: testTask(task_no=0)
T1: testTask(task_no=1)
T0: Done. Checking...T1: Done. Checking...T0: All good
T0: Stack High Water Mark: 1348
T1: All good
T1: Stack High Water Mark: 1348
T1: Done. Checking...T0: Done. Checking...T1: All good
T0: All good
T1: Stack High Water Mark: 1348
T1: Done. Checking...T0: Stack High Water Mark: 1348
T1: All good
T1: Stack High Water Mark: 1348
T0: Done. Checking...T1: Done. Checking...T0: All good
T0: Stack High Water Mark: 1348
T1: All good
T0: Done. Checking...T1: Stack High Water Mark: 1348
T0: /home/carlk/pi/pico/spi_dma/spi_dma.c:66: testTask
: Mismatch at 809/1024: expected 1b, got 5b
T1: Done. Checking...
T0: hexdump_8(rxbuf, 0x20001938, 1024)
f5 d3 8d 7f 87 58 88 fc cb e9 c9 00 a6 a6 19 d6 47 33 c9 fc 8e 35 88 11 0a e0 41 83 99 17 cd b8 
d8 99 28 c3 da 29 6b b0 1d 92 75 5a c8 23 25 e3 06 a0 ad 84 8c 28 3d e6 d5 1e 10 35 80 5f b0 d8 
bd 50 d3 10 4e 3f 9f b0 d1 e9 ac e3 e7 b8 e5 11 7b fa c7 f1 81 da 8d ae 13 f0 44 49 f4 a7 61 0c 
25 a5 12 c9 48 24 fd 9c e3 5b 4d f7 a3 f1 bd 27 84 af 4a b0 5a 2d 42 84 cc ff 6e 59 01 f2 5d 16 
9a 16 e4 20 a6 e6 ee 98 de 98 36 39 6b 5f d7 23 02 ab 68 c6 c5 2b 9d 28 15 85 1a d6 e3 09 7c ac 
11 e7 e0 26 d2 94 d5 4c 8c a3 7e 08 59 36 2c b3 12 95 cf fa 1c 74 1a fb ee f3 ea 31 ad 1e d1 81 
26 63 4d cb d0 c2 86 4a 66 ef e8 76 a6 a5 9e 44 07 f0 74 9b 74 b0 3c 0b d2 65 bd 13 27 a5 62 88 
8f 7e d8 1b a4 49 da 72 5b d8 52 89 ed 0e 53 54 82 1e 86 d8 7a da de 4c d4 d2 bc c2 ba 59 01 84 
bb 1b 59 7c 53 77 8b 4c ad 06 42 2c c3 2c a1 41 b9 d0 b6 bf 8a ba 6b 5a 48 2c ee 09 38 99 f4 0c 
97 39 05 1b ad 21 2d 5d af 5f 55 3a 11 ea a4 8b c8 95 5e 07 5a 3e 93 c8 70 81 4e d0 d7 6d f3 74 
cb 4c 99 0d b8 af 9b 50 5e b5 47 4c d8 a9 9b 31 b2 20 4a 71 ba 58 4a d8 1b 44 21 9c a1 5c 0c 07 
2c 31 98 26 69 2c c4 e6 5b 62 df 7a 92 27 e9 81 e9 97 6d 7e 4e 83 ce 0b 1c 5f 51 e6 c9 1b 53 32 
ca 7d 61 41 24 9a 11 00 c2 dc c3 bb 74 d6 19 6a a8 eb 1c 6b 33 b5 47 b7 fa e9 16 31 68 cc 96 51 
ba 3f aa 24 fc 5b d1 ba 19 98 c2 df 5e 06 0f a3 8f 9a 87 3e c5 d7 09 94 55 47 8e 20 d8 ea c6 91 
ac be 3f 1e 5f 35 2c 59 d2 92 41 f1 23 a9 ca f2 5f 86 c1 d2 6a 1b e3 78 5c 36 e2 d3 0f 09 f5 17 
65 3c 8b 83 48 a9 7c 8c 99 fe df 4a 8a 0e 16 19 78 b1 1f 60 82 99 51 1d f1 2d 09 1f b0 31 b2 b9 
52 33 ef a1 8f 89 3e 99 62 42 a8 54 ab 1b c5 fb 8c 81 3d 3c e5 1a bd 05 7a 70 74 af b0 20 97 f4 
1c 19 c2 c9 11 82 d7 b9 35 8c 39 cf b2 b9 a9 b5 6e 1f 9a d8 fa c2 2c 5c 3f 85 8e a4 d3 93 a4 f0 
76 d7 99 ee e4 25 66 a1 d2 73 6b 3e 78 fd bf 2a 2f 43 6f c2 94 51 b9 5a 7d 79 17 33 0b 55 13 17 
6f b3 4c 15 89 23 64 14 ea 2a 36 77 72 ad 06 d0 6e 44 97 8a e3 d9 78 fd 10 99 a4 23 f7 57 4f e8 
91 c4 2a ef 27 30 9b 1e bc ff 37 57 5d c2 94 37 e4 42 77 90 66 b2 aa c9 62 6b 9d 45 7b 4b a3 72 
06 20 a4 f0 c0 11 ef 2c ee c5 7d 59 2f a9 a9 fd e3 57 63 99 99 c1 5f ca ca 0e 35 c3 a1 aa 27 4e 
ce b7 a5 6b 13 23 11 41 92 65 ca 0e 3e 38 90 e6 bf 7f 99 39 62 53 1e 2f e5 3b 1c d3 21 23 00 3a 
aa 63 f2 a7 f7 0a 78 33 b3 fe 0c 20 bc 28 c3 27 79 2a 3b 6c bc cf 40 20 ed ec 2c 4d e6 4c ca 48 
e2 72 83 68 15 a0 41 f2 77 9d a3 0a c0 32 0c 32 03 80 cf c8 5a 40 5f b3 9d 84 9a 41 99 2b b0 e8 
87 f8 fb 22 c6 1c 33 44 f6 5b 2c c7 5f 37 0d 6a 7b 4c de ac 8e 30 25 44 0e d9 ff 57 55 9b f1 e0 
59 83 ba 0a af 6e 74 61 65 dd 4c cd 27 b2 ff b2 04 cc f0 3b 36 26 ec c5 fd 99 75 72 20 77 0d 97 
4c 3f ea e1 84 10 d5 cd be 72 08 5b 18 b7 14 a7 cf ae 0e 11 65 94 4b 1c ef e0 3b 72 6d ff 30 86 
17 5e 64 e8 23 e5 94 d9 18 13 6e b9 aa 49 96 f3 94 1f 4e 12 c3 85 a6 c9 76 91 27 01 5a 46 2e 47 
91 69 59 40 e5 16 b0 5a 62 dd 0d 56 a5 82 aa 5e e4 c8 2f 59 d2 aa 56 22 ba 56 d0 f5 c4 d2 b6 ce 
48 c6 c2 f0 65 b8 17 1e 4b 1c 9b 68 36 23 4b 91 27 88 09 6e c1 17 b2 45 b6 9d e4 3a 53 4b 04 e4 
28 5d 3d c6 12 91 6e aa cd d8 c8 4a ea fd ae 9a cf 07 aa c7 cc 8b fa 38 4d 3d a4 2b ec 6c cf 97 
Expected:
f5 d3 8d 7f 87 58 88 fc cb e9 c9 00 a6 a6 19 d6 47 33 c9 fc 8e 35 88 11 0a e0 41 83 99 17 cd b8 
d8 99 28 c3 da 29 6b b0 1d 92 75 5a c8 23 25 e3 06 a0 ad 84 8c 28 3d e6 d5 1e 10 35 80 5f b0 d8 
bd 50 d3 10 4e 3f 9f b0 d1 e9 ac e3 e7 b8 e5 11 7b fa c7 f1 81 da 8d ae 13 f0 44 49 f4 a7 61 0c 
25 a5 12 c9 48 24 fd 9c e3 5b 4d f7 a3 f1 bd 27 84 af 4a b0 5a 2d 42 84 cc ff 6e 59 01 f2 5d 16 
9a 16 e4 20 a6 e6 ee 98 de 98 36 39 6b 5f d7 23 02 ab 68 c6 c5 2b 9d 28 15 85 1a d6 e3 09 7c ac 
11 e7 e0 26 d2 94 d5 4c 8c a3 7e 08 59 36 2c b3 12 95 cf fa 1c 74 1a fb ee f3 ea 31 ad 1e d1 81 
26 63 4d cb d0 c2 86 4a 66 ef e8 76 a6 a5 9e 44 07 f0 74 9b 74 b0 3c 0b d2 65 bd 13 27 a5 62 88 
8f 7e d8 1b a4 49 da 72 5b d8 52 89 ed 0e 53 54 82 1e 86 d8 7a da de 4c d4 d2 bc c2 ba 59 01 84 
bb 1b 59 7c 53 77 8b 4c ad 06 42 2c c3 2c a1 41 b9 d0 b6 bf 8a ba 6b 5a 48 2c ee 09 38 99 f4 0c 
97 39 05 1b ad 21 2d 5d af 5f 55 3a 11 ea a4 8b c8 95 5e 07 5a 3e 93 c8 70 81 4e d0 d7 6d f3 74 
cb 4c 99 0d b8 af 9b 50 5e b5 47 4c d8 a9 9b 31 b2 20 4a 71 ba 58 4a d8 1b 44 21 9c a1 5c 0c 07 
2c 31 98 26 69 2c c4 e6 5b 62 df 7a 92 27 e9 81 e9 97 6d 7e 4e 83 ce 0b 1c 5f 51 e6 c9 1b 53 32 
ca 7d 61 41 24 9a 11 00 c2 dc c3 bb 74 d6 19 6a a8 eb 1c 6b 33 b5 47 b7 fa e9 16 31 68 cc 96 51 
ba 3f aa 24 fc 5b d1 ba 19 98 c2 df 5e 06 0f a3 8f 9a 87 3e c5 d7 09 94 55 47 8e 20 d8 ea c6 91 
ac be 3f 1e 5f 35 2c 59 d2 92 41 f1 23 a9 ca f2 5f 86 c1 d2 6a 1b e3 78 5c 36 e2 d3 0f 09 f5 17 
65 3c 8b 83 48 a9 7c 8c 99 fe df 4a 8a 0e 16 19 78 b1 1f 60 82 99 51 1d f1 2d 09 1f b0 31 b2 b9 
52 33 ef a1 8f 89 3e 99 62 42 a8 54 ab 1b c5 fb 8c 81 3d 3c e5 1a bd 05 7a 70 74 af b0 20 97 f4 
1c 19 c2 c9 11 82 d7 b9 35 8c 39 cf b2 b9 a9 b5 6e 1f 9a d8 fa c2 2c 5c 3f 85 8e a4 d3 93 a4 f0 
76 d7 99 ee e4 25 66 a1 d2 73 6b 3e 78 fd bf 2a 2f 43 6f c2 94 51 b9 5a 7d 79 17 33 0b 55 13 17 
6f b3 4c 15 89 23 64 14 ea 2a 36 77 72 ad 06 d0 6e 44 97 8a e3 d9 78 fd 10 99 a4 23 f7 57 4f e8 
91 c4 2a ef 27 30 9b 1e bc ff 37 57 5d c2 94 37 e4 42 77 90 66 b2 aa c9 62 6b 9d 45 7b 4b a3 72 
06 20 a4 f0 c0 11 ef 2c ee c5 7d 59 2f a9 a9 fd e3 57 63 99 99 c1 5f ca ca 0e 35 c3 a1 aa 27 4e 
ce b7 a5 6b 13 23 11 41 92 65 ca 0e 3e 38 90 e6 bf 7f 99 39 62 53 1e 2f e5 3b 1c d3 21 23 00 3a 
aa 63 f2 a7 f7 0a 78 33 b3 fe 0c 20 bc 28 c3 27 79 2a 3b 6c bc cf 40 20 ed ec 2c 4d e6 4c ca 48 
e2 72 83 68 15 a0 41 f2 77 9d a3 0a c0 32 0c 32 03 80 cf c8 5a 40 5f b3 9d 84 9a 41 99 2b b0 e8 
87 f8 fb 22 c6 1c 33 44 f6 5b 2c c7 5f 37 0d 6a 7b 4c de ac 8e 30 25 44 0e d9 ff 57 55 9b f1 e0 
59 83 ba 0a af 6e 74 61 65 dd 4c cd 27 b2 ff b2 04 cc f0 3b 36 26 ec c5 fd 99 75 72 20 77 0d 97 
4c 3f ea e1 84 10 d5 cd be 72 08 5b 18 b7 14 a7 cf ae 0e 11 65 94 4b 1c ef e0 3b 72 6d ff 30 86 
17 5e 64 e8 23 e5 94 d9 18 13 6e b9 aa 49 96 f3 94 1f 4e 12 c3 85 a6 c9 76 91 27 01 5a 46 2e 47 
91 69 59 40 e5 16 b0 5a 62 dd 0d 56 a5 82 aa 5e e4 c8 2f 59 d2 aa 56 22 ba 56 d0 f5 c4 d2 b6 ce 
48 c6 c2 f0 65 b8 17 1e 4b 1c 9b 68 36 23 4b 91 27 88 09 6e c1 17 b2 45 b6 9d e4 3a 53 4b 04 e4 
28 5d 3d c6 12 91 6e aa cd d8 c8 4a ea fd ae 9a cf 07 aa c7 cc 8b fa 38 4d 3d a4 2b ec 6c cf 97 
Once again, there is no difference between the expected data and the rxbuf when they are dumped.

ejolson
Posts: 8631
Joined: Tue Mar 18, 2014 11:47 am

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 11:26 pm

carlk3 wrote:
Mon Mar 15, 2021 10:13 pm
Major discovery: I removed SPI completely and I can still recreate the problem. This greatly simplifies the test case: https://github.com/carlk3/spi_dma/blob/no-spi/spi_dma.c

Output from a typical run:

Code: Select all

example
T0: testTask(task_no=0)
T1: testTask(task_no=1)
T0: Done. Checking...T1: Done. Checking...T0: All good
T0: Stack High Water Mark: 1348
T1: All good
T1: Stack High Water Mark: 1348
T1: Done. Checking...T0: Done. Checking...T1: All good
T0: All good
T1: Stack High Water Mark: 1348
T1: Done. Checking...T0: Stack High Water Mark: 1348
T1: All good
T1: Stack High Water Mark: 1348
T0: Done. Checking...T1: Done. Checking...T0: All good
T0: Stack High Water Mark: 1348
T1: All good
T0: Done. Checking...T1: Stack High Water Mark: 1348
T0: /home/carlk/pi/pico/spi_dma/spi_dma.c:66: testTask
: Mismatch at 809/1024: expected 1b, got 5b
T1: Done. Checking...
T0: hexdump_8(rxbuf, 0x20001938, 1024)
f5 d3 8d 7f 87 58 88 fc cb e9 c9 00 a6 a6 19 d6 47 33 c9 fc 8e 35 88 11 0a e0 41 83 99 17 cd b8 
d8 99 28 c3 da 29 6b b0 1d 92 75 5a c8 23 25 e3 06 a0 ad 84 8c 28 3d e6 d5 1e 10 35 80 5f b0 d8 
bd 50 d3 10 4e 3f 9f b0 d1 e9 ac e3 e7 b8 e5 11 7b fa c7 f1 81 da 8d ae 13 f0 44 49 f4 a7 61 0c 
25 a5 12 c9 48 24 fd 9c e3 5b 4d f7 a3 f1 bd 27 84 af 4a b0 5a 2d 42 84 cc ff 6e 59 01 f2 5d 16 
9a 16 e4 20 a6 e6 ee 98 de 98 36 39 6b 5f d7 23 02 ab 68 c6 c5 2b 9d 28 15 85 1a d6 e3 09 7c ac 
11 e7 e0 26 d2 94 d5 4c 8c a3 7e 08 59 36 2c b3 12 95 cf fa 1c 74 1a fb ee f3 ea 31 ad 1e d1 81 
26 63 4d cb d0 c2 86 4a 66 ef e8 76 a6 a5 9e 44 07 f0 74 9b 74 b0 3c 0b d2 65 bd 13 27 a5 62 88 
8f 7e d8 1b a4 49 da 72 5b d8 52 89 ed 0e 53 54 82 1e 86 d8 7a da de 4c d4 d2 bc c2 ba 59 01 84 
bb 1b 59 7c 53 77 8b 4c ad 06 42 2c c3 2c a1 41 b9 d0 b6 bf 8a ba 6b 5a 48 2c ee 09 38 99 f4 0c 
97 39 05 1b ad 21 2d 5d af 5f 55 3a 11 ea a4 8b c8 95 5e 07 5a 3e 93 c8 70 81 4e d0 d7 6d f3 74 
cb 4c 99 0d b8 af 9b 50 5e b5 47 4c d8 a9 9b 31 b2 20 4a 71 ba 58 4a d8 1b 44 21 9c a1 5c 0c 07 
2c 31 98 26 69 2c c4 e6 5b 62 df 7a 92 27 e9 81 e9 97 6d 7e 4e 83 ce 0b 1c 5f 51 e6 c9 1b 53 32 
ca 7d 61 41 24 9a 11 00 c2 dc c3 bb 74 d6 19 6a a8 eb 1c 6b 33 b5 47 b7 fa e9 16 31 68 cc 96 51 
ba 3f aa 24 fc 5b d1 ba 19 98 c2 df 5e 06 0f a3 8f 9a 87 3e c5 d7 09 94 55 47 8e 20 d8 ea c6 91 
ac be 3f 1e 5f 35 2c 59 d2 92 41 f1 23 a9 ca f2 5f 86 c1 d2 6a 1b e3 78 5c 36 e2 d3 0f 09 f5 17 
65 3c 8b 83 48 a9 7c 8c 99 fe df 4a 8a 0e 16 19 78 b1 1f 60 82 99 51 1d f1 2d 09 1f b0 31 b2 b9 
52 33 ef a1 8f 89 3e 99 62 42 a8 54 ab 1b c5 fb 8c 81 3d 3c e5 1a bd 05 7a 70 74 af b0 20 97 f4 
1c 19 c2 c9 11 82 d7 b9 35 8c 39 cf b2 b9 a9 b5 6e 1f 9a d8 fa c2 2c 5c 3f 85 8e a4 d3 93 a4 f0 
76 d7 99 ee e4 25 66 a1 d2 73 6b 3e 78 fd bf 2a 2f 43 6f c2 94 51 b9 5a 7d 79 17 33 0b 55 13 17 
6f b3 4c 15 89 23 64 14 ea 2a 36 77 72 ad 06 d0 6e 44 97 8a e3 d9 78 fd 10 99 a4 23 f7 57 4f e8 
91 c4 2a ef 27 30 9b 1e bc ff 37 57 5d c2 94 37 e4 42 77 90 66 b2 aa c9 62 6b 9d 45 7b 4b a3 72 
06 20 a4 f0 c0 11 ef 2c ee c5 7d 59 2f a9 a9 fd e3 57 63 99 99 c1 5f ca ca 0e 35 c3 a1 aa 27 4e 
ce b7 a5 6b 13 23 11 41 92 65 ca 0e 3e 38 90 e6 bf 7f 99 39 62 53 1e 2f e5 3b 1c d3 21 23 00 3a 
aa 63 f2 a7 f7 0a 78 33 b3 fe 0c 20 bc 28 c3 27 79 2a 3b 6c bc cf 40 20 ed ec 2c 4d e6 4c ca 48 
e2 72 83 68 15 a0 41 f2 77 9d a3 0a c0 32 0c 32 03 80 cf c8 5a 40 5f b3 9d 84 9a 41 99 2b b0 e8 
87 f8 fb 22 c6 1c 33 44 f6 5b 2c c7 5f 37 0d 6a 7b 4c de ac 8e 30 25 44 0e d9 ff 57 55 9b f1 e0 
59 83 ba 0a af 6e 74 61 65 dd 4c cd 27 b2 ff b2 04 cc f0 3b 36 26 ec c5 fd 99 75 72 20 77 0d 97 
4c 3f ea e1 84 10 d5 cd be 72 08 5b 18 b7 14 a7 cf ae 0e 11 65 94 4b 1c ef e0 3b 72 6d ff 30 86 
17 5e 64 e8 23 e5 94 d9 18 13 6e b9 aa 49 96 f3 94 1f 4e 12 c3 85 a6 c9 76 91 27 01 5a 46 2e 47 
91 69 59 40 e5 16 b0 5a 62 dd 0d 56 a5 82 aa 5e e4 c8 2f 59 d2 aa 56 22 ba 56 d0 f5 c4 d2 b6 ce 
48 c6 c2 f0 65 b8 17 1e 4b 1c 9b 68 36 23 4b 91 27 88 09 6e c1 17 b2 45 b6 9d e4 3a 53 4b 04 e4 
28 5d 3d c6 12 91 6e aa cd d8 c8 4a ea fd ae 9a cf 07 aa c7 cc 8b fa 38 4d 3d a4 2b ec 6c cf 97 
Expected:
f5 d3 8d 7f 87 58 88 fc cb e9 c9 00 a6 a6 19 d6 47 33 c9 fc 8e 35 88 11 0a e0 41 83 99 17 cd b8 
d8 99 28 c3 da 29 6b b0 1d 92 75 5a c8 23 25 e3 06 a0 ad 84 8c 28 3d e6 d5 1e 10 35 80 5f b0 d8 
bd 50 d3 10 4e 3f 9f b0 d1 e9 ac e3 e7 b8 e5 11 7b fa c7 f1 81 da 8d ae 13 f0 44 49 f4 a7 61 0c 
25 a5 12 c9 48 24 fd 9c e3 5b 4d f7 a3 f1 bd 27 84 af 4a b0 5a 2d 42 84 cc ff 6e 59 01 f2 5d 16 
9a 16 e4 20 a6 e6 ee 98 de 98 36 39 6b 5f d7 23 02 ab 68 c6 c5 2b 9d 28 15 85 1a d6 e3 09 7c ac 
11 e7 e0 26 d2 94 d5 4c 8c a3 7e 08 59 36 2c b3 12 95 cf fa 1c 74 1a fb ee f3 ea 31 ad 1e d1 81 
26 63 4d cb d0 c2 86 4a 66 ef e8 76 a6 a5 9e 44 07 f0 74 9b 74 b0 3c 0b d2 65 bd 13 27 a5 62 88 
8f 7e d8 1b a4 49 da 72 5b d8 52 89 ed 0e 53 54 82 1e 86 d8 7a da de 4c d4 d2 bc c2 ba 59 01 84 
bb 1b 59 7c 53 77 8b 4c ad 06 42 2c c3 2c a1 41 b9 d0 b6 bf 8a ba 6b 5a 48 2c ee 09 38 99 f4 0c 
97 39 05 1b ad 21 2d 5d af 5f 55 3a 11 ea a4 8b c8 95 5e 07 5a 3e 93 c8 70 81 4e d0 d7 6d f3 74 
cb 4c 99 0d b8 af 9b 50 5e b5 47 4c d8 a9 9b 31 b2 20 4a 71 ba 58 4a d8 1b 44 21 9c a1 5c 0c 07 
2c 31 98 26 69 2c c4 e6 5b 62 df 7a 92 27 e9 81 e9 97 6d 7e 4e 83 ce 0b 1c 5f 51 e6 c9 1b 53 32 
ca 7d 61 41 24 9a 11 00 c2 dc c3 bb 74 d6 19 6a a8 eb 1c 6b 33 b5 47 b7 fa e9 16 31 68 cc 96 51 
ba 3f aa 24 fc 5b d1 ba 19 98 c2 df 5e 06 0f a3 8f 9a 87 3e c5 d7 09 94 55 47 8e 20 d8 ea c6 91 
ac be 3f 1e 5f 35 2c 59 d2 92 41 f1 23 a9 ca f2 5f 86 c1 d2 6a 1b e3 78 5c 36 e2 d3 0f 09 f5 17 
65 3c 8b 83 48 a9 7c 8c 99 fe df 4a 8a 0e 16 19 78 b1 1f 60 82 99 51 1d f1 2d 09 1f b0 31 b2 b9 
52 33 ef a1 8f 89 3e 99 62 42 a8 54 ab 1b c5 fb 8c 81 3d 3c e5 1a bd 05 7a 70 74 af b0 20 97 f4 
1c 19 c2 c9 11 82 d7 b9 35 8c 39 cf b2 b9 a9 b5 6e 1f 9a d8 fa c2 2c 5c 3f 85 8e a4 d3 93 a4 f0 
76 d7 99 ee e4 25 66 a1 d2 73 6b 3e 78 fd bf 2a 2f 43 6f c2 94 51 b9 5a 7d 79 17 33 0b 55 13 17 
6f b3 4c 15 89 23 64 14 ea 2a 36 77 72 ad 06 d0 6e 44 97 8a e3 d9 78 fd 10 99 a4 23 f7 57 4f e8 
91 c4 2a ef 27 30 9b 1e bc ff 37 57 5d c2 94 37 e4 42 77 90 66 b2 aa c9 62 6b 9d 45 7b 4b a3 72 
06 20 a4 f0 c0 11 ef 2c ee c5 7d 59 2f a9 a9 fd e3 57 63 99 99 c1 5f ca ca 0e 35 c3 a1 aa 27 4e 
ce b7 a5 6b 13 23 11 41 92 65 ca 0e 3e 38 90 e6 bf 7f 99 39 62 53 1e 2f e5 3b 1c d3 21 23 00 3a 
aa 63 f2 a7 f7 0a 78 33 b3 fe 0c 20 bc 28 c3 27 79 2a 3b 6c bc cf 40 20 ed ec 2c 4d e6 4c ca 48 
e2 72 83 68 15 a0 41 f2 77 9d a3 0a c0 32 0c 32 03 80 cf c8 5a 40 5f b3 9d 84 9a 41 99 2b b0 e8 
87 f8 fb 22 c6 1c 33 44 f6 5b 2c c7 5f 37 0d 6a 7b 4c de ac 8e 30 25 44 0e d9 ff 57 55 9b f1 e0 
59 83 ba 0a af 6e 74 61 65 dd 4c cd 27 b2 ff b2 04 cc f0 3b 36 26 ec c5 fd 99 75 72 20 77 0d 97 
4c 3f ea e1 84 10 d5 cd be 72 08 5b 18 b7 14 a7 cf ae 0e 11 65 94 4b 1c ef e0 3b 72 6d ff 30 86 
17 5e 64 e8 23 e5 94 d9 18 13 6e b9 aa 49 96 f3 94 1f 4e 12 c3 85 a6 c9 76 91 27 01 5a 46 2e 47 
91 69 59 40 e5 16 b0 5a 62 dd 0d 56 a5 82 aa 5e e4 c8 2f 59 d2 aa 56 22 ba 56 d0 f5 c4 d2 b6 ce 
48 c6 c2 f0 65 b8 17 1e 4b 1c 9b 68 36 23 4b 91 27 88 09 6e c1 17 b2 45 b6 9d e4 3a 53 4b 04 e4 
28 5d 3d c6 12 91 6e aa cd d8 c8 4a ea fd ae 9a cf 07 aa c7 cc 8b fa 38 4d 3d a4 2b ec 6c cf 97 
Once again, there is no difference between the expected data and the rxbuf when they are dumped.
Thanks for simplifying the example case. This definitely looks like a hardware issue related to an in-progress transfer.

Have you checked if the mismatched data matches the previous contents of RAM and whether after the first mismatch the rest of the data is right or wrong? Is there a fence missing somewhere?

I'm sorry I don't have much to add to this conversation. Please take my post as an encouragement to continue debugging. My post here also serves as an easy way for me to stay updated on developments in this thread. Sorry for the noise.

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1033
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Context switching breaks SPI/DMA

Mon Mar 15, 2021 11:33 pm

hmm.. i have one thought; does the problem go away if you build with

Code: Select all

pico_set_divider_implementation(TARGET compiler)
in your CMakeLists.txt?

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Tue Mar 16, 2021 3:20 pm

kilograham wrote:
Mon Mar 15, 2021 11:33 pm
hmm.. i have one thought; does the problem go away if you build with

Code: Select all

pico_set_divider_implementation(TARGET compiler)
in your CMakeLists.txt?
Yes, indeed, the problem goes away!

Now to try this in my real application.

cleverca22
Posts: 4925
Joined: Sat Aug 18, 2012 2:33 pm

Re: Context switching breaks SPI/DMA

Tue Mar 16, 2021 3:25 pm

i think the problem, is that your context switch algo isnt saving/restoring the state of the hardware divider

kilogram's bit of code, switches you to an entirely software driven divider, which is slower, but uses regular arm regs

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Tue Mar 16, 2021 6:42 pm

cleverca22 wrote:
Tue Mar 16, 2021 3:25 pm
i think the problem, is that your context switch algo isnt saving/restoring the state of the hardware divider

kilogram's bit of code, switches you to an entirely software driven divider, which is slower, but uses regular arm regs
Makes sense. So, I need to stick hw_divider_save_state() and hw_divider_restore_state() somewhere into xPortPendSVHandler()?(FreeRTOSConfig.h does `#define xPortPendSVHandler isr_pendsv`).

carlk3
Posts: 82
Joined: Wed Feb 17, 2021 8:46 pm

Re: Context switching breaks SPI/DMA

Tue Mar 16, 2021 6:45 pm

carlk3 wrote:
Tue Mar 16, 2021 3:20 pm
...
Now to try this in my real application.
Works there, as well.

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1033
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Context switching breaks SPI/DMA

Tue Mar 16, 2021 6:59 pm

carlk3 wrote:
Tue Mar 16, 2021 6:42 pm
cleverca22 wrote:
Tue Mar 16, 2021 3:25 pm
i think the problem, is that your context switch algo isnt saving/restoring the state of the hardware divider

kilogram's bit of code, switches you to an entirely software driven divider, which is slower, but uses regular arm regs
Makes sense. So, I need to stick hw_divider_save_state() and hw_divider_restore_state() somewhere into xPortPendSVHandler()?(FreeRTOSConfig.h does `#define xPortPendSVHandler isr_pendsv`).
Yes something needs to be done for sure, and this would work.

What we do in general (when using the hw divider) is to always read the results in a defined order. I think this is mentioned in both the data sheet and the C/C++ book. IRQ save/restore of divider state takes advantage of this to only save/restore if the interrupted task is "mid division", i.e. it hasn't read the results of the last divide (according to the fact that the dirty flag on the hw divider is cleared when you read the quoitent). you can look (though it isn't well commented) at pico_divider/divider.S

Return to “SDK”