Zheoni
Posts: 5
Joined: Sat May 08, 2021 3:47 pm

Can the default alarm pool stop for some reason?

Sat Sep 18, 2021 6:49 pm

Hello,

I'm trying to debug my program and after a couple of days of losing my head, I'm just asking for help. It's a big program, I have tried to reproduce the error in a smaller snippet but It does not occur.

I know that not having the program or any other code does not help, but maybe with the behaviour I describe below someone can think of a cause why this is happening.

The program communicates with my PC with stdio USB and receives commands.
I have added a repeating timer that makes the on-board LED blink.
When I run a specific command, the LED stops blinking, the Pico stops responding to commands, but everything else (other LEDs, a display, etc.) is still functioning.

When the blinking LED is on another alarm pool, no matter if it's in the same core or on core1, the communication stops but the LED continues blinking. As the USB communication task is also in the default alarm pool, that makes me think the default alarm pool for some reason has stopped calling the callbacks to the alarms.

Another interesting fact is that this does not happen when the program is compiled with pico_set_binary_type(program no_flash).

When this happens, the pico:
- Has a repeating timer in core1 reading data from a sensor and storing it in an SDK queue. (set up by another command)
- After receiving a command, core0 fills up a buffer with data from the queue and sends it through stdio USB.

Thanks for your attention.

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 949
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Can the default alarm pool stop for some reason?

Mon Sep 20, 2021 12:21 am

Stack overflow maybe?
I’m not aware of any alarm pool bugs

If the alarm is not being called then either something bad happened, or something equal or higher priority is stuck or looping (or another alarm callback in the same pool)

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 949
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Can the default alarm pool stop for some reason?

Mon Sep 20, 2021 12:24 am

Note you at the communication stops but the led blinks if another pool; it seems like u might want to understand why the communication is stopping

Zheoni
Posts: 5
Joined: Sat May 08, 2021 3:47 pm

Re: Can the default alarm pool stop for some reason?

Mon Sep 20, 2021 11:25 pm

Thanks for your response. I still don't know what is exactly happening... but I have managed to make it work modifying the SDK. I have created a new alarm pool only for the tinyusb task in stdio_usb_init. Doing this all seems to works.

This and your response made me thing the issue was that something was stuck looping in the default alarm pool and other callbacks were stuck.

Is there a way I could see what alarms are in the pool so I can know what is happening? I would prefer not to modify the SDK to make it work.

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 949
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Can the default alarm pool stop for some reason?

Tue Sep 21, 2021 12:01 am

Debug it over SWD is the best… I’d be curious to see the stack trace

Zheoni
Posts: 5
Joined: Sat May 08, 2021 3:47 pm

Re: Can the default alarm pool stop for some reason?

Tue Sep 21, 2021 4:56 pm

I'm setting up the SWD debug and I stumbled upon this.
When using SWD for debugging you need to use a UART based serial connection (see Chapter 4) as the USB stack will be paused when the RP2040 cores are stopped during debugging, which will cause any attached USB devices to disconnect. You cannot use a USB CDC serial connection during debugging.
(Page 22 of Getting Started with Raspberry Pi Pico)

So I'm thinking of changing the USB CDC serial connection with a UART, however, my program already uses another UART connection, so I'll need to modify the program... and if the problem was with the USB CDC task, maybe the problem does not appear.

TL;DR. I need some time to tinker with this, but I'll try my best to get that stack trace in a couple of days.

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 949
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Can the default alarm pool stop for some reason?

Tue Sep 21, 2021 9:47 pm

Zheoni wrote:
Tue Sep 21, 2021 4:56 pm
I'm setting up the SWD debug and I stumbled upon this.
When using SWD for debugging you need to use a UART based serial connection (see Chapter 4) as the USB stack will be paused when the RP2040 cores are stopped during debugging, which will cause any attached USB devices to disconnect. You cannot use a USB CDC serial connection during debugging.
(Page 22 of Getting Started with Raspberry Pi Pico)

So I'm thinking of changing the USB CDC serial connection with a UART, however, my program already uses another UART connection, so I'll need to modify the program... and if the problem was with the USB CDC task, maybe the problem does not appear.

TL;DR. I need some time to tinker with this, but I'll try my best to get that stack trace in a couple of days.
well you can run with USB CDC under the debugger, and pause (and look at stack trace when it hangs). It's just that when you are paused in the debugger, TinyUSB will stop responding to packets from the host. I'm pretty sure the probably will go away without USB anyway.

Zheoni
Posts: 5
Joined: Sat May 08, 2021 3:47 pm

Re: Can the default alarm pool stop for some reason?

Thu Sep 23, 2021 2:48 am

I finally made the debugger work! And I have interesting news... or problems. When I compile the program with -DCMAKE_BUILD_TYPE=Debug... it doesn't hang up. So I need to just read the assembly.

So it's only with a regular optimised build and being on the flash when it gets stuck. If it's on RAM or compiled with debug, it works.

I can now confirm 100% that when the communication stops, neither the blink callback nor the tinyusb task are being called. As breakpoints in those functions are never reached once the bug happens, but before they are.

Maybe more worrying, alarm_pool_alarm_callback does not seem to be called in core0, a breakpoint on that function always fires in core1, where there is a repeating timer with its own pool.

As for the call stack, it looks like core0 is looping in my waiting loop calling getchar_timeout_us(). That loop is:

Code: Select all

bool clear = false;
int command;
while ((command = getchar_timeout_us(0)) == PICO_ERROR_TIMEOUT) {
        if (!clear && time_reached(clear_timeout)) {
            // some gpio stuff
            clear = true;
        }
}
Every time I pause execution I'm in a different part of calling getchars_timeout_us(0), most of the time in busy_wait_us, but stdio_usb_in_chars is eventually called.

But now... I don't know how to use the debugger to get the source of the error... as I'm not sure what I'm looking for.

This is an image of the call stack in a random moment that I manually paused, as the active breakpoints are not reached.
Image

Zheoni
Posts: 5
Joined: Sat May 08, 2021 3:47 pm

Re: Can the default alarm pool stop for some reason?

Thu Oct 14, 2021 5:10 pm

My final solution to this has been to use an UART connection to my PC with a CP2102 USB TTL.

I don't know what was exactly the problem, but this was the only thing I changed of my entire program and all works perfectly. So this leads me to think that the problem has to be in TinyUSB.

Return to “SDK”