beedaddy
Posts: 14
Joined: Fri Mar 04, 2016 11:13 am

[Solved] Weird problem with C++ and strings - fragmented ram?

Sat Feb 24, 2024 8:29 pm

I noticed a weird behaviour programming the Pico with C++. I'm doing some string operations and concatenations. And everything is fine with the equivelent test program on the PC. But on the Pico weird things happen. Either, strings are empty, or the contain things they shouldn't (can't!) contain. It took me a while to realize that the problem might be due to a memory problem, perhaps a fragmented RAM? I'm not sure about that, but if I do the operations on a very short string, everything is ok, even on the Pico. But after extending it a little bit, it goes crazy again.

So the question is, is it likely that the reason is fragmented memory? I think of reallocations when for example chars are added to a string. I'm not very experienced with microcontrollers, perhaps this is a well known issue? Is it perhaps adviseable or necessary to avoid using std::string in embedded programming? Perhaps someone could tell me whether my assumption of fragmented memory is reasonable, or how I could tackle it down, or even what could I do against it. Working with C-strings only?

By the way, this is the part of program which works flawlessly on my PC machine. But integrating it on the Pico results in the problems described above:

Code: Select all

#include <algorithm>
#include <iostream>
#include <map>
#include <regex>
#include <string>

std::map<char, std::string> morseCode = {
    {'A', ".-"},     {'B', "-..."},   {'C', "-.-."},
    {'D', "-.."},    {'E', "."},      {'F', "..-."},
    {'G', "--."},    {'H', "...."},   {'I', ".."},
    {'J', ".---"},   {'K', "-.-"},    {'L', ".-.."},
    {'M', "--"},     {'N', "-."},     {'O', "---"},
    {'P', ".--."},   {'Q', "--.-"},   {'R', ".-."},
    {'S', "..."},    {'T', "-"},      {'U', "..-"},
    {'V', "...-"},   {'W', ".--"},    {'X', "-..-"},
    {'Y', "-.--"},   {'Z', "--.."},   {'1', ".----"},
    {'2', "..---"},  {'3', "...--"},  {'4', "....-"},
    {'5', "....."},  {'6', "-...."},  {'7', "--..."},
    {'8', "---.."},  {'9', "----."},  {'0', "-----"},
    {',', "--..--"}, {'?', "..--.."}, {'/', "-..-.-"},
    {'=', "-...-"},  // '=' == <BT>
    {'k', "-.--."},  // k == <KN>
    {'s', "...-.-"}, // s == <SK>
    {'+', ".-.-."},  // + == <AR>
    {'a', "-.-.-"},  // a == <KA>
};

void messageToMorse(const std::string &msg) {
  std::string msgUpper;
  std::string morse;
  msgUpper.resize(msg.length());

  std::transform(msg.cbegin(), msg.cend(), msgUpper.begin(),
                 [](unsigned char c) { return std::toupper(c); });

  // Encode the special characters as we like it
  msgUpper = std::regex_replace(msgUpper, std::regex("<BT>"), "=");
  msgUpper = std::regex_replace(msgUpper, std::regex("<KN>"), "k");
  msgUpper = std::regex_replace(msgUpper, std::regex("<SK>"), "s");
  msgUpper = std::regex_replace(msgUpper, std::regex("<AR>"), "+");
  msgUpper = std::regex_replace(msgUpper, std::regex("<KA>"), "a");

  // Remove all other unknown characters
  msgUpper.erase(remove_if(msgUpper.begin(), msgUpper.end(),
                           [](const char &c) {
                             return c != ' ' &&
                                    morseCode.find(c) == morseCode.end();
                           }),
                 msgUpper.end());

  // Remove spaces, if there are too many of them
  msgUpper = std::regex_replace(msgUpper, std::regex("(\\s+)"), " ");

  for (int i = 0; i < msgUpper.length(); i++) {
    auto c = msgUpper[i];
    if (c == ' ') {
      morse += 'w';
      continue;
    }

    // Ignore and continue with next char, if not found
    auto search = morseCode.find(c);
    if (search == morseCode.end()) {
      continue;
    }

    for (int j = 0; j < morseCode[c].length(); j++) {
      auto m = morseCode[c][j];
      if (j == 0 && i > 0 && msgUpper[i - 1] != ' ') {
        morse += 'c';
      }
      
      morse += m;
      
      if (j < morseCode[c].length() - 1) {
        morse += 'i';
      }
    }
  }

  // Append word space if last char was not a blank
  if(msgUpper.back() != ' ') {
      morse += 'w';
  }
  
  std::cout << morse << std::endl;
}

int main() {
  std::string message{"cq cq de ab1cde ab1cde pse k"};

  messageToMorse(message);
}
Last edited by beedaddy on Sun Feb 25, 2024 8:45 pm, edited 1 time in total.

alastairpatrick
Posts: 753
Joined: Fri Apr 22, 2022 1:39 am
Location: USA

Re: Weird problem with C++ and strings - fragmented ram?

Sat Feb 24, 2024 8:45 pm

It will fail for a sufficiently long string for sure because of either fragmentation or simply not having enough memory. I doubt it would fail for the string "cq cq de ab1cde ab1cde pse k" though.

On microcontrollers, it's often preferable to use a memory allocation strategy where either fragmentation doesn't happen (e.g. static allocation) or where fragmentation doesn't matter (e.g. freelists). I think 256KB is enough for dynamic allocation to not be beyond hope but I would be careful.

If you think it's memory allocation failure related, perhaps you can arrange for the LED to turn on when that happens. Perhaps by modifying pico_malloc.c in the SDK, for example here:

Code: Select all

static inline void check_alloc(__unused void *mem, __unused uint size) {
#if PICO_MALLOC_PANIC
    if (!mem || (((char *)mem) + size) > &__StackLimit) {
        panic("Out of memory");
    }
#endif
}
I haven't tried it though. I would also send it into an infinite loop after turning the LED on, to ensure it doesn't subsequently reset and turn the LED off. Keep in mind that C++ allows the compiler to optimize away this loop to nothing:

Code: Select all

// Not necessarily an infinite loop!
for(;;) {
}

beedaddy
Posts: 14
Joined: Fri Mar 04, 2016 11:13 am

Re: Weird problem with C++ and strings - fragmented ram?

Sat Feb 24, 2024 8:58 pm

Thanks for your reply. On the pico, the code is integrated in other code, so perhaps it might be indeed out of memory. The whole code is here, by the way. I will see if I can follow your advice with the led. Or are there any other helpful functions which could show me the still available ram (I know, it could be fragmented anyway).
Last edited by beedaddy on Wed Feb 28, 2024 11:17 am, edited 1 time in total.

User avatar
AndyD
Posts: 2468
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia

Re: Weird problem with C++ and strings - fragmented ram?

Sat Feb 24, 2024 9:15 pm

The reason that smaller strings work and longer strings cause a problem, is probably due to the C++ Short String Optimisation.

The SSO allows smaller strings to put in to a fixed size buffer in the std::string class. Strings that don't fit into the buffer use a dynamically allocated buffer.

rvt
Posts: 14
Joined: Tue Aug 22, 2023 7:17 pm

Re: Weird problem with C++ and strings - fragmented ram?

Sat Feb 24, 2024 10:04 pm

Might not be a solution to your problem, but I have been using this to avoid std on microcomtrollers: https://www.etlcpp.com/string.html

dthacher
Posts: 1025
Joined: Sun Jun 06, 2021 12:07 am

Re: Weird problem with C++ and strings - fragmented ram?

Sat Feb 24, 2024 11:50 pm

List of things which can simulate fragmentation: (not complete)
  • Caches
  • Memory accesses in hardware
  • Data structures
  • Dynamic memory
Using iostream is not recommended. I would also avoid using std::map. I am not sure I see the point of std::string here. You are using ascii, so do this:

Code: Select all

char *map[256];
const char *table[] = { "...", "-." };

void build_map() {
	// TODO: Associate table to map.
	//	Assign null pointers to unused characters
}

void random_function() {
	if (map['a'] != 0)
		printf("%s\n", map['a']);
}

carlk3
Posts: 210
Joined: Wed Feb 17, 2021 8:46 pm

Re: Weird problem with C++ and strings - fragmented ram?

Sun Feb 25, 2024 1:07 am

It might be worth putting something like

Code: Select all

target_compile_definitions(${PROGRAM_NAME} PRIVATE
    PARAM_ASSERTIONS_ENABLE_ALL=1
    PICO_USE_STACK_GUARDS=1
    PICO_MALLOC_PANIC=1
)
into your CMakeLists.txt.

User avatar
jahboater
Posts: 8893
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Weird problem with C++ and strings - fragmented ram?

Sun Feb 25, 2024 8:32 am

dthacher wrote:
Sat Feb 24, 2024 11:50 pm
You are using ascii, so do this:

Code: Select all

const char *table[] = { "...", "-." };
There is an 8 byte overhead for each entry here.
If you know the longest string length and it is fairly small, as here, this might save some memory (and be much faster):

Code: Select all

static constexpr char table[][4] = { "...", "-." };
Also, if its in a function, declaring the array static means code doesn't have to included to assemble or load the array every single time.
If its not in a function, then static just saves some space in the symbol table.

beedaddy
Posts: 14
Joined: Fri Mar 04, 2016 11:13 am

Re: Weird problem with C++ and strings - fragmented ram?

Sun Feb 25, 2024 12:44 pm

carlk3 wrote:
Sun Feb 25, 2024 1:07 am
It might be worth putting something like

Code: Select all

target_compile_definitions(${PROGRAM_NAME} PRIVATE
    PARAM_ASSERTIONS_ENABLE_ALL=1
    PICO_USE_STACK_GUARDS=1
    PICO_MALLOC_PANIC=1
)
into your CMakeLists.txt.
That was a promising hint. But unfortunately, this did not result in any changes or error messages on minicom.

beedaddy
Posts: 14
Joined: Fri Mar 04, 2016 11:13 am

Re: Weird problem with C++ and strings - fragmented ram?

Sun Feb 25, 2024 12:51 pm

dthacher wrote:
Sat Feb 24, 2024 11:50 pm
Using iostream is not recommended.
Yes. I didn't point out that the program on the pico doesn't use iostream. This was just part of the reference code on my desktop pc.

dthacher
Posts: 1025
Joined: Sun Jun 06, 2021 12:07 am

Re: Weird problem with C++ and strings - fragmented ram?

Sun Feb 25, 2024 4:35 pm

There is an 8 byte overhead for each entry here.
If you know the longest string length and it is fairly small, as here, this might save some memory (and be much faster):
I could combine the two structures and yes this would save some memory. The idea here was to allow a look up table (map) in one structure. The table is likely all in ROM. (This structure is also smaller than the map.)

beedaddy
Posts: 14
Joined: Fri Mar 04, 2016 11:13 am

Re: Weird problem with C++ and strings - fragmented ram?

Sun Feb 25, 2024 8:45 pm

Thanks a lot for your suggestions. It turned out, that the problem was completely unrelated to any ram shortage or fragmentation. :oops:
It was a beginners fault, passing a string by reference, where by copy would have been wiser. :lol:

Return to “SDK”