User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Sun Feb 11, 2024 12:37 am

Hi,

I could really use some help here.

I have a C++ progrm that uses fgets to read messages (commands) out of a fifo. These commands tell the C++ program to do things with the GPIO pins (and potentially other hardware interfaces).

I'm no expert in C++, so it's possible I have done something wrong, but it does look to me as if there is a bug in the C++ implementation of fgets, at least under Bookworm on a Pi 5. I will be testing other environments, but that takes time.

There seem to be two bugs:

1.
On the first call to fgets it apparently works correctly, waiting untill there is data in the pipe, reading it upto and including the first \n character,, and flushing what it has read from the fifo. But thereafter it no longer waits for data to be present, but returns immediately from each call with rubbish in the input buffer (always 6 characters long) but no error on the read, maxing out one CPU core in a tight loop.

I think the correct behaviour is called "blocking mode). If so, the issue is that a sucessful call to fgets sets the mode to not_blocking

I have managed to fix this by opening the fifo (fopen) immediately before the call to fgets, and closing it (fclose) immediately after the fgets call returns. So it's only ever on the "first call" to fgets, which works. But this is clearly a bodge.

2.
On further testing it appears that even the first call to fgets is not quite working correctly. I does not flush what it has read from the fifo: instad it flushes the entire contents of the fifo. Normally you would not notice the difference as the commands are quite short and you typically put them into the fifo one at a time, so what fgets normall reads IS the entire fifo contents; but in general (and during testing) it should only flush what it has read (as I understand it.


This is part of a larger program: I'm in the process of cutting it down to check that this is not a side effect of something apparently unrealated. I'm also going to run it on Bullseye on a Pi 4 to check the behaviour is the same.

Once I've got this down to something managable I'll post the code , but the guts of it is:

Code: Select all

      std::stringstream msgsst;          // stringstream to accumulate the input message
      char bufl;                                       // last character in current fgets read buffer
      do{                                                 // loop til you get a \n
            char msgbuf[msglen];          // fgets read buffer
            int len;                                     // number of characters read
            fd = fopen(fifoname, "r");     // open the fifo
            fgets(msgbuf, msglen, fd);  //read til \n, buffer full or EOF
            fclose(fd);                               // close the fifo
            len = strlen(msgbuf);            // find number of characters read
            bufl = msgbuf[len-1];             // find the last character
            msgsst << msgbuf;               // build up to complete input stringstream                
         } while (bufl != '\n');                   // keep going til you have a \n
Which looks pretty unexceptional to me...
Last edited by Peter Cyriax on Sun Feb 11, 2024 8:44 am, edited 1 time in total.

User avatar
AndyD
Posts: 2456
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia

Re: C++ fgets behaving very oddly on Bookworm

Sun Feb 11, 2024 2:05 am

I suspect the issue is that you are not checking the return values of fopen or fgets. If fopen fails it will return NULL and if fgets fails it will also return NULL. There is no guarantee that there the msgbuf will contain anything sensible if fgets fails.

If I understand what you are trying to do, I would read from the FIFO using a std::filestream and read the lines using std::getline

And rather than putting the data into std::stringstream, consider using a container like std::vector or std:deque

I suspect there is nothing wrong with the implementation of fgets, there would be a lot of issues if there was.

ejolson
Posts: 12155
Joined: Tue Mar 18, 2014 11:47 am

Re: C++ fgets behaving very oddly on Bookworm

Sun Feb 11, 2024 2:34 am

Peter Cyriax wrote:
Sun Feb 11, 2024 12:37 am
Hi,

I could really use some help here.

I have a C++ progrm that uses fgets to read messages (commands) out of a fifo. These commands tell the C++ program to do things with the GPIO pins (and potentially other hardware interfaces).

I'm no expert in C++, so it's possible I have done something wrong, but it does look to me as if there is a bug in the C++ implementation of fgets, at least under Bookworm on a Pi 5. I will be testing other environments, but that takes time.

There seem to be two bugs:

1.
On the first call to fgets it apparently works correctly, waiting untill there is data in the pipe, reading it upto and including the first \n character,, and flushing what it has read from the fifo. But thereafter it no longer waits for data to be present, but returns immediately from each call with rubbish in the input buffer (always 6 characters long) but no error on the read, maxing out one CPU core in a tight loop.

I think the correct behaviour is called "blocking mode). If so, the issue is that a sucessful call to fgets sets the mode to not_blocking

I have managed to fix this by opening the fifo (fopen) immediately before the call to fgets, and closing it (fclose) immediately after the fgets call returns. So it's only ever on the "first call" to fgets, which works. But this is clearly a bodge.

2.
On further testing it appears that even the first call to fgets is not quite working correctly. I does not flush what it has read from the fifo: instad it flushes the entire contents of the fifo. Normally you would not notice the difference as the commands are quite short and you typically put them into the fifo one at a time, so what fgets normall reads IS the entire fifo contents; but in general (and during testing) it should only flush what it has read (as I understand it.


This is part of a larger program: I'm in the process of cutting it down to check that this is not a side effect of something apparently unrealated. I'm also going to run it on Bullseye on a Pi 4 to check the behaviour is the same.

Once I've got this down to something managable I'll post the code , but the guts of it is:

Code: Select all

      std::stringstream msgsst;          // stringstream to accumulate the input message
      char bufl;                                       // last character in current fgets read buffer
      do{                                                 // loop til you get a \n
            char msgbuf[msglen];          // fgets read buffer
            int len;                                     // number of characters read
            fd = fopen(fifoname, "r");     // open the fifo
            fgets(msgbuf, msglen, fd);  //read til \n, buffer full or EOF
            fclose(fd);                               // close the fifo
            len = strlen(msgbuf);            // find number of characters read
            bufl = msgbuf[len-1];             // find the last character
            msgsst << msgbuf;               // build up to complete input stringstream                
         } while (bufl != '\n');                   // keep going til you have a \n
Which looks pretty unexceptional to me...
I guess the program sending the commands closes the FIFO at the other end and after that the receiver returns an error with the rubbish. Opening the FIFO again causes the receiver to wait until the sender opens the other end and sends another message, but then the sender closes it and subsequent reads again fail. In conclusion either the receiver has to keep opening the FIFO or the sender has to stop closing it.

Maybe you want a local socket instead.

BigRedMailbox
Posts: 515
Joined: Sat Aug 20, 2022 10:37 pm

Re: C++ fgets behaving very oddly on Bookworm

Sun Feb 11, 2024 3:52 am

Any post that contains (either in the Subject or the body) the words:
C++ fgets behaving very oddly on Bookworm
Must also contain the phrase:
But it works fine in <otherVersion>
If it doesn't, then the phrase "on Bookworm" should be dropped.


(If you see what I mean...)
My posts may be "controversial" and/or out-of-sync with the party line.

Nothing I write should in any way be taken as an official statement by any organization connected with (any branch of) RPi and/or any of its funding sources.

User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

Re: C++ fgets behaving very oddly on Bookworm

Sun Feb 11, 2024 8:39 am

Thanks for al the replies:

1. The behaviour occurrs under Bookworm, and I do say that I will test other environments: I will replace the Pi5 with a Pi 4 in order to do that. The subject was intended to convey information, not criticism: I have edited the subject to emphasize this.

2. I have used fgets on a pi before, but embedd in an APL program, not C++, under Buster, and did not see this behaviour. I will, as I said run the C++ on older systems to establish the scope of this behaviour

3. I am testing this by sending input jessages from a terminal using fprintf. So yes... that almost certainly does open and close the fifo each time it is used, which would not be the case in the production system. However I I have tried putting multiple messages (each terminated by \n) in a single call from printf, and fgets reads (and correctly actions) only the first message. If I then try to cat or tail the fifo from another terminal (before or after terminating the C++ program) the messages that fgets has not read are not there, so it does look as if fgets flushes the whole fifo, not just the bit it has read, but I'll re-run these tests to check and re-post.

4. I am using fgets rather than getline because getline discards the terminating \n which means the C++ program has no way to tell if it has in fact received a complete message.

5. I have tried checking for errors on the fgets call:

Code: Select all

     if (fgets(msgbuf, msglen, fd) == NULL){
          printf("read error : %d\n", ferror(fd));
      }
and no errors are reported

6. Just to be clear, yes... I am trying to write a C++ program that, in effect, tails the fifo, receiving \n terminated messages one at a time, which it then actions. The program works absolutely fine provided that mesaages are written to the fifo one at a time, even in bits using multiple writes, and are less than 80 characters long. These assumptions are reasonable, but I don't want to leave them embedded in a production system that should run, unsupported, in multiple environments, for many years.

7. I have taught myself what little I know about C++ starting a week ago by googling and reading helpful forums like this one. I am putting the message into a stringstream because they are defined as space delimited \n terminated strings, and I found I could then easily parse the message using getline with ' ' as the delimiter to convert the message into a vector of words. I'll read up about sockets and these other containers, but it's actually reading the fifo that is the only issue I currently have.

8. If I could find the source code of tail ...

Thanks...

Peter

User avatar
AndyD
Posts: 2456
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia

Re: C++ fgets behaving very oddly on Bookworm

Sun Feb 11, 2024 9:37 am

Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
8. If I could find the source code of tail ...
https://github.com/coreutils/coreutils/ ... src/tail.c

User avatar
AndyD
Posts: 2456
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia

Re: C++ fgets behaving very oddly on Bookworm

Sun Feb 11, 2024 10:36 am

Hi Peter,

I really do think that the problem is not in the implementation of fgets, but your expectation of the behaviour of your program.

The fgets function is provided by the glibc library and is the same function independent of you using C or C++. It is a very commonly used function and any regression would be found quickly in Linux.
Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
1. The behaviour occurrs under Bookworm, and I do say that I will test other environments: I will replace the Pi5 with a Pi 4 in order to do that. The subject was intended to convey information, not criticism: I have edited the subject to emphasize this.
Understood, but without a fully working example of the issue, it is very difficult to work out if this is an actual problem in fgets or a misunderstanding of the expected behaviour. If you could provide a minimal working example that would be a good start.
Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
2. I have used fgets on a pi before, but embedd in an APL program, not C++, under Buster, and did not see this behaviour. I will, as I said run the C++ on older systems to establish the scope of this behaviour
That is a good idea.
Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
3. I am testing this by sending input jessages from a terminal using fprintf. So yes... that almost certainly does open and close the fifo each time it is used, which would not be the case in the production system. However I I have tried putting multiple messages (each terminated by \n) in a single call from printf, and fgets reads (and correctly actions) only the first message. If I then try to cat or tail the fifo from another terminal (before or after terminating the C++ program) the messages that fgets has not read are not there, so it does look as if fgets flushes the whole fifo, not just the bit it has read, but I'll re-run these tests to check and re-post.
So the lines are consumed by your program, but don't appear in your output. That is a possible clue.
Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
4. I am using fgets rather than getline because getline discards the terminating \n which means the C++ program has no way to tell if it has in fact received a complete message.
I disagree, but that is OK. The std::getline function will only read complete lines terminated by a newline, unless you have a line that is larger than the maxim string size, which is a little less than 5 GB (4,294,967,295 bytes) on my Raspberry Pi 5 8GB.
Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
5. I have tried checking for errors on the fgets call:

Code: Select all

     if (fgets(msgbuf, msglen, fd) == NULL){
          printf("read error : %d\n", ferror(fd));
      }
and no errors are reported
That is good, what about fopen?
Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
6. Just to be clear, yes... I am trying to write a C++ program that, in effect, tails the fifo, receiving \n terminated messages one at a time, which it then actions. The program works absolutely fine provided that mesaages are written to the fifo one at a time, even in bits using multiple writes, and are less than 80 characters long. These assumptions are reasonable, but I don't want to leave them embedded in a production system that should run, unsupported, in multiple environments, for many years.
There are excellent aims. As I say above, it is highly unlikely that there is a bug in fgets, and more likely that your expectations of the programs behaviour are not complete.
Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
7. I have taught myself what little I know about C++ starting a week ago by googling and reading helpful forums like this one. I am putting the message into a stringstream because they are defined as space delimited \n terminated strings, and I found I could then easily parse the message using getline with ' ' as the delimiter to convert the message into a vector of words. I'll read up about sockets and these other containers, but it's actually reading the fifo that is the only issue I currently have.
You already have each command in a buffer. It would be better to assign the buffer to a std::string and use C++ functions to remove the newline.
Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
8. If I could find the source code of tail ...
I provided a link above to the source for the coreutils implementation of tail.

ejolson
Posts: 12155
Joined: Tue Mar 18, 2014 11:47 am

Re: C++ fgets behaving very oddly on Bookworm

Sun Feb 11, 2024 10:43 am

Peter Cyriax wrote:
Sun Feb 11, 2024 8:39 am
8. If I could find the source code of tail ...
I think the command tail -f will exit if reading from a FIFO and the other end is closed.

User avatar
AndyD
Posts: 2456
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Sun Feb 11, 2024 11:01 am

Peter Cyriax wrote:
Sun Feb 11, 2024 12:37 am

Code: Select all

      std::stringstream msgsst;          // stringstream to accumulate the input message
      char bufl;                                       // last character in current fgets read buffer
      do{                                                 // loop til you get a \n
            char msgbuf[msglen];          // fgets read buffer
            int len;                                     // number of characters read
            fd = fopen(fifoname, "r");     // open the fifo
            fgets(msgbuf, msglen, fd);  //read til \n, buffer full or EOF
            fclose(fd);                               // close the fifo
            len = strlen(msgbuf);            // find number of characters read
            bufl = msgbuf[len-1];             // find the last character
            msgsst << msgbuf;               // build up to complete input stringstream                
         } while (bufl != '\n');                   // keep going til you have a \n
Your code exits the do loop after reading the first line. It will read the first line and discard the rest because you are closing the file. The fopen function is buffered I/O and will read all the lines in and discard them when fclose is called.

I tried the following pair of programs on my Pi 5, running Bookworm.

C code "server" side program.

Code: Select all

#include <stdio.h>

int main()
{
    FILE *fp = fopen("/tmp/my_named_pipe", "w");

    if (fp)
    {
        fprintf(fp, "one\ntwo\nthree\nfour\nfive\nsix\nseven\neight\nnine\nten\n");
    }

    fclose(fp);
    return 0;
}
C++ "client"

Code: Select all

#include <cstdio>
#include <cstring>
#include <iostream>

int main()
{
    FILE *fp = fopen("/tmp/my_named_pipe", "r");

    if (fp)
    {
        char msgbuf[1024];
        while (fgets(msgbuf, sizeof(msgbuf), fp))
        {
            printf("Received %s", msgbuf);
        }
        fclose(fp);
    }

    return 0;
}
Output of client.cxx is

Code: Select all

Received one
Received two
Received three
Received four
Received five
Received six
Received seven
Received eight
Received nine
Received ten

User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Sun Feb 11, 2024 12:02 pm

Hi AndyD,

Thank you.

You have explained the reason why the whole fifo is getting flushed. I will revert the program to its original design, so that it keeps the fifo open all the time, and see if this behaviour vanishes.

However I opened the and closed the fifo on each read to sove the first problem, which was:
On the first call to fgets it apparently works correctly, waiting untill there is data in the pipe, reading it upto and including the first \n character,, and flushing what it has read from the fifo. But thereafter it no longer waits for data to be present, but returns immediately from each call with rubbish in the input buffer (always 6 characters long) but no error on the read, maxing out one CPU core in a tight loop.

I think the correct behaviour is called "blocking mode). If so, the issue is that a sucessful call to fgets sets the mode to not_blocking

I have managed to fix this by opening the fifo (fopen) immediately before the call to fgets, and closing it (fclose) immediately after the fgets call returns. So it's only ever on the "first call" to fgets, which works. But this is clearly a bodge.

I will try using std::getline. I had not appreciated that std::getline ONLY terminates when it encounters a \n.


Actually... I agree that a bug in fgets does seem unlikely. There will be all manner of issues. That is why I spent several days testing this and trying to figure it out before posting... and the subject:
fgets behaving very oddly
was intended to convey that the behaviour is odd (which it is) without laying blame.

Just to get the terminology right can anybody confirm or correct this, please:
I think the correct behaviour is called "blocking mode). If so, the issue is that a sucessful call to fgets sets the mode to not_blocking

Peter

User avatar
AndyD
Posts: 2456
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Sun Feb 11, 2024 12:44 pm

I assume you are writing to the FIFO multiple times, by running a program more than once. Your code should keep opening the FIFO and then reading until it hits the End of File. The easiest way to do this is to loop as follows:-

Code: Select all

#include <cstdio>
#include <cstring>
#include <iostream>

int main()
{
    while (true)
    {
        FILE *fp = fopen("/tmp/my_named_pipe", "r");

        if (fp)
        {
            char msgbuf[1024];
            while (fgets(msgbuf, sizeof(msgbuf), fp))
            {
                printf("Received %s", msgbuf);
            }
            fclose(fp);
        }
    }

    return 0;
}
Sorry, I mixed up my client and server in above. The server should be reading from the FIFO and the client should be writing to it.

So each time the FIFO is opened for writing by the "client", the "server" will open the FIFO for reading. The "server" will keep reading from the FIFO until the "client" closes its writing handle. The fgets will then fail, by returning NULL and you should close the FILE handle and try again.

Hope this helps!

User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Sun Feb 11, 2024 6:20 pm

Hi AndyD,


Again... thanks.

This is all embedded in a yacht racing system which uses websockets, where the server is naturally the writer and the client the reader, not the other way round... Can we stick to "reader" and "writer" please :?: I did not find what you posted confusing at all... the file read and write calls define which program is doing what.

I'm currently feeding data into the FIFO from a terminal session using printf. I'm sure this must open and close the fifo for each write. In the production system the writer would typically keep the FIFO open until the entire system is switched off ; but I could change it to open and close the FIFO each time it has a message to send if that helps any. :?:

You say:
So each time the FIFO is opened for writing by the "client", the "server" will open the FIFO for reading. The "server" will keep reading from the FIFO until the "client" closes its writing handle. The fgets will then fail, by returning NULL and you should close the FILE handle and try again.
This is very helpful but a shade perplexing.

May I please first rewrite this as:
So each time the writer opens the FIFO, the reader will open it for reading, and will keep reading from the FIFO until the writer closes it for writing. The fgets will then fail, returning NULL; and you should close the FILE handle and try again
.

My reading of the documentation is that NULL is returned on the call to fgets if the FIFO is empty, whether or not the writer has closed the FIFO, and this is not classed as an error. Maybe that's just semantics, because what matters is that fgets returns a NULL. Which leads us to this:
So each time the writer opens the FIFO, the reader will open it for reading, and will keep reading from the FIFO until fgets returns a NULL. At this point the reader should close the FIFO and try again.
Again, appologies if I have misquoted you: I'm simply trying to understand.

But... the reader has no way to know when the writer is going to open the FIFO, or (if the writer keeps it open all the time) write to it. The reader can only try again immediately, and wait. So, the reader calls fgets, and two different things can then happen:
Scenario 1. fgets executes immediately, returning a NULL to indicate EOF (or FIFO empty): exactly what we DON'T want - it leads to a tight loop maxing out a CPU core.
Scenario 2. fgets waits in suspended animation until the OS tells it that there is data in the FIFO., whereupon it proceeds. :)

In the 2nd, desireable scenario, what the reader has done is to close the FIFO on receipt of the NULL, immediately re-open it and issue a new fgets call, which obligingly wats until there is data in the FIFO.

I will code it based on that, and see what happens.

But this does beg the question of why it is necessary for the reader to close the FIFO and immediately re-open it. However, my tests suggest that if the reader does not do the "close+open" bit, fgets will then move to Scenario 1.

Seems perverse to me, but if that's what fgets is meant to do, and the "close+open" solves my problem... well... THANKS


Peter

User avatar
AndyD
Posts: 2456
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Sun Feb 11, 2024 8:35 pm

Peter Cyriax wrote:
Sun Feb 11, 2024 6:20 pm
My reading of the documentation is that NULL is returned on the call to fgets if the FIFO is empty, whether or not the writer has closed the FIFO, and this is not classed as an error. Maybe that's just semantics, because what matters is that fgets returns a NULL. Which leads us to this:
So each time the writer opens the FIFO, the reader will open it for reading, and will keep reading from the FIFO until fgets returns a NULL. At this point the reader should close the FIFO and try again.
I would be interested to know where you read that. I don't believe this to be the case. A simple experiment will show this.

If you run the following "reader" code in one terminal

Code: Select all

#include <cstdio>
#include <cstring>
#include <iostream>

int main()
{
    FILE *fp = fopen("/tmp/my_named_pipe", "r");

    if (fp)
    {
        char msgbuf[1024];
        while (fgets(msgbuf, sizeof(msgbuf), fp))
        {
            printf("Received %s", msgbuf);
        }
        fclose(fp);
    }

    return 0;
}
And replace the "writer" with the command

Code: Select all

# cat > /tmp/my_named_pipe
and start typing lines of text, you will see that fgets will block until a new line of text is entered. The "reader" will only exit once the cat command exits. You can either kill it using ctrl-c or send the eof marked using ctrl-d.

The "reader" code above will block in fopen until there is a "writer" on the FIFO, and fgets will block waiting for input from the FIFO, and will return NULL once the "writer" has exited. That is why there is a need for a loop and the fclose and then fopen. You want to block in fopen until there is a "writer" for the FIFO.

User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Mon Feb 12, 2024 12:57 pm

Hi AndyD,


Thanks... you are being enormously helpful.

I have a lot of things to test and try: and, as I enter my 2nd week of writing C++, it all take a much longer than it should.

But first, appologies: it seems I failed to make myself clear. When I say "My reading of the documentation is" that is shorthand for "I have been reading all the documentstion I can find, and I find it somewhat sketchy - vague and incomplete - but what I think it is trying to say is". This particular choice of words is intended to convey a significant level of uncertainty.

But to try to answer your question:

In the specific case of C++ lots of sites describe five possible outcomes:
1. fgets terminates because it reads the max number of characters before encountering a \n or the end-of-file condition (EOF)
2. fgets terminates on encountering a \n
3. fgets terminates on encountering EOF after reading some (at least one) Characters
In all these three cases the the buffer will contain all the characters read (including the \n if encountered) followed by \0; and the resut will be the same as the buffer
4. fgets terminates on encountering EOF before reading any Characters
5. fgets terminates on some genuine error condition
In both cases 4 and 5 the result will be NULL, and the buffer will be unaltered. In case 4 ferror, called after fgets, will return 0, but in case 5 ferror will return a non zero error code

Have I got this right?

I can find no documentation at all which describes the specific behaviour of fgets when the file is a FIFO. I'm pretty sure it removes the characters read from the FIFO - because that's the whole point of a FIFO; but I have so far failed to find any documentation that says this.

I can find no documetation about fopen blocking on a FIFO until a writer opens the FIFO or fgets on a FIFO blocking until a writer actually puts some data into the FIFO; but that does sound right. I will run the tests you suggest to investigate this.

I can find no documentation that suggests that fgets cares a hoot about whether some writer has the file open. Indeed, I have communication between C++ (the writer, written by one of my crew) and APL (the reader) that has been working fault free for several years where both programs keep the FIFO open all the time the system is running. fgets always blocks, and that seems to be sufficient.

Your say:
The "reader" code above will block in fopen until there is a "writer" on the FIFO, and fgets will block waiting for input from the FIFO, and will return NULL once the "writer" has exited. That is why there is a need for a loop and the fclose and then fopen. You want to block in fopen until there is a "writer" for the FIFO.
This is extarordinarily helpful... thanks. It answers the the final two points above, and shows how they are interconnected.

However, the evidence I have so far suggests that:
1. fgets in an APL frogram under Buster always blocks waiting for input and never return NULL, with no need for the writer to exit
2. fgets in a C++ program under Bookworm blocks initially blocks waiting for input, but after encountering some condition stops blocking.
Originally I thought this condition was "the first successful read using fgets", but I now realize that the condition could equally well be "on encountering EOF": my early tests would did not discriminate between these to cases.



Lots of things to try.... I'll be back. But all thoughts welcome.

And I do have one question, please.

In your code you have

Code: Select all

while (fgets(msgbuf, sizeof(msgbuf), fp))
rather than

Code: Select all

while (fgets(msgbuf, sizeof(msgbuf), fp) != NULL)
I understand how any non-zero value can be treated as TRUE; but is the comparison made with the pointer to the result (I'm guessing the null ponter is actually int 0) or to the value the pointer addresses. If it's the latter case. what happens if the value is the character string "\0"?

Now to work...


Cheers

Peter

BigRedMailbox
Posts: 515
Joined: Sat Aug 20, 2022 10:37 pm

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Mon Feb 12, 2024 1:36 pm

fgets returns (on success) the value of its first arg (so, whatever type "msgbuf" is/was, that's the type of thing that fgets returns). Yes, I know this language is informal; don't bother nitpicking it.

Anyway, using: while (fgets(...))
is perfectly fine. No need to compare to NULL.
My posts may be "controversial" and/or out-of-sync with the party line.

Nothing I write should in any way be taken as an official statement by any organization connected with (any branch of) RPi and/or any of its funding sources.

User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Mon Feb 12, 2024 5:56 pm

Thanks... just seking to understand properly.

User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Mon Feb 12, 2024 7:06 pm

Hi,

I now have a working program... thanks all, especially AndyD.

There may be better ways to do this; but for now I'm just happy that it works:

Code: Select all

#include <stdio.h>
#include <iostream>
#include <cstdio>
#include <fstream>
#include <lgpio.h>
#include <regex>
#include <string>
#include <sstream>
#include <fcntl.h> 
#include <sys/stat.h> 
#include <sys/types.h> 
#include <unistd.h>
#include <cstring>
#include <bits/stdc++.h> 


int main(int argc, char *argv[])
{
printf("Starting fgets Fifo Test Program\n");
 
  char *fifoname = argv[1];
  std::string onemsg = "";
  bool exit = false;  
      while (!exit){                                       
         FILE *fd;          
         std::stringstream msgsst;
	 msgsst << onemsg;                                                     
         fd = fopen(fifoname, "r");  
	 printf ("reading %s\n", fifoname);  
         #define msglen 81                       
         char msgbuf[msglen];                    
         while (fgets(msgbuf, msglen, fd)){
		    msgsst << msgbuf;
          }
         fclose(fd); 
	 while (msgsst.peek() != EOF){
	    getline(msgsst,onemsg);
	    if (!msgsst.eof()){
	       printf("Message: %s\n", onemsg.c_str());
	       if (onemsg == "End"){
		  exit = true;
		}  
	       onemsg = "";
	     }
	  }
       }
printf("Closing Down fgets Fifo Test Program\n");
sleep(10);
return 0;
}	  
The long list of #includes is all the stuff that the real program (of which this was the one bit I could not get to work) requires.

@ AndyD: particular thanks.

You are absolutely right - if I use

Code: Select all

cat > fifo
to feed theFIFO, it blocks on the fopen until I terminate the cat (woof woof :) ). This may be something to do with cat rather than with fopen - we'll see when we put the real APL program in as the "writer". All that I know for certain is that when C++ is the writer (using open(..., O_WRONLY)) and APL the reader (using fopen (..., "r+")) no blocking occurrs except for the desired blocking in fgets while it waits for data.

Thanks again


Peter

tttapa
Posts: 155
Joined: Mon Apr 06, 2020 2:52 pm

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Mon Feb 12, 2024 10:06 pm

That looks like an extremely roundabout way to read from a file ...
You first read a line into a local buffer, then copy the buffer to a string stream, then copy the string stream to a std::string, and then get a C string from the std::string.

Why all the detours? Why not just start from a file stream (e.g. std::ifstream) in the first place?

User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Mon Feb 12, 2024 10:42 pm

Mainly because I first started writing C++ just one week ago; and I could not understand the documentation on std::ifstream.

Also, (I think) you access an ifstream with getline, which discards the \n delimiter, and in consequence, until today, I had not figured out a way to determine whether the final messge in thr input was complete (i.e terminated by the discarded \n) or incomplete (terminated by the EOF condition).

Having got it to work I'm now lookig to do it better.

Part of the problem is to do with blocking vs non-blocking reads. This may cease to be an issue when using a std:ifstream. But with fgets, on some condition, the calls to fgets stopped blocking, going into a tight loop and maxing out one CPU core while no longer reading any data that was written to the FIFO. The only way I found to stop this hapening was to close the FIFO and reopen it it. This, in turn cused further problems: namely that any data that had been in the FIFO, and read into RAM as part of fgets, but not returned in its result, got flushed on closing the FIFO. The soluton was to read the entire file before closing it (though I nos suspect that it is encountering EOF that causes the change in fgets behaviour, so other solutions may be possible). Hence the need to read the whole of the FIFOs contents, then close the FIFO, then re-process the local variable that now hold all the FIFo contents it to to get the individual messages.

Yes... I would love to access the FIFO one message at a time, action it when the \n delimiter tells me it is complete, and them move on to the next one. If I use std::ifstream will getline (or some other method) allow me to do that, sticking in blocking mode whatever happens with no need to close and re-open the file?

deepo
Posts: 1286
Joined: Sun Dec 30, 2018 8:36 pm
Location: Denmark

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Tue Feb 13, 2024 5:36 pm

In the past I have handled this by having the caller/receiver check if there is a message ready in a buffer.
A message is ready if there is a newline character in the buffer. Then the command is returned without the newline character, and the command is removed from the buffer.
Next call checks for the next newline character, if found return as above. If not found, then read 1 byte from pipe and do it all over again.
Only if the read call fails can you return the message without having a newline character.

User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Tue Feb 13, 2024 6:10 pm

That makes sense. How do you check for a \n, and is this a blocking call?

But the annoying thing is that all I actually need is a read (or fread or getline) function that will just read the FIFO in blocking mode, accessing the FIFO whenever, ad as soon as any data is written into it.

And when I write from C++ and read from APL using fgets, that's exactly what fgets does, reliably, and indefinately...

I have to be missing something :?:

Thanks


Peter

ejolson
Posts: 12155
Joined: Tue Mar 18, 2014 11:47 am

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Tue Feb 13, 2024 8:59 pm

Peter Cyriax wrote:
Tue Feb 13, 2024 6:10 pm
That makes sense. How do you check for a \n, and is this a blocking call?

But the annoying thing is that all I actually need is a read (or fread or getline) function that will just read the FIFO in blocking mode, accessing the FIFO whenever, ad as soon as any data is written into it.

And when I write from C++ and read from APL using fgets, that's exactly what fgets does, reliably, and indefinately...

I have to be missing something :?:

Thanks


Peter
As mentioned in

viewtopic.php?p=2192158#p2192158

don't close the other end of the FIFO.

User avatar
AndyD
Posts: 2456
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Sat Feb 17, 2024 2:05 am

Hi Peter,

I have more to add to this, but we have had a storm damage here in Melbourne. I was without power from Tuesday afternoon to Friday morning.

I will hopefully have time tonight (Saturday).

Andrew.

User avatar
AndyD
Posts: 2456
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Mon Feb 19, 2024 8:50 am

Hi Peter,

Not sure if you are still interested, but there was a few things that I thought was worth adding.

FIFO behaviour - Writer and Reader
  • If a program that is only a Writer Opens the FIFO for writing, then the Writer will be blocked in the Open until a Reader Opens the FIFO for reading.
  • If a program that is only a Reader Opens the FIFO for reading, the the Reader will be blocked in the Open until a Writer Opens the FIFO for writing.
  • If a program Opens the FIFO for both reading and writing, then the program won't block in Open.
Programs will block in Open, until the FIFO has been Opened for both Reading and Writing.

https://man7.org/linux/man-pages/man7/fifo.7.html
https://man7.org/linux/man-pages/man7/pipe.7.html

Reading from the FIFO

The Reader will block in the read, until there is something in the FIFO to read. There are ways to enable non-blocking IO, but that is not the default operation. Non-blocking IO needs to be set up.

If the Writer Closes the FIFO, then the Reader will receive an End of File. To continue reading from the FIFO once the Writer Re-Opens the FIFO, the Reader will need to Close and Re-Open the FIFO to continue reading from the FIFO.

Using fgets

Calls to fgets will contain the entire line, if the buffer used to big enough to hold the entire line.

https://en.cppreference.com/w/c/io/fgets

There is no need to worry about getting part of a line of text or command as long as your buffer is large enough.

As suggested, you could use a C++ stream to read from the pipe as follows:-

Code: Select all

#include <fstream>
#include <iostream>

int main()
{
    std::ifstream ins{"/tmp/my_named_pipe"};

    if (ins.is_open())
    {
        std::string line;
        while(std::getline(ins, line))
        {
            std::cout << "Received - " << line << '\n';
        }
    }
}
There is no magic cat

With possible exception for The Cheshire Cat from Alice in Wonderland and The Magical Mister Mistoffelees from Cats the Musical.
Peter Cyriax wrote:
Mon Feb 12, 2024 7:06 pm
...This may be something to do with cat rather than with fopen - we'll see when we put the real APL program in as the "writer". All that I know for certain is that when C++ is the writer (using open(..., O_WRONLY)) and APL the reader (using fopen (..., "r+")) no blocking occurrs except for the desired blocking in fgets while it waits for data.
You will get the same behaviour by running the following C program.

Code: Select all

#include <stdio.h>
#include <string.h>

int main()
{
    FILE *fp = fopen("/tmp/my_named_pipe", "w");

    if (fp)
    {
        char msgbuf[1024];
        while (fgets(msgbuf, sizeof(msgbuf), stdin))
        {
            fputs(msgbuf, fp);
            fflush(fp);
        }
        fclose(fp);
    }

    return 0;
}
I hope this helps Peter. You may have moved on. Good luck with your project.

User avatar
Peter Cyriax
Posts: 15
Joined: Sat Feb 10, 2024 8:44 pm
Location: United Kingdom

Re: C++ fgets behaving very oddly on Bookworm (no other environment tested yet)

Thu Feb 22, 2024 8:09 am

Hi Andy,


We have weather too here - though it seldom results in a power outage.


Thanks for that. I have half moved on, if I may explain:

Due to the massive help I've had here I now have a neat program that works under all circumstances that I can use: it uses ifstream and getline.

But it also uses a sledgehammer to break a nut, and I'm not happy with that. Specifically, if the reader encounters the EOF condition, my program closes and re-opens the FIFO to ensure that if no writers have the FIFO open at thst instant, the subsequent call to getline remains blocked.

The reader is a a part of a general capability, providing an interface to mutiple writers which (for a variety of reasons) are unable to call C++ or access the hardware directlty. I cannot guarantee that at least one such caller/writer will have the FIFO open at any given instant.

I have in fact tried opening the reader in read-write mode so that the FIFO always thinks there is a writer who has the FIFO open. This did not appear to work, but as you (on first reading) seem to think it should, I will read what yhou have said very carefully and try again.

I've had a problem with my Pi; as soon as I've solved that I'll post the program that I have. Obviously I'll convert it to a function once I'm completely happy to insert it into the larger system: the spec for that (ideally) is that it takes one input parameter (the ifstream pointer) and returns the next complete line of input as a stringstream (since that's the class most suitable for parsing the line). If I have to close and reopen the FIFO inside this function to ensure that calls to getline are always blocked, the input parameter woulold be the FIFO name rather than the pointer to the ifstream.

Cheers


Peter

Return to “C/C++”