User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

C scripts / C++ scripts

Sun Sep 12, 2021 4:36 pm

I know there are good reasons for only one language per source file.
Nevertheless I really like multi-language scripts.
All together in a single source makes changing anything easy, as well as makes it easy to move the file anywhere.

I remembered today when answering a posting [bash + C]:
viewtopic.php?f=37&t=319444&p=1912504#p1912448

Especially the head of my github "gamepad" tool, that allows to control a 4DoF robot arm with cheap SNES gamepad (the bash script uses pigpio library pigs commands to control the 4 servos):
https://github.com/Hermann-SW/4DoF_robo ... ls/gamepad

Code: Select all

#!/bin/bash
# contains and uses this version of Jason White's joystick.c at bottom
# https://gist.github.com/jasonwhite/c5b2048c15993d285130/eb3bcd6e1cd88002d21764b9965d075eceaf4b83
#
# requires SNES joystick connected to Raspberry USB, pigpio library installed
#
js=/tmp/joystick
if [ ! -f $js ]; then sed -n "/^\/\*\*$/,\$p" $0 | gcc -x c - -o $js; fi

g=1500
...
The bash script tests whether /tmp/joystick is existent, and if not extracts the C code from bottom of gamepad and compiles it to /tmp/joystick:

Code: Select all

...
exit
/**
 * Author: Jason White
 *
...

In my day job I am responsible as L3 support for all "IBM DataPower Gateway" compilers.
"3lang.js" is a multi-language script that utilizes 3 languages.
I used it as example in a blog posting (unfortunately IBM deleted all developerworks blogs end of 2019, inclusive my 268 blog posts -- only few are available):
https://stamm-wilbrandt.de/en/blog/3lang.js
Three of the languages DataPower supports are XSLT 1.0 (oldest), XQuery 1.0 and GatewayScript (similar to nodejs). It is possible to call XSLT from GatewayScript [with transform.xslt()], as well as to call XQuery from XSLT [with dp:xquery-transform()]. So 3lang.js executed in a GatewayScript action allows to utilize XQuery language features (most notably XPath 2.0 support) only that way. This was just demonstrator code for what is possible in case if needed.


We have seen above how C code can be extracted from bash script part after "exit" command and compiled into /tmp executable, and executed from there. Tiny C compiler "tcc" (sudo apt install tcc) allows for even easier C script with its "-⁠run" option (compile and run directly, do not store executable), and Unix shebang. This is from tcc man page:

Code: Select all

$ cat shtcc
#!/usr/bin/tcc -run
#include <stdio.h>

int main()
{
    printf("Hello World\n");
    return 0;
}
$
Directly executing this script just does what you think:

Code: Select all

$ ./shtcc
Hello World
$

Now that is a C only script, without any bash script part.
"shtcc2" utilizes the idea in initial "gamepad" script, but without storing compiled executable:
shtcc2.html.png
shtcc2.html.png
shtcc2.html.png (10.42 KiB) Viewed 2241 times
Execution:

Code: Select all

$ ./shtcc2
foo
bar
$

There is absolutely no need to use C scripts, but for simple tasks they can be quite useful.


P.S:
Syntax highlighted HTML page created with "TOHtml" tool (based on vim's "+TOhtml" feature, for 600 different syntaxes):
viewtopic.php?f=33&t=317616
Last edited by HermannSW on Sun Sep 19, 2021 7:31 am, edited 1 time in total.
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts

Mon Sep 13, 2021 4:28 pm

One possible application for C script is "multi-piping".
In bash shell you have the pipe operator, but I don't know of any way to split input onto 2 or more pipes in bash.
In C execution of "popen()" does execute a binary with a pipe to it, allowing for splitting into multiple pipes.

I will use that to split 4056x3040 12MP HQ camera YUV i420 frame into six 2028x1080 subframes covering the whole frame, and then use i420tohtml tool (executed by popen) to .h264 encode each of the 6 frames. Just opening 6 pipes with popen(), code to extract 2028x1080 i420 subframe works already:
viewtopic.php?f=43&t=316426&p=1912087#p1912087

While I will likely create a pure C program doing so, the "multi-piping" idea can be used in a C script for something else, allowing the bash part of C script utilize "multi-piping".
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

trejan
Posts: 3857
Joined: Tue Jul 02, 2019 2:28 pm

Re: C scripts

Mon Sep 13, 2021 5:49 pm

HermannSW wrote:
Mon Sep 13, 2021 4:28 pm
In bash shell you have the pipe operator, but I don't know of any way to split input onto 2 or more pipes in bash.
https://tldp.org/LDP/abs/html/process-sub.html

Heater
Posts: 18758
Joined: Tue Jul 17, 2012 3:02 pm

Re: C scripts

Mon Sep 13, 2021 10:19 pm

Memory in C++ is a leaky abstraction .

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts

Tue Sep 14, 2021 7:23 am

I don't see how sub processes should help to split "one" input pipe onto more than one output pipe.
I did not think of "tee" command for this scenario.
I can only get it working with result in wrong order (output first 3 lines of input using first pipe, then last two lines of input using second pipe):

Code: Select all

pi@raspberrypi400:~ $ echo -e "1\n2\n3\n4\n5\n6\n7" | tee >(head -3) | tail -5
6
7
1
2
3
pi@raspberrypi400:~ $ 

Any idea for solving this test problem with "tee", or with sub processes?


I went ahead and created C script solution "headtail.c" for this.
I like to have ".c" suffix although it is a script for indication that C part is contained.
First time that a "*.c" file needs to be executable for me ;-)

Here you can see that it does its job, demonstrating splitting input pipe onto more than one output pipe:

Code: Select all

pi@raspberrypi400:~ $ echo -e "1\n2\n3\n4\n5\n6\n7" | ./headtail.c 
1
2
3
6
7
pi@raspberrypi400:~ $ 

Here is headtail.c:
("$0 -nt $exc" is better than "! -f $exc" used before, since it triggers recompilation in case script has been modified)
https://gist.github.com/Hermann-SW2/209 ... 12e438dc37
headtail.c.html.png
headtail.c.html.png
headtail.c.html.png (30.32 KiB) Viewed 2024 times
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts

Tue Sep 14, 2021 10:26 am

I would not have thought that generalized output to n pipes would be so easy to have.
Of course I did it with C script.
Demo with 3 pipes:

Code: Select all

$ echo -e "1\n2\n3\n4\n5\n6\n7\n8\n9\n10" | ./pipn.c "head -3" "wc" "tail -2"
1
2
3
     10      10      21
9
10
$ 

Now demo with "pipe in pipe":

Code: Select all

$ echo -e "1\n2\n3\n4\n5\n6\n7\n8\n9\n10" | ./pipn.c "head -3 | wc" "tail -2"
      3       3       6
9
10
$ 

"pipn.c" processes each passed argument command line with bash -c "...", which is generated in "cmd()". In previous posting "fgets()" was for text files only. Now with "fread()" and "fwrite()" arbitrary (binary) data can be pipe processed as well:
https://gist.github.com/Hermann-SW2/c69 ... e33673c162
pipn.c.html.png
pipn.c.html.png
pipn.c.html.png (61.56 KiB) Viewed 1959 times

P.S:
New bash "operator" |n pipes into more than 1 pipe:
viewtopic.php?f=66&t=319603
Simple demo for piping into 3 pipes, with 2nd "pipe of pipes":

Code: Select all

$ cat /etc/X11/rgb.txt |n 'head -3' "head -55 | tail -17 | wc" "tail -2"
! $Xorg: rgb.txt,v 1.3 2000/08/17 19:54:00 cpqbld Exp $
255 250 250		snow
248 248 255		ghost white
     17      79     407
144 238 144		light green
144 238 144		LightGreen
$ 
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

trejan
Posts: 3857
Joined: Tue Jul 02, 2019 2:28 pm

Re: C scripts

Tue Sep 14, 2021 3:00 pm

HermannSW wrote:
Tue Sep 14, 2021 7:23 am
I don't see how sub processes should help to split "one" input pipe onto more than one output pipe.

Code: Select all

$ cat /etc/X11/rgb.txt | tee >(head -3) >(head -55 | tail -17 | wc) >(tail -2) > /dev/null
! $Xorg: rgb.txt,v 1.3 2000/08/17 19:54:00 cpqbld Exp $
255 250 250             snow
248 248 255             ghost white
     17      79     407
144 238 144             light green
144 238 144             LightGreen

trejan
Posts: 3857
Joined: Tue Jul 02, 2019 2:28 pm

Re: C scripts

Tue Sep 14, 2021 3:17 pm

You should change your pipn program to call setvbuf(pip[stupidforumwontletmeuseihere], NULL, _IONBF, 0) on each process stream to turn off buffering.

User avatar
Paeryn
Posts: 3366
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: C scripts

Tue Sep 14, 2021 4:01 pm

You might want to change the call to sprintf() into a call to snprintf() to prevent a possible buffer overflow

Code: Select all

  // limit the number of chars written to the size of the buffer
  snprintf( cmds, siz, "bash -c \"%s\"", s );
Though you'll also want to either make sure the whole string was written in case it was cropped or dynamically allocate the buffer so it's always big enough to hold the complete string.
She who travels light — forgot something.
Please note that my name doesn't start with the @ character so can people please stop writing it as if it does!

Heater
Posts: 18758
Joined: Tue Jul 17, 2012 3:02 pm

Re: C scripts

Tue Sep 14, 2021 4:12 pm

HermannSW wrote:
Tue Sep 14, 2021 7:23 am
I did not think of "tee" command for this scenario.
I can only get it working with result in wrong order (output first 3 lines of input using first pipe, then last two lines of input using second pipe):
I'm pretty sure the C code your presented is not guaranteed to produce the output in the order you expect either. You have a race condition.

You have spun up two processes, one for 'head' and one for 'tail'. They then run independently and write their output to stdout.

There is no guarantee that the Linux kernel will schedule them in any order you expect.
Memory in C++ is a leaky abstraction .

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts

Tue Sep 14, 2021 8:03 pm

With the help of @Trejan in the other thread
viewtopic.php?f=66&t=319603&p=1913408#p1913414
tee command can definitely do what I tried to do with "pipn.c" .
So no need for "pipn.c" anymore (besides demo for "C script").
It is most times the best to use existing Linux command line tools.

The best general form of piping into N pipes is this:
viewtopic.php?f=66&t=319603&p=1913408#p1913540

Code: Select all

... | (tee >(cmd1) >(cmd2) ... >(cmdN) > /dev/null) ...
Heater wrote:
Tue Sep 14, 2021 4:12 pm
I'm pretty sure the C code your presented is not guaranteed to produce the output in the order you expect either. You have a race condition.

You have spun up two processes, one for 'head' and one for 'tail'. They then run independently and write their output to stdout.

There is no guarantee that the Linux kernel will schedule them in any order you expect.
You are right, this happened in posting pointed to last, same command, different order of outputs (see posting on final "| cat"):

Code: Select all

$ cat /etc/X11/rgb.txt | (tee >(head -3) >(head -55 | tail -17 | wc) >(tail -2) > /dev/null) | cat
! $Xorg: rgb.txt,v 1.3 2000/08/17 19:54:00 cpqbld Exp $
255 250 250		snow
248 248 255		ghost white
144 238 144		light green
144 238 144		LightGreen
     17      79     407
$ 

Code: Select all

$ cat /etc/X11/rgb.txt | (tee >(head -3) >(head -55 | tail -17 | wc) >(tail -2) > /dev/null) | cat
! $Xorg: rgb.txt,v 1.3 2000/08/17 19:54:00 cpqbld Exp $
255 250 250		snow
248 248 255		ghost white
     17      79     407
144 238 144		light green
144 238 144		LightGreen
$

P.S:
So Linux tee command allows to pipe into two pipes.
That would allow for "divide" step of divide and conquer merge sort.

Just solved the "conquer" step, merging of two sorted seuqences into one sorted sequence.
I think I never used Linux comm command before.
Use of sed is needed in order to output data identical in both inputs (column 3) two times.
comm_sed.png
comm_sed.png
comm_sed.png (13.42 KiB) Viewed 1843 times

It is currently not clear how to combine for complete recursive divide and conquer pipe mergesort.
Definitely not needed since we have Linux sort command.
But doing pipe mergesort of 2^n rows file with 2^n parallel pipes in between (before conquer steps) sounds like fun ...
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts

Wed Sep 15, 2021 7:36 pm

I did split off this new thread as followup on "comm" usage mentioned in previous posting:
"Parallel recursive bash mergesort / fun / challenge"
viewtopic.php?f=31&t=319687
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts

Thu Sep 16, 2021 1:46 pm

I answered on stackoverflow question "How to compile and execute from memory directly?"
https://stackoverflow.com/a/69204024/5674289

I mention tcc's "-run" option and showed tcc base C script from previous posting in this thread. Then I realized that the way I did the C script, the script's stdin is not available to C code executed by "tcc -run". Reason was piping C code extracted from C script via pipe.

"run_from_memory_stdin.c" demonstrates accessing C script stdin by C code, as well as working argv:
https://gist.github.com/Hermann-SW/7bc1 ... 5b15139c5f

Code: Select all

$ route -n | ./run_from_memory_stdin.c 
foo
bar 42
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.29.58.98    0.0.0.0         UG    306    0        0 wlan1
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 wlan0
169.254.0.0     0.0.0.0         255.255.0.0     U     303    0        0 wlan0
172.29.58.96    0.0.0.0         255.255.255.252 U     306    0        0 wlan1
$ 

The trick was to pass extracted C code this way:

Code: Select all

...
echo "foo"
tcc -run <(sed -n "/^\/\*\*$/,\$p" $0) 42
...
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts / C++ scripts

Sun Sep 19, 2021 7:28 am

I created C memrun repo, that allows to execute ELF binary in memory (verified for x86_64 as well as armv7l):
https://github.com/Hermann-SW/memrun/tr ... r/C#memrun

With that "run from memory" tool gcc as well can be used for "C script" without auxiliary filesystem files created. Since tcc is a C compiler only, "tcc -run" cannot help for "C++ script", while memrun can!
https://github.com/Hermann-SW/memrun/tr ... c-script-1

Code: Select all

pi@raspberrypi400:~/memrun/C $ echo "2^64" | bc -q | ./run_from_memory_cin.cpp 
foo
bar 42
18446744073709551616
pi@raspberrypi400:~/memrun/C $ 

This is (executable) run_from_memory_cin.cpp producing above mixed bash and C++ output:
run_from_memory_cin.cpp.html.png
run_from_memory_cin.cpp.html.png
run_from_memory_cin.cpp.html.png (20.23 KiB) Viewed 1410 times
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts / C++ scripts

Sun Sep 19, 2021 11:12 am

The C/C++ scripts seen sofar only showed how bash and C/C++ in single file can work. New section "C/C++ interaction with bash in script" describes a two year old C script, converted to "run from memory" (gamepad.c):
https://github.com/Hermann-SW/memrun/tr ... -in-script

This is how bash and C code parts work together (C code stdout output is input for bash part), in "done" line with 3 nested process substitutions:

Code: Select all

...
while IFS= read -r line; do
    case $line in
        ...
    esac
    ...
    echo $g $u $l $p $d $calib

Code: Select all

    pigs s 8 $g; pigs s 9 $u; pigs s 10 $l; pigs s 11 $p

done < <(./memrun <(gcc -o /dev/fd/1 -x c <(sed -n "/^\/\*\*$/,\$p" $0)))

exit
/**
 * Author: Jason White
...  
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts / C++ scripts

Wed Oct 06, 2021 3:14 pm

HermannSW wrote:
Sun Sep 19, 2021 7:28 am
...
This is (executable) run_from_memory_cin.cpp producing above mixed bash and C++ output:
Image
Last night I made compilation simpler (and making sure that all g++ temporary files during compilation get stored in RAM as well) by simply making use of bin/g++. I only left memrun use in gamepad.c, the other scripts use memfd_create instead:
https://github.com/Hermann-SW/memrun

Code: Select all

...
echo "foo"
#./memrun <(g++ -o /dev/fd/1 -x c++ <(sed -n "/^\/\*\*$/,\$p" $0)) 42
bin/g++ -run <(sed -n "/^\/\*\*$/,\$p" $0) 42
...

Since last night use of dd as well as mkfs.ext4 are eliminated from bin/grun (liked by bin/gcc and bin/g++) by extracting prebuilt filesystem into the "memfd_create"ed memory file:

Code: Select all

...
# create 10MB filesystem into memory file
#dd if=/dev/zero of="$mf" bs=1024 count=10240 2> /dev/null
#mkfs.ext2 "$mf" 2>&1 | cat > /dev/null 
gzip -dc "$dn"/10MB.fs.gz > "$mf"
...

In case no interaction between bash and C/C++ is needed, "bin/g(cc|++)" can be used in shebang like tcc:

Code: Select all

pi@raspberrypi400:~/memrun/C $ ./HelloWorld.c
Hello, World!
pi@raspberrypi400:~/memrun/C $ cat HelloWorld.c
#!/home/pi/memrun/C/bin/gcc -run
#include <stdio.h>

int main(void)
{
  printf("Hello, World!\n");
  return 0;
}
pi@raspberrypi400:~/memrun/C $
Last edited by HermannSW on Wed Oct 06, 2021 3:21 pm, edited 2 times in total.
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

GlowInTheDark
Posts: 2098
Joined: Sat Nov 09, 2019 12:14 pm

Re: C scripts / C++ scripts

Wed Oct 06, 2021 3:16 pm

Did you ever get around to implementing this as a bash built-in?
Poster of inconvenient truths.

Back from a short, unplanned vacation. Did you miss me?

User avatar
HermannSW
Posts: 4821
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: C scripts / C++ scripts

Wed Oct 06, 2021 3:24 pm

GlowInTheDark wrote:
Wed Oct 06, 2021 3:16 pm
Did you ever get around to implementing this as a bash built-in?
Not yet, will have to learn what you did in the other thread:
viewtopic.php?f=31&t=320170&p=1921110#p1921107
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

Return to “C/C++”