Umadbro
Posts: 4
Joined: Tue Sep 14, 2021 10:58 am

Understanding and customising default directories on raspberry pi

Tue Sep 14, 2021 11:43 am

Hi. This will be my first post, although I already have some experience using this device.

I own a Raspberry Pi 3 Model B, Ver.1.2, an amazing little device which is capable of more than I have ever anticipated. In a single setup I had my Pi doing vnc server, pi hole, vpn, media server, bluetooth sound server, plex server, ffmpeg converter and Sonarr+Radarr+Lidarr setup with torrents and nzb working. I was amazed with the fact that I was able to have all of this working at the same time, with minimal to no issues.

The big problem came when once I had to unplug my external hard drive, and a bunch of files filled up my internal memory. Suddenly I wasn't able to boot into GUI.

I managed to fix it the first time, with a lot of patience, I looked for the rogue files, deleted them, and my setup was, once again, ready to rock.

Unfortunately it happened again :roll:

This time, I decided to do it from scratch, so I can freshen up my code, and have it a bit more "organized".

In windows, i always prefer to setup the basic in my main OS drive. So I setup windows, update, install my most used software and when I feel I have my base software good to go, I back up this drive, and use my other disks for installations, data management e.t.c., always making my own directories, even for new software I install.

I was looking to do it the same way this time with my raspberry Pi. For example, instead of having commands like "apt-get" installing the software/libraries in their default places, being able to organize it my own way, so I can always have my main system directories more or less the same, while storing all user added software in a separate folder/memory.

The thing is, although I have a fair amount of experience using this device, I'm still a bit green with the whole system directory default layout, and I'm quite sure what I want to achieve isn't as easy as creating a new folder and dumping anything and everything into it. So I come here to ask for some enlightenment.

I would really appreciate if someone could go a bit in depth into this with me, I'm already sitting in front of a barebone setup of raspberry O.S., so I'm ready to rock!!

bjtheone
Posts: 1502
Joined: Mon May 20, 2019 11:28 pm
Location: The Frozen North (AKA Canada)

Re: Understanding and customising default directories on raspberry pi

Tue Sep 14, 2021 1:29 pm

While I share you lack of enthusiasm for some of the directory placement decisions (big fan of /opt), giving up on apt puts a very significant burden on you, both in installing and maintaining. You could consider some of the "new fangled" distribution systems (snap, flatpak) that allow you to separate out all the "user applications" but they come with their own issues.

I suspect your file system filing issues came from files getting written to the mount point directory, when the external drive was removed.

swampdog
Posts: 777
Joined: Fri Dec 04, 2015 11:22 am

Re: Understanding and customising default directories on raspberry pi

Tue Sep 14, 2021 3:20 pm

Linux considers all devices to be in one filing system under "/" (rootfs). With the rpi you have two by default - rootfs under / and /boot/ which is a vfat. Any further devices are mounted on top of arbitrary directories. The GUI automounter (which can be disabled) will invent these when you plug a usb device or you can force mounts into fixed places via custom scripts or /etc/fstab.

Should you mount a device onto a directory which already contains stuff, the old stuff will be inaccessible until you unmount the device off that directory, at which point the old stuff will reappear.

All sane user apps store their data somewhere inside $HOME. If you can't boot with the rpi set to "console/noautologin" then it's an indication of cockpit error: you're storing your data in the wrong place which itself may be on account of being the root user when you shouldn't be.

klricks
Posts: 8018
Joined: Sat Jan 12, 2013 3:01 am
Location: Grants Pass, OR, USA
Contact: Website

Re: Understanding and customising default directories on raspberry pi

Tue Sep 14, 2021 3:49 pm

I had a similar issue when my external disk failed. Everything was being written to the mount point on the SD card and my SD soon filled up with no warning. Regarding a better way of mounting see the last post here: viewtopic.php?f=28&t=124983
Unless specified otherwise my response is based on the latest and fully updated RPiOS Buster w/ Desktop OS.

bjtheone
Posts: 1502
Joined: Mon May 20, 2019 11:28 pm
Location: The Frozen North (AKA Canada)

Re: Understanding and customising default directories on raspberry pi

Tue Sep 14, 2021 7:04 pm

swampdog wrote:
Tue Sep 14, 2021 3:20 pm
All sane user apps store their data somewhere inside $HOME. If you can't boot with the rpi set to "console/noautologin" then it's an indication of cockpit error: you're storing your data in the wrong place which itself may be on account of being the root user when you shouldn't be.
I would gently disagree and go with all sane apps should store their data where I want/specify them to. User config data most likely should be stored in a standard place in the user account. Other data, especially data accessed by multiple users has absolutely zero business being inside a user account and Linux and Windows applications that assume I am effectively running a single user system annoy the heck out of me.

A great example... Calibre. I run a epub library system on one of my pi. I do the admin and volume addition, multiple folks in my household consume books from the library. My app config (mostly defaults and look & feel stuff) lives in my account, library info (location, etc) lives in the app configuration directory, and the actual library lives on a seperate partition where data lives.

swampdog
Posts: 777
Joined: Fri Dec 04, 2015 11:22 am

Re: Understanding and customising default directories on raspberry pi

Tue Sep 14, 2021 11:25 pm

bjtheone wrote:
Tue Sep 14, 2021 7:04 pm
swampdog wrote:
Tue Sep 14, 2021 3:20 pm
All sane user apps store their data somewhere inside $HOME. If you can't boot with the rpi set to "console/noautologin" then it's an indication of cockpit error: you're storing your data in the wrong place which itself may be on account of being the root user when you shouldn't be.
I would gently disagree and go with all sane apps should store their data where I want/specify them to. User config data most likely should be stored in a standard place in the user account. Other data, especially data accessed by multiple users has absolutely zero business being inside a user account and Linux and Windows applications that assume I am effectively running a single user system annoy the heck out of me.

A great example... Calibre. I run a epub library system on one of my pi. I do the admin and volume addition, multiple folks in my household consume books from the library. My app config (mostly defaults and look & feel stuff) lives in my account, library info (location, etc) lives in the app configuration directory, and the actual library lives on a seperate partition where data lives.
More recent apps tend to create a folder in $HOME/.config/ folder whereas older ones (like mine) had to roll their own solution ($HOME/.sd/appname/) and real old ones simply create their config file directly in $HOME and haven't changed due to convention (bash for instance). In fact bash is a good trivial example. Edit /etc/bash.bashrc for global changes. Many apps have such a config file installed somewhere.

I'm not familiar with calbre but it does sound like it is doing the right thing.

Even without a global config file, judicious use of symlinks can work wonders.

Umadbro
Posts: 4
Joined: Tue Sep 14, 2021 10:58 am

Re: Understanding and customising default directories on raspberry pi

Thu Sep 16, 2021 11:38 am

I suspect your file system filing issues came from files getting written to the mount point directory, when the external drive was removed.

I can confirm that this is true. After creating this post I ended up going back to my project and with little to no effort, I managed to find the rouge folder taking all the space. It ended up being FFmpeg converter the culprit. Due to the lack of audio/video codecs with which Pi's processor can work with, many of the media I tried to stream to my smart tv would stop or buffer, making it impossible to actually watch anything. So I managed to compile a version to work with my raspberry pi, except that I never managed to automate the process. Well, it seems that that wasn't the only issue :?
giving up on apt puts a very significant burden on you, both in installing and maintaining
You could consider some of the "new fangled" distribution systems (snap, flatpak) that allow you to separate out all the "user applications" but they come with their own issues.

I do appreciate your suggestion, but as you said this will bring more problems than anything, and is probably not worth the trouble, being that the objective is only to make it so the file tree structure seems a bit more familiar (like windows).

Even though I managed to get all this software working in conjunction to an end objective, I am not as experienced with Linux, as I am with windows. I do have a good memory to recall directories or keyword commands, but the way files are organized by the system seems a bit scattered and not so easy to understand, which made me have to research for the many tutorials I already followed, countless times, only to find a single command line to direct me to file stored in a completely different folder.

In Windows, as we all know, mostly, files as stored in: Root; Program files; Program files (x86); Documents; ProgramData; AppData; Custom directory. Generally, when a piece of software is installed, all its core files are stored in Program Files/Program Files (x86), while its data is stored in ProgramData/AppData, and its created files in Documents, making it relatively easy to look for specific files, for any given software.

It doesn't seem to be the case with Linux. I can't give the right examples now, as I'm momentarily unable to connect to my pi via vnc, but for instance, if a software is installed without the apt-get command, instead of being stored in /usr or /usr/local, it will most likely be stored in / or /home/usr, as it happened with some of the software I used to make ffmpeg work, probably my fault because I didn't specify the right place for files to be stored, but once I noticed, I was afraid to move files around, as I didn't know if by doing it would make the whole software to stop working, being that I couldn't tell what other files would be married to the location the main software files were on.

I did find a well-structured explanation for the default file directories used in Linux, see https://www.thegeekstuff.com/2010/09/li ... structure/

I guess my best bet is to start from scratch but paying close attention to where I let my software be stored, and taking notes of all the directories I'll ever need to access, so I can avoid having files scattered all over my SD.

User avatar
thagrol
Posts: 5754
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Understanding and customising default directories on raspberry pi

Thu Sep 16, 2021 12:13 pm

Umadbro wrote:
Thu Sep 16, 2021 11:38 am
I suspect your file system filing issues came from files getting written to the mount point directory, when the external drive was removed.

I can confirm that this is true.
Try making your empty mount points read only. Things will then fail rather than fill up your SD card.
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

Umadbro
Posts: 4
Joined: Tue Sep 14, 2021 10:58 am

Re: Understanding and customising default directories on raspberry pi

Thu Sep 16, 2021 1:47 pm

Try making your empty mount points read only. Things will then fail rather than fill up your SD card.[/quote


That's a great idea, I'll take care of that.

It makes total sense now that you mention it, although I can't figure out why would this happen with the converter that fetches files from the downloaded folder on an external drive, and not with the downloaders creating those same files in the external hard drive. By this order of ideas, it would mean that when the hard drive is not found, it would start writing to the mount point where otherwise that drive would be addressed to, not only by the converter, but also every active software writing to the disk... maybe the converter was only faster than anything else, explaining why I'd only find those files and nothing else.

User avatar
thagrol
Posts: 5754
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Understanding and customising default directories on raspberry pi

Thu Sep 16, 2021 2:02 pm

More likely missing sub directories.

A process that writes to /mountpoint/foo/somefile but that does not creat the subdirectory "foo" will fail if the drive is unmounted as /montpoint/foo does not exist.

A process that writes to /mountpoint/somefile will succeed as will a process that creates any missing subdirectoriesd it needs.
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

bjtheone
Posts: 1502
Joined: Mon May 20, 2019 11:28 pm
Location: The Frozen North (AKA Canada)

Re: Understanding and customising default directories on raspberry pi

Thu Sep 16, 2021 6:08 pm

Umadbro wrote:
Thu Sep 16, 2021 11:38 am
I suspect your file system filing issues came from files getting written to the mount point directory, when the external drive was removed.

I can confirm that this is true. After creating this post I ended up going back to my project and with little to no effort, I managed to find the rouge folder taking all the space. It ended up being FFmpeg converter the culprit. Due to the lack of audio/video codecs with which Pi's processor can work with, many of the media I tried to stream to my smart tv would stop or buffer, making it impossible to actually watch anything. So I managed to compile a version to work with my raspberry pi, except that I never managed to automate the process. Well, it seems that that wasn't the only issue :?
giving up on apt puts a very significant burden on you, both in installing and maintaining
You could consider some of the "new fangled" distribution systems (snap, flatpak) that allow you to separate out all the "user applications" but they come with their own issues.

I do appreciate your suggestion, but as you said this will bring more problems than anything, and is probably not worth the trouble, being that the objective is only to make it so the file tree structure seems a bit more familiar (like windows).

Even though I managed to get all this software working in conjunction to an end objective, I am not as experienced with Linux, as I am with windows. I do have a good memory to recall directories or keyword commands, but the way files are organized by the system seems a bit scattered and not so easy to understand, which made me have to research for the many tutorials I already followed, countless times, only to find a single command line to direct me to file stored in a completely different folder.

In Windows, as we all know, mostly, files as stored in: Root; Program files; Program files (x86); Documents; ProgramData; AppData; Custom directory. Generally, when a piece of software is installed, all its core files are stored in Program Files/Program Files (x86), while its data is stored in ProgramData/AppData, and its created files in Documents, making it relatively easy to look for specific files, for any given software.

It doesn't seem to be the case with Linux. I can't give the right examples now, as I'm momentarily unable to connect to my pi via vnc, but for instance, if a software is installed without the apt-get command, instead of being stored in /usr or /usr/local, it will most likely be stored in / or /home/usr, as it happened with some of the software I used to make ffmpeg work, probably my fault because I didn't specify the right place for files to be stored, but once I noticed, I was afraid to move files around, as I didn't know if by doing it would make the whole software to stop working, being that I couldn't tell what other files would be married to the location the main software files were on.

I did find a well-structured explanation for the default file directories used in Linux, see https://www.thegeekstuff.com/2010/09/li ... structure/

I guess my best bet is to start from scratch but paying close attention to where I let my software be stored, and taking notes of all the directories I'll ever need to access, so I can avoid having files scattered all over my SD.
So my background is very long time Unix/Linux user and former sysadmin. Installed "properly", most things do go into directories that are mostly predictable, consistent and make sense. Between that and appropriately setup paths, it works very well and you should not need to "know" where the actual files are. With a correctly configured user environment, the user should be able to just launch and run applications. However, they do get installed in "standard" places that are easy to find, with predictable configurations. Typically applications have both a system and user level configuration, so you always get sane/functional default behaviour, that can be overridden at the user level.

If you install and update software as root (or via sudo) using apt, well built applications "should" end up installed in correct directories. There is a mostly "standard" structure for both Unix and Linux that varies slightly distro to distro. Potentially some of your issue is that you just do not yet have the experience/background with Linux for this to make sense. It makes much more sense and is easier to administer than the Windows registry. :-)

If you are installing stuff manually, you need to have a consistent plan and ensure you are installing stuff in rational (correct) places. It certainly is possible to manually compile and install stuff that behaves properly.

Installing stuff via multiple processes, as different users, is typically what gets you binaries scattered all over the place with some inside user accounts (where they have absolutely no business being, in my opinion). This is what gets you stuff installed at / or in /home/user. Stuff should almost never be installed in either of these places. Applications have no business at / and applications installed inside one user's account are typically not available/visible to other users. Having said this, many folks run Pi's as "single user" systems with only one user level (pi) account. That hides a multitude of sins.

swampdog
Posts: 777
Joined: Fri Dec 04, 2015 11:22 am

Re: Understanding and customising default directories on raspberry pi

Fri Sep 17, 2021 7:26 am

Umadbro wrote:
Thu Sep 16, 2021 11:38 am
I guess my best bet is to start from scratch but paying close attention to where I let my software be stored, and taking notes of all the directories I'll ever need to access, so I can avoid having files scattered all over my SD.
When compiling from source you need to make some fundamental decisions. It can be trivially easy (in that everything ends up installed into /usr/local/) but the moment you encounter a dependency, a plan is required. It might simply require you to install some "dev" packages but there may come a time when those dev packages themselves are too old so you are forced to rebuild those: it may be you can get away with removing the old dev packages but that's not always possible if you have other projects which still require them. At this point things get very tricky because you need to construct a build environment where your bleeding edge build ignores the system dev packages in favour of the replacements you've compiled whereas your other projects still favour the system dev packages and don't see the newly compiled ones: and yes, various projects may need a mixture of both. This is exacerbated by runtime issues - where you manage to compile against the new dev library but at runtime the project loads the old system library.

For 'configure' type projects, always set "--prefix" and for cmake it's CMAKE_INSTALL_PREFIX. Runtime issues LD_LIBRARY_PATH.

Many builds, even if they do have a functional uninstall will still leave config files around. The solution to this is to install everything down a silly path until it all builds cleanly. This implies you need to write some shell script to control it which is no bad thing as it serves as your documentation for future reference. The main advantage of using a silly path is you can delete the entire folder in order to start from scratch. Additionally you can "chown" that folder as yourself thus removing the requirement for 'sudo' on "make install" etc.

Fwiw, as I build a lot of stuff from source, I always create a kind of administrative user but you won't need that unless you plan on porting source code builds around from different unix distros.

So, generically, start your build script with a few basic variables. I use PFX for the prefix, SRC for the package name, OBJ for the build folder and CFG for configuration options. I'm using a linux PC hence the "--disable-x86asm" (because I don't want to install an assembler)..

Code: Select all

foo@sdu ~/usr/src/ffmpeg $ cp -v ffmpeg-snapshot.tar.bz2 ffmpeg.tar.bz2 
'ffmpeg-snapshot.tar.bz2' -> 'ffmpeg.tar.bz2'

Code: Select all

#!/bin/bash

: ${PFX:="/usr/local/junk"}
: ${SRC:="ffmpeg"}
OBJ="obj-""$SRC"
CFG="
--prefix=$PFX
--disable-x86asm
"

tar xvjf "$SRC"".tar.bz2"

[ -d "$PFX" ] || {
	sudo mkdir "$PFX"
	sudo chown `id -un`:`id -gn` "$PFX"
	chmod 755 "$PFX"
}

mkdir -p "$OBJ" log || exit 1

(
 cd "$OBJ"
 ../"$SRC"/configure --help 2>&1 | tee ../log/hlp.log
 ../"$SRC"/configure $CFG 2>&1 | tee ../log/cfg.log
 [ ${PIPESTATUS[0]} -eq 0 ] || return 1
) || exit 1

(
 cd "$OBJ" || exit 1
 make -j`nproc` 2>&1 | tee ../log/mak.log
 [ ${PIPESTATUS[0]} -eq 0 ] || return 1
) || exit 1

(
 cd "$OBJ" || exit 1
 make install 2>&1 | tee ../log/ins.log
 [ ${PIPESTATUS[0]} -eq 0 ] || return 1
) || exit 1

echo rm -r "$OBJ" "$SRC"
echo rm -r "$PFX"
I crammed in as much info as possible into as little space as possible. You'd rewrite the above into functions. Note I haven't used (say) "PFX=$HOME/junk" because if if I accidentally edited that to "SRC=$HOME/" I'd lose my entire home directory on the "rm -r $PFX" line whereas /usr/local/ is owned by root so it will fail (because we never(*) have to be root, above).

Note the contents of "log/hlp.log" and the top of "log/cfg.log". The former yields options you can tweak and the latter confirms where it will install.

(*) well ok, we are root to create $PFX but that only needs to be done once, manually. In reality you'd set (say) PFX=/usr/local/junk/ffmpeg and be able to dispense with "sudo" entirely.

Umadbro
Posts: 4
Joined: Tue Sep 14, 2021 10:58 am

Re: Understanding and customising default directories on raspberry pi

Fri Sep 17, 2021 10:07 am

If you install and update software as root (or via sudo) using apt, well built applications "should" end up installed in correct directories. There is a mostly "standard" structure for both Unix and Linux that varies slightly distro to distro. Potentially some of your issue is that you just do not yet have the experience/background with Linux for this to make sense. It makes much more sense and is easier to administer than the Windows registry. :-)

Yes, I understand that this is the main issue. I work constantly with Windows for a long time, as my IT job forces me to, so I feel very comfortable managing or modifying files in this environment, obviously, things are very different in Linux, in its own way, and I'm still getting used to it.

Many builds, even if they do have a functional uninstall will still leave config files around. The solution to this is to install everything down a silly path until it all builds cleanly. This implies you need to write some shell script to control it which is no bad thing as it serves as your documentation for future reference. The main advantage of using a silly path is you can delete the entire folder in order to start from scratch. Additionally you can "chown" that folder as yourself thus removing the requirement for 'sudo' on "make install" etc.

I'm having a hard time decoding the rest of your post :lol:

But this you said seems like a good step forward, not exactly what I had in mind but it works for me. Creating a separate folder in which I could download and setup my next project files, before adding them to their respective place in the default system's folders. If I'm not happy with the result, I can simply delete the whole thing and be sure I won't be leaving anything behind. Did I get it right?

bjtheone
Posts: 1502
Joined: Mon May 20, 2019 11:28 pm
Location: The Frozen North (AKA Canada)

Re: Understanding and customising default directories on raspberry pi

Fri Sep 17, 2021 12:39 pm

Umadbro wrote:
Fri Sep 17, 2021 10:07 am
If you install and update software as root (or via sudo) using apt, well built applications "should" end up installed in correct directories. There is a mostly "standard" structure for both Unix and Linux that varies slightly distro to distro. Potentially some of your issue is that you just do not yet have the experience/background with Linux for this to make sense. It makes much more sense and is easier to administer than the Windows registry. :-)

Yes, I understand that this is the main issue. I work constantly with Windows for a long time, as my IT job forces me to, so I feel very comfortable managing or modifying files in this environment, obviously, things are very different in Linux, in its own way, and I'm still getting used to it.

Many builds, even if they do have a functional uninstall will still leave config files around. The solution to this is to install everything down a silly path until it all builds cleanly. This implies you need to write some shell script to control it which is no bad thing as it serves as your documentation for future reference. The main advantage of using a silly path is you can delete the entire folder in order to start from scratch. Additionally you can "chown" that folder as yourself thus removing the requirement for 'sudo' on "make install" etc.

I'm having a hard time decoding the rest of your post :lol:

But this you said seems like a good step forward, not exactly what I had in mind but it works for me. Creating a separate folder in which I could download and setup my next project files, before adding them to their respective place in the default system's folders. If I'm not happy with the result, I can simply delete the whole thing and be sure I won't be leaving anything behind. Did I get it right?
Swampdog's solution is tailored around compiling from source and not particularly relevant if you mostly are installing prebuilt stuff. The key is to try and stick with stuff from the standard repository (installed using apt, or one of the GUI frontends to apt) and pay attention when you are installing stuff from other places.

You certainly could install to a seperate/test space and move, but that puts getting all the symlinks and config files right on you.

swampdog
Posts: 777
Joined: Fri Dec 04, 2015 11:22 am

Re: Understanding and customising default directories on raspberry pi

Fri Sep 17, 2021 2:00 pm

Umadbro wrote:
Fri Sep 17, 2021 10:07 am
swampdog wrote:Many builds, even if they do have a functional uninstall will still leave config files around. The solution to this is to install everything down a silly path until it all builds cleanly. This implies you need to write some shell script to control it which is no bad thing as it serves as your documentation for future reference. The main advantage of using a silly path is you can delete the entire folder in order to start from scratch. Additionally you can "chown" that folder as yourself thus removing the requirement for 'sudo' on "make install" etc.

I'm having a hard time decoding the rest of your post :lol:

But this you said seems like a good step forward, not exactly what I had in mind but it works for me. Creating a separate folder in which I could download and setup my next project files, before adding them to their respective place in the default system's folders. If I'm not happy with the result, I can simply delete the whole thing and be sure I won't be leaving anything behind. Did I get it right?
Pretty much, except you'd download the source code into your home folder because you'll want to keep that and your build script. I use "~/usr/src/" for instance (aka $HOME/usr/src) the tilde (~) being a shortcut for $HOME and $HOME is set for you when you login. For the "pi" user $HOME will be "/home/pi" and for me it will be "/home/foo" because I have created a "foo" user. So.. you'd..

Code: Select all

$ mkdir -p ~/usr/src/ffmpeg
$ cd ~/usr/src/ffmpeg
..and download (say) "ffmpeg-snapshot.tar.bz2"..

Code: Select all

$ wget -c https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2
..and create a script like the one I posted in that folder. I call mine "go" for historic reasons (nothing to do with the Go language) but lets say you call your "z" (I'm good inventing names)..

Code: Select all

foo@sdu ~/usr/src/ffmpeg $ cat z
#!/bin/bash

: ${PFX:="/usr/local/junk"}
: ${SRC:="ffmpeg"}
OBJ="obj-""$SRC"
CFG="
--prefix=$PFX
--disable-x86asm
--enable-static
--disable-shared
"

tar xvjf "$SRC"".tar.bz2"

[ -d "$PFX" ] || {
	sudo mkdir "$PFX"
	sudo chown `id -un`:`id -gn` "$PFX"
	chmod 755 "$PFX"
}

mkdir -p "$OBJ" log || exit 1

(
 cd "$OBJ"
 ../"$SRC"/configure --help 2>&1 | tee ../log/hlp.log
 ../"$SRC"/configure $CFG 2>&1 | tee ../log/cfg.log
 [ ${PIPESTATUS[0]} -eq 0 ] || return 1
) || exit 1

(
 cd "$OBJ" || exit 1
 make -j`nproc` 2>&1 | tee ../log/mak.log
 [ ${PIPESTATUS[0]} -eq 0 ] || return 1
) || exit 1

(
 cd "$OBJ" || exit 1
 make install 2>&1 | tee ../log/ins.log
 [ ${PIPESTATUS[0]} -eq 0 ] || return 1
) || exit 1

echo rm -r "$OBJ" "$SRC"
echo rm -r "$PFX"

Code: Select all

$ chmod u+x z

foo@sdu ~/usr/src/ffmpeg $ ls -l
total 11336
-rw-r--r-- 1 foo foo 11602745 Sep 16 17:24 ffmpeg-snapshot.tar.bz2
-rwxr--r-- 1 foo foo      753 Sep 17 08:15 z

Ordinarily it would work. The convention is "thing-version.tarball.compression" (eg: gcc-11.2.0.tar.xz with the expectation it'll unpack to "./gcc-11.2.0") but as this is a snapshot there is no version number, it unpacks to "./ffmpeg" so 'z' will fail..

Code: Select all

foo@sdu ~/usr/src/ffmpeg $ ./z
tar (child): ffmpeg.tar.bz2: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
./z: line 25: ../ffmpeg/configure: No such file or directory
./z: line 26: ../ffmpeg/configure: No such file or directory
./z: line 27: return: can only `return' from a function or sourced script
..this isn't "our" problem: 'z' should work for most sources. Answer is fudge it with a symlink..

Code: Select all

foo@sdu ~/usr/src/ffmpeg $ rm -r log obj-ffmpeg/
foo@sdu ~/usr/src/ffmpeg $ ln -s ffmpeg-snapshot.tar.bz2 ffmpeg.tar.bz2 
foo@sdu ~/usr/src/ffmpeg $ ls -l
total 11336
-rw-r--r-- 1 foo foo 11602745 Sep 16 17:24 ffmpeg-snapshot.tar.bz2
lrwxrwxrwx 1 foo foo       23 Sep 17 13:36 ffmpeg.tar.bz2 -> ffmpeg-snapshot.tar.bz2
-rwxr--r-- 1 foo foo      753 Sep 17 08:15 z
Now it should compile..

Code: Select all

$ ./z
[snip build output]

foo@sdu ~/usr/src/ffmpeg $ ls -l
total 11348
drwxr-xr-x 17 foo foo     4096 Sep 16 17:24 ffmpeg
-rw-r--r--  1 foo foo 11602745 Sep 16 17:24 ffmpeg-snapshot.tar.bz2
lrwxrwxrwx  1 foo foo       23 Sep 17 13:36 ffmpeg.tar.bz2 -> ffmpeg-snapshot.tar.bz2
drwxr-xr-x  2 foo foo     4096 Sep 17 13:37 log
drwxr-xr-x 13 foo foo     4096 Sep 17 13:37 obj-ffmpeg
-rwxr--r--  1 foo foo      753 Sep 17 08:15 z

foo@sdu ~/usr/src/ffmpeg $ ls -l /usr/local/junk/
total 16
drwxr-xr-x 2 foo foo 4096 Sep 17 13:37 bin
drwxr-xr-x 9 foo foo 4096 Sep 17 08:16 include
drwxr-xr-x 3 foo foo 4096 Sep 17 13:38 lib
drwxr-xr-x 5 foo foo 4096 Sep 17 08:16 share

foo@sdu ~/usr/src/ffmpeg $ ls -l /usr/local/junk/bin/
total 55112
-rwxr-xr-x 1 foo foo 18890520 Sep 17 13:37 ffmpeg
-rwxr-xr-x 1 foo foo 18759704 Sep 17 13:37 ffplay
-rwxr-xr-x 1 foo foo 18780056 Sep 17 13:37 ffprobe

foo@sdu ~/usr/src/ffmpeg $ /usr/local/junk/bin/ffmpeg -h 2>&1 | head -5
ffmpeg version N-103646-g8f92a1862a Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.12) 20160609
  configuration: --prefix=/usr/local/junk --disable-x86asm --enable-static --disable-shared
  libavutil      57.  5.100 / 57.  5.100
  libavcodec     59.  7.103 / 59.  7.103
Now if I clean that up and copy it to an rpi (I didn't have one available earlier and the PC is faster)..

Code: Select all

foo@sdu ~/usr/src/ffmpeg $ rm -r ffmpeg obj-ffmpeg/ log/
foo@sdu ~/usr/src/ffmpeg $ pushd ..
foo@sdu ~/usr/src $ scp -r ffmpeg/ foo@pilap:~/usr/src/
foo@sdu ~/usr/src $ popd
~/usr/src/ffmpeg
foo@sdu ~/usr/src/ffmpeg $
It should compile there..

Code: Select all

foo@pi18:~ $ cd usr/src/ffmpeg/
foo@pi18:~/usr/src/ffmpeg $ ./z
[snip]
LD	ffprobe_g
/usr/bin/ld: libavformat/libavformat.a(fifo.o): in function `fifo_init':
/home/foo/usr/src/ffmpeg/obj-ffmpeg/src/libavformat/fifo.c:519: undefined reference to `__atomic_store_8'
[snip]
..and it's failed. My bad for leaving in an x86 option and trying to build without shared libs. Repeat with CFG in "z" back to allowing shared libs..

Code: Select all

CFG="
--prefix=$PFX
"
..repeat..

Code: Select all

foo@pi18:~/usr/src/ffmpeg $ rm -r obj-ffmpeg/
foo@pi18:~/usr/src/ffmpeg $ ./z
^^^drat still failed. Now I'll have to engage a brain cell or google. Hmm https://stackoverflow.com/questions/623 ... -to-atomic. I'll just add "--extra-ldflags="-latomic""..

Code: Select all

CFG="
--prefix=$PFX
--extra-ldflags=\"-latomic\"
"
..and repeat. Hurrah!

Code: Select all

foo@pi18:~/usr/src/ffmpeg $ ls -l /usr/local/junk/
total 16
drwxr-xr-x 2 foo foo 4096 Sep 17 14:42 bin
drwxr-xr-x 9 foo foo 4096 Sep 17 14:42 include
drwxr-xr-x 3 foo foo 4096 Sep 17 14:42 lib
drwxr-xr-x 4 foo foo 4096 Sep 17 14:42 share

foo@pi18:~/usr/src/ffmpeg $ ls -l /usr/local/junk/bin/
total 49872
-rwxr-xr-x 1 foo foo 17091024 Sep 17 14:42 ffmpeg
-rwxr-xr-x 1 foo foo 16976448 Sep 17 14:42 ffplay
-rwxr-xr-x 1 foo foo 16995808 Sep 17 14:42 ffprobe

foo@pi18:~/usr/src/ffmpeg $ LD_LIBRARY_PATH=/usr/local/junk/lib /usr/local/junk/bin/ffmpeg -h 2>&1 | head -5
ffmpeg version N-103646-g8f92a1862a Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --prefix=/usr/local/junk --extra-ldflags='"-latomic"'
  libavutil      57.  5.100 / 57.  5.100
  libavcodec     59.  7.103 / 59.  7.103
Hang on..

Code: Select all

foo@pi18:~/usr/src/ffmpeg $ ls -l /usr/local/junk/lib/
total 181188
-rw-r--r-- 1 foo foo 109536734 Sep 17 14:42 libavcodec.a
-rw-r--r-- 1 foo foo   1714354 Sep 17 14:42 libavdevice.a
-rw-r--r-- 1 foo foo  26263418 Sep 17 14:42 libavfilter.a
-rw-r--r-- 1 foo foo  40030380 Sep 17 14:42 libavformat.a
-rw-r--r-- 1 foo foo   3615660 Sep 17 14:42 libavutil.a
-rw-r--r-- 1 foo foo    454348 Sep 17 14:42 libswresample.a
-rw-r--r-- 1 foo foo   3907370 Sep 17 14:42 libswscale.a
drwxr-xr-x 2 foo foo      4096 Sep 17 14:42 pkgconfig
..no shared libs have been built in which case LD_LIBRARY_PATH not required..

Code: Select all

foo@pi18:~/usr/src/ffmpeg $ /usr/local/junk/bin/ffmpeg -h 2>&1 | head -5
ffmpeg version N-103646-g8f92a1862a Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --prefix=/usr/local/junk --extra-ldflags='"-latomic"'
  libavutil      57.  5.100 / 57.  5.100
  libavcodec     59.  7.103 / 59.  7.103
When you finally have things as you want, either leave it in /usr/local/junk/ or change PFX to /usr/local/ and use "sudo make install". I realise it's complex. There's a lot to take in but note the actual "z" script itself is fairly trivial - once you understand the nuances of what it does! ;-)

Return to “Beginners”