Darktrax
Posts: 17
Joined: Sun Nov 30, 2014 1:40 pm

Copy-Paste? What you see is NOT what you get!

Sat Jan 03, 2015 12:11 am

This is a difficult thing to describe since copy-paste is needed to show examples, but I will try.
The Pi is like any other, with Raspbian downloaded last month.
The problem came when using copy-paste with using LeafPad (or Nano) to make a simple bash script to try out the vcgencmd commands to explore the Pi.
I was following the examples from..
HERE --> http://www.elinux.org/RPI_vcgencmd_usage
It went badly wrong. First I tried making a simple bash script ..

Code: Select all

#!/bin/bash
echo Bash script to try out vcgencmd commands
#
for src in arm core h264 isp v3d uart pwm emmc pixel vec hdmi dpi ; do \
echo -e "$src:\t$(vcgencmd measure_clock $src)" ; \
done
The result was ..

Code: Select all

pi@PiBox ~ $ ./bash-script1.sh 
Bash script to try out vcgencmd commands
./bash-script1.sh: line 4: $'for\302\240src\302\240in\302\240arm\302\240core\302\240h264\302\240isp\302\240v3d\302\240uart\302\240pwm\302\240emmc\302\240pixel\302\240vec\302\240hdmi\302\240dpi\302\240': command not found
./bash-script1.sh: line 5: $'vcgencmd\302\240measure_clock\302\240': command not found
./bash-script1.sh: line 5: $'\302\240do\302\240echo\302\240-e\302\240:\\t\302\240': command not found
./bash-script1.sh: line 6: $'\302\240done': command not found
I tried a more determined attempt using nano. This also failed.
A copy-paste from NetSurf straight into a terminal will not work, but you can first copy it into LeafPad.
The a middle-click will deliver the line - BUT it only looks OK.
As you press <Return>, you discover there were non-display characters in there.
The only way to get it to work was to deliberately type in the list direct into the editor.
It seems the SPACE characters and LINE FEEDS are the problem.
If this is something simple - like I have a strange locale setup or something like that, then do tell.
Any help would be much appreciated.

User avatar
DougieLawson
Posts: 41900
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Copy-Paste? What you see is NOT what you get!

Sat Jan 03, 2015 12:32 am

You've converted the script from unix format to dos format (probably by cutting'n'pasting on Windows or using notepad).

sudo apt-get install dos2unix
dos2unix foo.sh
./foo.sh
Languages using left-hand whitespace for syntax are ridiculous

DMs sent on Twitter/LinkedIn will be answered next month.
Fake doctors - are all on my foes list.

The use of crystal balls and mind reading is prohibited.

User avatar
topguy
Posts: 7240
Joined: Tue Oct 09, 2012 11:46 am
Location: Trondheim, Norway

Re: Copy-Paste? What you see is NOT what you get!

Sat Jan 03, 2015 12:39 am

It looks more like a UNICODE / UTF-8 character set problem. \302\240 is in hex 0xc2 0xA0 which is in this table http://www.utf8-chartable.de/ mean NO_BREAK_SPACE.

I'm not sure if Dougies suggestions fixes these things. It could be the OS/Webbrowser combination that creates this.

Darktrax
Posts: 17
Joined: Sun Nov 30, 2014 1:40 pm

Re: Copy-Paste? What you see is NOT what you get!

Sat Jan 03, 2015 1:18 am

My thanks for the reply.
Yes, the numbers in there might indeed be associated with CR and LF escape codes, but I am not completely sure. It begs the question about spaces. I thought a SPACE character ASCII is the same in DOS as well as UNIX formats. The only difference I am aware of is that DOS format texts deliver a newline using CR+LF whereas UNIX format I think uses only line-feed (LF).

The text, open in LeafPad of course does not show the DOS <carriage-return> nor the <line-feed> codes, so gives no indication that the file contains DOS escape characters. I had thought that using SAVE in LeafPad would save in UNIX format OK

Unfortunately, they seem to survive the SAVE when one expects the seen text to be saved in UNIX format .
I found that by carefully going through the whole script, using <backspace> to back out every space, and every new line, and then re-inserting them with the space-bar and <Return> key, I made a file which when saved, would finally run OK.

The Pi comes with the (lightweight) NetSurf, Epiphany Browser, Web 3.8.2 and the somewhat limited Dillo. They would be displaying page texts that might well have DOS escape characters. I have never noticed before that copy-paste into a UNIX editor would do this - except in the Pi.

Maybe the page they came from http://elinux.org/RPI_vcgencmd_usage has the text in some special frame or something

Darktrax
Posts: 17
Joined: Sun Nov 30, 2014 1:40 pm

Re: Copy-Paste? What you see is NOT what you get!

Sat Jan 03, 2015 1:41 am

Thanks also topguy
topguy wrote:It looks more like a UNICODE / UTF-8 character set problem. \302\240 is in hex 0xc2 0xA0 which is in this table http://www.utf8-chartable.de/ mean NO_BREAK_SPACE.
Being in the UK, I used (as root) raspi-config to set the Locale to en_GB.UTF-8 UTF-8, which is the recommended choice for new installations.

There is a problem properly selecting text out of a browser (NetSurf) anyway, and it has to be pasted into an editor (LeafPad) before one can move it anywhere else. I admit it felt strange replacing spaces with new spaces when I was fixing the shell script, but I just kept running it until I had discovered every "unwanted hidden extra". The command outputs end up neatly formatted with TABs because of the \t.

User avatar
joan
Posts: 15787
Joined: Thu Jul 05, 2012 5:09 pm
Location: UK

Re: Copy-Paste? What you see is NOT what you get!

Sat Jan 03, 2015 10:44 am

On *nix a fix is

tr -d "\240\302" <file >file.new

Should copy file to file.new and remove those unneeded characters.

User avatar
rpdom
Posts: 19283
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Copy-Paste? What you see is NOT what you get!

Sat Jan 03, 2015 1:43 pm

joan wrote:On *nix a fix is

tr -d "\240\302" <file >file.new

Should copy file to file.new and remove those unneeded characters.
Problem there is that those characters are representing spaces which should be there. Perhaps something like

sed "s/\240\302/ /g" <file >file.new

(which I haven't time to check during my lunch break)

Darktrax
Posts: 17
Joined: Sun Nov 30, 2014 1:40 pm

Re: Copy-Paste? What you see is NOT what you get!

Sat Jan 03, 2015 5:35 pm

joan wrote:On *nix a fix is
tr -d "\240\302" <file >file.new
Should copy file to file.new and remove those unneeded characters.
Thanks Joan - indeed it does.
We can understand that that most browsers have to be able to recognise and handle all the many flavours of EOL newline combinations, but this is not true of any and every editor. A good file offered into Notepad will still end up all on one long long line!

When the editor has to save a script or program, the newline becomes very significant.
I found the full explanation HERE --> https://en.wikipedia.org/wiki/Newline. It gets quite interesting.
Even if a program supports different newline conventions, these features are often not sufficiently labeled, described, or documented. Typically a menu or combo-box enumerating different newline conventions will be displayed to the user without an indication if the selection will re-interpret, temporarily convert, or permanently convert the newlines. Some programs will implicitly convert on open, copy, paste, or save - often inconsistently.
If it was only the End-of Line incompatibility hazard, I can understand it. The incompatibility surely does not extend to spaces! I am thinking that maybe stuff copied from the original site http://www.elinux.org/RPI_vcgencmd_usage from out of those grey boxes came with some baggage, or possibly just with baggage that is not handled well on a Pi running NetSurf and LeafPad. For anyone who wants to make an executable file, it would be a kindness not to have the editor automatically bury the unwanteds. Some kind of warning mark would be nice.

Return to “Troubleshooting”