Article Index - new topic, old question

[jreebel]

I'm baffled by the lack of an official online index for MagPi articles. Looked at the old posts, saw that some folks even started keying in the data but apparently nobody at the magazine picked up the ball to run with it. For an organization based on promoting computing and coding this should be child's play. A simple database with just columns for article name, issue number and date would be invaluable. You could throw in page number and keywords as well.

I'm speaking from experience here, as I was a developer, mainly database programming, for 35 years. This could be developed in a week and data added for each new issue in an hour. Get an intern to start entering data from back issues and Bob's your uncle. If that's too much to ask for, how about a downloadable csv file updated with each new issue?

Many magazines that are not tech related already have this type of feature, so why not MagPi? Kind of embarrassing, really. This came to me as I laboriously looked thru old issues trying to find articles on the Pico W that I just purchased. Frustrating!

[neilgl]

Could you do a prototype by downloading all PDFs of the MagPi, extract the contents pages and insert into a database with issue number, date etc.?

[hippy]

Could you do a prototype by downloading all PDFs of the MagPi, extract the contents pages and insert into a database with issue number, date etc.?

I have that and it works, at least gives something better than nothing.

I do it as a multi-step process -

• Download the PDF files using the Bookshelf App
• Convert each new PDF to '.raw' text files using the command line 'pdftotext'
• Process '.raw' to get a '.lst' of potential index text lines
• Process '.lst' to get a '.txt' index for each PDF
• Collate all '.txt' to be a complete index.

So, for example, the latest MagPi 127 ...

Code: Select all

-rw-r--r-- 1 pi pi 35261900 Mar  4 18:19 MagPi127.pdf
-rw-r--r-- 1 pi pi   174552 Mar  4 18:22 MagPi127.raw
-rw-r--r-- 1 pi pi     3206 Mar  4 18:22 MagPi127.lst
-rw-r--r-- 1 pi pi      505 Mar  4 18:22 MagPi127.txt

And from that 35 MB I arrive at this for the contents, MagPi127.txt ...

Code: Select all

Case Study: BioBusiness
Desert Eye 2.0
Mini PC
Proto Shield with Robot Arm
Rudolph the Red Nose Reindeer
Raspberry Pi Backpack
Alternative Flight Simulator
Geiger Counter
The MagPi magazine is published monthly by Raspberry Pi Ltd. Publishers Service Associates, 2406 Reach Road, Williamsport, PA,
POSTMASTER: Send address changes to The MagPi magazine c/o Publishers Service Associates, 2406 Reach Road, Williamsport, PA, 17701.
Raspberry Pi 400
Emulate a BBC Micro
Make a giant Pomodoro timer
PiBoy XRS

It was June 2020 when I wrote that code and last ran it, before just now, so I can't remember what the exact challenges were but you can see it's not perfect from that publication entry.

I think I gave up on it because it was just too difficult to decide what was index what wasn't, what needed culling while not losing things which were legitimate.

I ultimately went for 'Project Index' than full 'Article Index' because I found that overwhelming, too much information, obscuring the things I wanted the index for.

But ultimately it's not that useful. It needs more detail than "Desert Eye 2.0" as you can't guess what that is from its description, at least an intro paragraph or some keywords to determine if it's an expose of some covert cop project which spies on people from the Middle East, about using a Pi with a satellite to monitor wildlife in the Sahara or for recording the expanding consequences of global warming, a webcam project, or even an autonomous vehicle.

But add a description or keywords and it makes the index harder to scan. It's basically no-win.

Some project descriptions are more obvious than others and it's probably fine if you want to find something when you can remember what it was called, but not useful if you want to find that project which, for example, connected a Pi Zero to another Pi to give it access to the internet.

I guess it could be improved if someone were willing to put the effort in but I couldn't justify doing so.

Here's the full index for other MagPi I downloaded so you can get an idea of how good or not I got it - It gets better as it goes on -

Code: Select all

MAGPI ISSUE 10

Welcome to isssue 10,
Thank you to all of those who have ordered volume one (issues 1-8). After
We are very grateful for your continued patience. Such delays are not expected
In this month's issue there is a great selection of hardware and software
We are on the look out for willing volunteers to help with layout, testing and
If you can dedicate some time, please email
Chief Editor of The MagPi
The MagPi Team
Ash Stone - Chief Editor / Administration
Chris 'tzj' Stagg - Administration
Matt '0ther0judge0' - Website / Administration
Tim 'meltwater' Cox - Administration
Lix - Administration
Aaron Shaw - Page Design / Graphics
Bryan Butler - Page Design & Theme / Graphics
Ian McAlpine - Page Design / Graphics
Isa McKenty - Page Design
Simon Johnson - Page Design
Steve Drew - Page Design
W.H. Bell - Page Design / Administration
Mark Robson - Proof Reading
Michael Beaucage - Graphics
CONTROLLED ROBOT CAM - PART 2
Robot remote control with raspberry Pi REST Framework (WebIOPi)
MONTH'S COMPETITION
Win a 512mb Rasp. Pi model B and case, from PC Supplies LTD
UP - PART 2
Keeping the SD card images safe and restoring backups
Learn some lashup scripts with the Bourne-again shell
Find this month's events in your area
Encouraging others to get coding with the Raspberry Pi
Introducing C++ streams, reading and writing files
PATCH - GPIO CONTROL PART 2
Celebrate the anniversary of the Raspberry Pi with a LEDborg candle
Generate fractal images with Scratch
Using a simple client-server model for parallel calculations

MAGPI ISSUE 31

Issue 31 March 2015
Dr Simon Monk shows us how to build a parking sensor
Martin Minecraft OHanlon shares five of his favourite
Drive through the boxes, it makes the car look good!
Create romantic lighting using pulse-width modulation
Capture incredible DSLR photos with the help of Pi
Join Richard Smedley on the first part of his quest to help
In part one of our new series, Sean M Tracey creates the
RASPBERRY Pi 2
Its priceless and infinitely powerful, at least according to our account of the
Raspberry Pi Foundations new credit-card-sized PC. Heres why
Meet the technologists of tomorrow as
ASTRO Pi
Find out how UK schoolchildren can win the chance to run
March 2015
Coming soon
Keep up to date with the biggest stories in the Pi community
The latest computer books reviewed and rated
Find a community gathering near you in the coming weeks
Matt Richardson shares his earliest Raspberry Pi memories
LEGO-LUTION OF Pi
Les Pounder looks at three of the best emulation applications
Should you control your Pi with a flick of the wrist?
Anonymise your web browsing with this new Pi-powered app
Pi 2S
What would you do with
To celebrate the third birthday of the Raspberry Pi, the Hayler family did
Pi CLUSTER
We catch up with David Guill to see what hes
The MagPi talks to Andy Proctor about his
Raspberry Pi -powered truck
SNES Pi CASE
Mark Parrish shares details of his
Image:
Evan Amos
CC BY-SA 3.0

MAGPI ISSUE 80

Tips & tricks
A Whisper of Moths
E-ink Calendar
Underwater Drone
YogAI
Clapper
Lego Boost
Pygame Zero
Samba file server
Pygame Zero: AmazeBalls pt 3
Comic creator
Karaoke machine
Pictures from space
Breakout Garden HAT
Ham Radio Projects
Zero Dongle
Top 10 HATs
Learn the command line
Claire Pollard interview

MAGPI ISSUE 81

Build a Raspberry
Pi project
Seans Pi robot
GTA: San Andreas Radio Set
Commodore monitor
Chord Assist
Intelli-T Pi Sensor Alarm
Retro games console
Lego Boost
Electronics with GPIO Zero 1.5
Make a retro games console
Party popper launcher
Lego Boost part 2
Start creating GUIs with GTK
Car computer
Breadboard Pi Bridge
PiTalk
Top 10 laptop kits & projects
Learn game development
Nikole Vaughn interview

MAGPI ISSUE 82

Yuri 3 Mars rover
Telepresence Hand
Cat shower
DIY Hardware Password Keeper
Pi VizuWall
Hologram Machine
Yuri 3
Build a Pi keyring
Lego Boost part 3
Display cabinet lights
Make a Marauders Map
Buttons and labels with GTK
Model railway projects
PiBug 2WD robot
LibreELEC 9.0
Top 10 health projects
Learn AI resources
Nicole Parrot interview

MAGPI ISSUE 83

Raspberry Pi 4
TV Wall
Community water sprinkler
PiNG video doorbell
Smart palm greenhouse
Get started with PICO-8
Squeeze controller
Updated QuickStart Guide
Self-healing Raspberry Pi
More advanced GUI layout
Enviro+
Space science!
Learn upcycling
Chris Barnatt interview

MAGPI ISSUE 84

Raspberry Pi
Manual
Smart Home Herb Garden
Marvin Go-Kart
Intelligent Drowsiness Monitor
Smart Window Fan
PIC-20
Learn Lua with PICO-8
LED Racer
Low-cost robot
Use SSH with Raspberry Pi
CircuitPython servos
Swear fine machine
GUI user input with GTK
Pibow Coup 4 and Fan SHIM
Scorching Summer Projects
Keybow MINI
Top 10 projects to upgrade
Learn Sense HAT
Anne Carlill interview

MAGPI ISSUE 85

Raspberry Pi 4
Desktop test
Pi Fighter
Immersive Bar
MyPi
Bird Feeder Monitor
Rocket Rescue
Making sprites in PICO-8
Get started with Mathematica
Build a low-cost robot part 2
Build your own NAS
GUI combo boxes and lists
GPIO music box
Back to school
RockyBorg
Pi2Go Mk 2
LibreELEC on Raspberry Pi 4
NanoSound DAC 2 Pro
Top 10 media players
Learn web design
Denise Leonard interview

MAGPI ISSUE 86

Maker Projects
MudPi
Dride Zero
Stonehenge Skyscape
Magic Wand
Night Vision
Throttle & Tilt
Make music in PICO-8
Interactive tabletop RPG table
Throttle & Tilt Scratch 3 racer
Build a low-cost robot, part 3
The Photon Lab
Tree views with C and GTK
Add game music in PICO-8
VNC with Raspberry Pi
PecanPi
Halloween costume projects!
NanoSTEM IOT Weather Kit
Top 10 gaming accessories
Learn Scratch 3
Sean Raser interview

MAGPI ISSUE 87

Raspberry Pi 4K
Cryptography ICE Cube
UK train departure screen
BBC Box
DNA Gel Imager
Social Media without the Internet
Perpetual Chimes
Make a cluster computer
Create maps &
Build a low-cost robot part 4
Hack GraviTrax
Create maps & levels in PICO-8
GUI menus in C and GTK
FLIRC Raspberry Pi 4
Power up
PiArm
Steam Link
Top 10 AI projects
Learn to code with toys
Alex Mous interview

MAGPI ISSUE 88

Retro computing
ZX Spectrum Next
The Swirl Machine
Astrophotography Autoguider
Smart Christmas lights
Polish your PICO-8 game
Build a low-cost robot
Hack GraviTrax
Retro gaming with Lakka
C and GTK GUI dialogs
GPIO Xmas Tree
Thermal testing update!
Top 10 Christmas projects
Learn computing systems
Liz Clark interview

MAGPI ISSUE 89

Tips & Tools
TIPS & TOOLS
AMOS sailboat
Organnery pipe organ
Airdrum
Cheeseborg
CutiePi
Bit Time Rotary Dial Phone
Aquatic Mini Observation System
Add navigation to a robot
Design a 3D printer pendant
Smart classroom assistant
Vintage computer internet
Hack GraviTrax
C and GTK custom widgets
Add internet to a vintage computer
Raspberry Pi kit & gadget guide
Enviro
SmartiPi Touch 2
Top 10 advanced projects
Learn R with Raspberry Pi
Andrew Suttle interview

MAGPI ISSUE 90

Magic mirror
WeatherMan
DARVA: The Event Assistant
Heart-rate monitor
Audio radar
Tri-Lasers for Felines
Raspberry Pi Recovery Kit
Overclock Raspberry Pi 4
Room guard
Pi Bakery: Hex-A-Pad
C & GTK Glade layout editor
Thermal cases group test
New Year projects!
Pirate Audio
RetroFlag GPi Case
Top 10 game making projects
Learn Python
Pranjali Pathak interview

MAGPI ISSUE 91

Le
NeoPixel LED Mirror
AdventurePi
Reachy
El Carrillon
Bellagio Water Show
Raspberry Pine
Raspberry Shake
Hex-A-Pad
Magic Mirror
C and GTK cheat sheet
NexDock 2
Starter Electronics
RedBoard+
Top 10 wearable projects
Learn SQL
Ben Nuttall interview

MAGPI ISSUE 92

Raspberry Pi
Instaclock
CNC Etch-A-Sketch
Solar-powered Camera
Flying Saucer Clock & NAS
Chamber: Sourdough Incubator
PiVidBox
Tweet with Morse code
Magic mirror
Starter electronics
Raspberry Shake
Simon Safe Crack game
Track weather & pollution
Maker pHAT
SaniaBOX
Top 10 audio projects
Learn retro gaming
Dr Wednaud Ronelus interview

MAGPI ISSUE 93

Work from home
Vineyard Kikushima
Hamster Feeder
DropStitch
Hot Spotter
DIY MIDI Door
RIoT Brick
Get started with the HQ Camera
Sense HAT rainbows
Updated QuickStart guide
Get started with HQ Camera
Magic Mirror
Learn Java and electronics
Polar Etch-a-Sketch
Sense HAT rainbow display
Marble Madness
ZX Spectrum Next
At home with IoT
Automation HAT Mini
Top 10 upcycling projects
Learn JavaScript

MAGPI ISSUE 94

HAL 9000 Google Assistant
Game of Life
Apollo Pi
Smart Doorbell
Giant Battleships
Coffee Maker Greenhouse
BeeMonitor
Raspberry Pi loft bed
Make a 3D camera
Rotary encoders
Java electronics
Make some noise
High Quality Camera Projects
Unicorn HAT Mini
Strato Pi CM Duo
Top 10 robot kits
Learn Python
Caroline Dunn interview

Just looking through that I did see things which made me think "that looks interesting" so may be it does have a use.

[neilgl]

What I meant was - could the OP do it?

[ejolson]

I'm baffled by the lack of an official online index for MagPi articles. Looked at the old posts, saw that some folks even started keying in the data but apparently nobody at the magazine picked up the ball to run with it. For an organization based on promoting computing and coding this should be child's play. A simple database with just columns for article name, issue number and date would be invaluable. You could throw in page number and keywords as well.

I'm speaking from experience here, as I was a developer, mainly database programming, for 35 years. This could be developed in a week and data added for each new issue in an hour. Get an intern to start entering data from back issues and Bob's your uncle. If that's too much to ask for, how about a downloadable csv file updated with each new issue?

Many magazines that are not tech related already have this type of feature, so why not MagPi? Kind of embarrassing, really. This came to me as I laboriously looked thru old issues trying to find articles on the Pico W that I just purchased. Frustrating!

Is pdfgrep too slow to find what you are looking for among to PDF files? This has the advantage of searching the entire text rather than just the headline.

[lucyhattersley]

> Get an intern to start entering data from back issues and Bob's your uncle.

For some back story, we don't have interns on The MagPi (we pay people fairly for work - it's something we're quite determined about). Some people have requested this feature and I think the project should be community-led. All our magazines are freely downloadable and PDF is searchable so personally, I don't find the cost-benefit worth it but I am open to the discussion.

[hippy]

For some back story, we don't have interns on The MagPi (we pay people fairly for work - it's something we're quite determined about).

I have found the term "intern" carries baggage and can conjure up as much disdain and contempt as using "senior" does.

I don't personally consider "interns" to mean only those working for free or being exploited; I see suggesting an intern be brought in more as believing a task appears simple enough that it would be well suited to a student or other work experience placement to undertake, not that they shouldn't be paid fairly for their work.

I don't like companies looking to employ unpaid or low paid interns as a way of saving money but I don't oppose companies taking on people who offer their services for free to gain experience.

Some people have requested this feature and I think the project should be community-led. All our magazines are freely downloadable and PDF is searchable so personally, I don't find the cost-benefit worth it but I am open to the discussion.

By "community-led" do you mean the community should be doing it, providing what is wanted, or should be guiding Raspberry Pi Press or MagPi staff as to what is wanted, what should be delivered, and how ?

BTW : Asking to have a "staff" or appropriate forum badge may make it more obvious who you are for casual readers who may not know.

If I were asked to provide a business case for doing it I would suggest being able to pop-up a better description and article list for each magazine in the Pi Bookshelf App to be a very useful feature.

It could be useful for Google searching where Google hasn't scanned the articles, can't identify the magazines articles are in if those articles haven't been published separately as on-line pages.

It could also be added to MagPi pages - eg https://magpi.raspberrypi.com/issues/127

The contents are already provided in PDF form but having a textual index would suit others better.

For a DIY approach, scanning those may be a better option than searching the downloaded PDF's, and they are certainly smaller in size, but getting at them is more work, adding an extra step of having to scrape the pages to discover the rather unguessable URL -


https://magpi.raspberrypi.com/downloads/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBaFVnIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--e81137c36508ed348f9ced58cae65fef632b610d/005-006_MagPi127_Contents_SAMPLE.pdf

If they could be given predictable page names a DIY approach would be easier.

[hippy]

Is pdfgrep too slow to find what you are looking for among to PDF files?

Is there a easily installable version of 'pdfgrep' for Windows, ideally with a GUI interface, and a one-click 'open that' for any PDF it has found ?

One problem with downloading the PDF's to a Pi is they require GB of SD Card space and it can be incredibly slow to open and browse PDF on older Pi. To get round that I usually download MagPi to my Windows PC where they are easier to store and read and that's where lack of having a local index proves a problem for me.

I created my own 'Pi Bookshelf' App which produces a HTML web page with mouse-over pop-up indexes and clickable links to download or open PDF editions but it wasn't perfect and far from ideal. Haven't used it in ages; I end up searching on Google, then having to check if I have download it already.

There's often no easy way to tell which MagPi may be of interest or relevant to what one's doing other than glance at the cover, go through them, open them up, one after the other.

[tpyo kingg]

I'm baffled by the lack of an official online index for MagPi articles. Looked at the old posts, saw that some folks even started keying in the data but apparently nobody at the magazine picked up the ball to run with it. For an organization based on promoting computing and coding this should be child's play. A simple database with just columns for article name, issue number and date would be invaluable. You could throw in page number and keywords as well.

I'm speaking from experience here, as I was a developer, mainly database programming, for 35 years. This could be developed in a week and data added for each new issue in an hour. Get an intern to start entering data from back issues and Bob's your uncle. If that's too much to ask for, how about a downloadable csv file updated with each new issue?

Many magazines that are not tech related already have this type of feature, so why not MagPi? Kind of embarrassing, really. This came to me as I laboriously looked thru old issues trying to find articles on the Pico W that I just purchased. Frustrating!

I took a look a few years ago. The way the PDFs are set up, are least were up to number 74, there is no way, with the current technology, to automatically extract the titles and other metadata. The only accurate way was re-keying. The least difficult way to re-key was using LibreOffice Calc with the OpenDocument format. It could be exported to SQLite for other uses, but the main editing is easiest in spreadsheet form. Maybe someone can set up an instance of OnlyOffice for collaborative editing.

The metadata: Number (aka issue), page, section, title and standfirst.

The names of each Maker and Author were not collected.

I kept up with it for a while but there was active disinterest in it at the time and stopped after issue 74. You can have that OpenDocument Formatted file if you'd like or I can push it into SQLite later.

[hippy]

I took a look a few years ago. The way the PDFs are set up, are least were up to number 74, there is no way, with the current technology, to automatically extract the titles and other metadata. The only accurate way was re-keying.

That was probably also my conclusion; get as much as one can automatically then go through the determined results, delete what's erroneous, add what's missing, edit to make it appropriate. Too much is easier to delete than having to add missing.

That means going through all the magazines, a laborious slog, but is at least a one off task. Hence why suggested as suited to an 'intern' rather than tying up others whose skills are better used elsewhere.

Once it's done, out on GitHub where people can issue pull requests to correct errors, it's just a once per month effort to copy what's already being placed into the magazine as an update.

Maybe a blank index on GitHub is the way to start - and from your own effort that is over half done - and let the community fill it up. A formal definition is needed for format and it would help to have a mechanism so multiple people aren't wasting efforts on creating the same magazine index as someone else is working on.

[scruss]

Most of the issues are on the Internet archive, which has already done a (slightly ropey) text extraction that might be a start for making an index: Internet Archive: Digital Library of Free & Borrowable Books, Movies, Music & Wayback Machine

[tpyo kingg]

Attached are the aforementioned metadata for numbers 1 through 74.

There are, however, still 53 issues to go to reach number 127, the current issue. The back issues for the MagPi in PDF form are all available from the official site. But it is easiest to re-key with the paper copies at hand.

One way would probably be to set up an instance of OnlyOffice for collaborative editing. As second choice might be to set up a some in Git self-hosting (or maybe at sourcehut, codeberg, gitlab, one of the acceptable Git hosters) and use one tab-delimited text file per issue. That would allow the work to be divided up in a reasonable manner, it could be done in a spreadsheet such as LibreOffice Calc or Calligra Sheets, and at then at the end the files can easily be merged into an SQLite3 table. With an additional table, SQLite3 supports full-text searching too.

[hippy]

I would prefer something text-based, CSV, XML, JSON, INI file format, something which can be manipulated using Windows and Mac tools, easily used cross-platform from Python etc without installing additional libraries. But I guess it's up to whoever takes it on to decide and curation, creation, editing and updating is separate from any deliverables which end-users will be using.

[MH2]

I use a program called BeeBase, which is open source. It can be found here:

https://sourceforge.net/projects/beebase/
https://beebase.sourceforge.io/index.php?page=welcome
https://groups.google.com/g/beebase-bbs?pli=1

[tpyo kingg]

I would prefer something text-based, CSV, XML, JSON, INI file format, something which can be manipulated using Windows and Mac tools, easily used cross-platform from Python etc without installing additional libraries. But I guess it's up to whoever takes it on to decide and curation, creation, editing and updating is separate from any deliverables which end-users will be using.

CSV or tab-delimited is easy to work with in LibreOffice for data entry, and easy to check out and in with git. With one file per number it would be feasible to split up the data entry plus verification. Tab-delimited CSV can then be very, very easily imported into SQLite or manipulated with Python etc. I've tried cataloging a few additional numbers that way and it seems to take about 15 to 30 minutes per number. I expect the first few times it might take 45 to 60 minutes to get comfortable with the structure of the magazine.

Though I am not sure myself of the main feature section. Sometimes it seems monolithic and other times seems to have distinct subsections. As mentioned, from looking at the actual PDFs, the only way is to go through the paper copies or PDFs and re-key, though the autocomplete in LibreOffice can help a lot with author and section names.

[snip]
Some people have requested this feature and I think the project should be community-led. All our magazines are freely downloadable and PDF is searchable so personally, I don't find the cost-benefit worth it but I am open to the discussion.

Would it be acceptable to the MagPi staff if one of us out here were to try setting up a short lived git repository with CSV or Tab-delimited text files and see about further community participation? I'm only marginally familiar with git, but might have enough interest to try configuring such a setup.

[lucyhattersley]

Would it be acceptable to the MagPi staff if one of us out here were to try setting up a short lived git repository

Of course. Please let us know where to find it and we can take a look

[tpyo kingg]

Would it be acceptable to the MagPi staff if one of us out here were to try setting up a short lived git repository

Of course. Please let us know where to find it and we can take a look :-)

Thanks. I've used Git a little but am a total newb in this context. It'll take me a while to manage Git at this level. My repository is certainly not polished yet.

That said, here is what I had in mind tentatively:

https://git.sr.ht/~ldn/The_MagPi_Metadata

I've tried to incorporate the wishes listed in the posts above.

[tpyo kingg]

If that's too much to ask for, how about a downloadable csv file updated with each new issue?

A preliminary draft of the metadata for issues 1 through 128 is available in CSV form for further editing and can be checked out / cloned using Git.

What is needed now is to go through the issues one section at a time to ensure consistency in describing the sections across the issues. For example, check all the book reviews to ensure that the books' authors are included (column e) and not just the reviewers (column g). Or, as another example, go through and make the various sections are the end are described accurately. Or check that none of the events are missing. Spoiler alert -- there are a lot of omissions and, surprise, typos.

[scruss]

Spoiler alert -- there are a lot of omissions and, surprise, typos.

Your loyal subjects expect nothing less, Your Majesty ...

(on other boards I'm on, this would be marked eponysterical: when one's user name and comment collide)

More seriously though - thanks! Much appreciated.