Making an outbreak statistics screen?

[nautilus27]

Is it possible to make a info screen that grabs recent statistics of current covid-19 outbreak from any website that shows this information? Like total cases, death and recovery counts.

I've never done such a thing before and suck at web based projects so am asking this whether these numbers are accessible. There are many pages out there and each update fairly regularly. Such as John Hopkins University's website, Bing Covid Tracker and Google's Covid-19 map.

https://gisanddata.maps.arcgis.com/apps ... 7b48e9ecf6
https://www.bing.com/covid
https://google.com/covid19-map/

I was thinking of using a 7" official Pi screen and I can code and app that shows these numbers on a full screen GUI app and update it regularly. It's just web site of things i'm fairly inexperienced in.

[MrYsLab]

Here is a nice article on web scraping that may be of use.

[nautilus27]

Here is a nice article on web scraping that may be of use.

Thank you very much, I'm diving in!

[hippy]

Some of the web pages reporting Covid-19 status are quite huge, packed with formatting mark-up etc. Those could consume an excess of data usage if downloaded regularly.

My 'go to' has been - https://www.worldometers.info/coronavirus

That's a 1MB page but is a lot easier to scrape than some dashboards. Using 'mobile' versions of pages will probably help rather than desktop versions.

Some sort of 'raw database' would be best; text, CSV, JSON, XML. But I didn't find anything suitable when I had a look myself, though I haven't looked recently.

There are datasets such as below but they only update daily. Not so good if you want a real time tracker, counter or grapher.

https://www.ecdc.europa.eu/en/publicati ... -worldwide

[nautilus27]

OK, so here's an update.

Code: Select all

import requests
from bs4 import BeautifulSoup

URL = 'https://www.worldometers.info/coronavirus/'

page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')

results = soup.find(id='maincounter-number')

print(results.prettify())

I'm using Chrome browser btw. I followed the web scraper guide, right clicked the total cases number on the page, clicked "Inspect" on the popup menu and then copy & pasted what was inside <div class = " ">. Which was "maincounter-number". When I execute the code above, I get this result:

Code: Select all

Traceback (most recent call last):
  File "scrape.py", line 12, in <module>
    print(results.prettify())
AttributeError: 'NoneType' object has no attribute 'prettify'

[MrYsLab]

The exception is because the result contains None. I am by no means a beautiful soup, HTML or CSS expert, but I tried the following and it returned the 3 main numbers.

Try the following code and see if it gets you any closer. Learning how to parse the HTML tree in soup is somewhat confusing and overwhelming .

Code: Select all

import requests
from bs4 import BeautifulSoup

URL = 'https://www.worldometers.info/coronavirus/'

page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')

results = soup.findAll("div", {"class": "maincounter-number"})

for result in results:
    print(result.prettify())
    
    

[nautilus27]

I found a beautifully scraped and parsed info from this guide: https://towardsdatascience.com/coronavi ... 14b5652088

He does it on Windows and uses push notifications. I replaced the Toast Notifier part with "print()" and successfully saw the info in the terminal. But he combined all the values into a string in his code, like an Array in C++ and i could not make those values recorded individually as variables.

If only this was C++ and Arduino, I'd ace it... Python is a new beast to tackle.

Code: Select all

#For informations check : https://towardsdatascience.com/coronavirus-track-coronavirus-in-your-country-by-displaying-notification-c914b5652088

from win10toast import ToastNotifier
from bs4 import BeautifulSoup
import requests
import time

country = "Tunisia"
notification_duration = 10
refresh_time = 10 #minutes
data_check= []
worldmetersLink = "https://www.worldometers.info/coronavirus/"

def data_cleanup(array):
    L = []
    for i in array:
        i = i.replace("+","")
        i = i.replace("-","")
        i = i.replace(",",".")
        if i == "":
            i = "0"
        L.append(i.strip())
    return L

while True:
    try:
        html_page = requests.get(worldmetersLink)
    except requests.exceptions.RequestException as e: 
        print (e)
        continue
    bs = BeautifulSoup(html_page.content, 'html.parser')

    search = bs.select("div tbody tr td")
    start = -1
    for i in range(len(search)):
        if search[i].get_text().find(country) !=-1:
            start = i
            break
    data = []
    for i in range(1,8):
        try:
            data = data + [search[start+i].get_text()]
        except:
            data = data + ["0"]
    
    data= data_cleanup(data)
    message = "Total infected = {}, New Case = {}, Total Deaths = {}, New Deaths = {}, Recovred = {}, Active Case = {}, Serious Critical = {}".format(*data)

    
    if data_check != data:
        data_check = data
        toaster = ToastNotifier()
        toaster.show_toast("Coronavirus {}".format(country) , message, duration = notification_duration , icon_path ="icon.ico")
    else:
        time.sleep(refresh_time*60)
        continue

[MrYsLab]

Very nice article. Thanks for the link. If you are going to use the 7" display you can get notifications in Raspberry Pi.
I just tried this and it works. Here are the steps I used:

Open a command window and type:

Code: Select all

sudo apt-get install mate-notification-daemon mate-notification-daemon-common

Next type:

Code: Select all

sudo apt-get install libnotify-bin notification-daemon dbus

To test if it works type:

Code: Select all

notify-send "Urgent Message" "Something quite pressing..."

You should see a notify pop-up appear.

I modified your program somewhat to use the notifier. Here is the code:

Code: Select all

# from win10toast import ToastNotifier
from bs4 import BeautifulSoup
import requests
import time
import subprocess 

country = "USA"
notification_duration = 10
refresh_time = 10  # minutes
data_check = []
worldmetersLink = "https://www.worldometers.info/coronavirus/"


def data_cleanup(array):
    L = []
    for i in array:
        i = i.replace("+", "")
        i = i.replace("-", "")
        i = i.replace(",", ".")
        if i == "":
            i = "0"
        L.append(i.strip())
    return L


while True:
    try:
        html_page = requests.get(worldmetersLink)
    except requests.exceptions.RequestException as e:
        print(e)
        continue
    bs = BeautifulSoup(html_page.content, 'html.parser')

    search = bs.select("div tbody tr td")
    start = -1
    for i in range(len(search)):
        if search[i].get_text().find(country) != -1:
            start = i
            break
    data = []
    for i in range(1, 8):
        try:
            data = data + [search[start + i].get_text()]
        except:
            data = data + ["0"]

    data = data_cleanup(data)
    message = "Total infected = {}, New Case = {}, Total Deaths = {}, New Deaths = {}, Recovred = {}, Active Case = {}, Serious Critical = {}".format(
        *data)
    print(message)
    subprocess.run(["notify-send", "urgent", message])

    # if data_check != data:
    #     data_check = data
        # toaster = ToastNotifier()
        # toaster.show_toast("Coronavirus {}".format(country), message, duration=notification_duration,
        #                    icon_path="icon.ico")
    # else:
    #     time.sleep(refresh_time * 60)
    #     continue
    print('done')