urequests POST problems on Pico W

[danfascia]

Hi there

I've just received my Pico W this week and want to setup what I think is probably a very common use case scenario for this board but I'm really struggling with HTTP POST requests.

I'm trying to periodically send a POST request to an API which I have made to receive and log the data so that I can remotely monitor various captured data. At this stage I'm literally just trying to make it regularly ping the server.

The Pico W successfully pings the endpoint and logs that it has done so but the HTTP statuscode returned is 426 which is something to do with "Needs Upgrade" - which I do not understand.

When I test the endpoint (A simple FastAPI test server at this stage) from curl or any other HTTP request generator it works fine and increments the counter on the API (which I can monitor by visiting https://results.up.railway.app/pico in browser.

I've also tested other API POST endpoints and they all return some flavour of 4xx error.

What is wrong with the requests generated by the PicoW with urequests? I'm struggling to work it out.

Code: Select all

import time, network, machine
import urequests as request

# Boot sequence to show it is on
led = machine.Pin("LED", machine.Pin.OUT)
led.on()
time.sleep(3) # LED on for 3-secs on boot
led.off()

# Connect to WLAN
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect('xxxx', 'xxxx')

# Wait for connect or fail
max_wait = 10
while max_wait > 0:
    if wlan.status() < 0 or wlan.status() >= 3:
        break
    max_wait -= 1
    print('waiting for connection...')
    time.sleep(1)
 
# Handle connection error
if wlan.status() != 3:
    raise RuntimeError('network connection failed')
else:
    print('connected')
    status = wlan.ifconfig()
    print( 'ip = ' + status[0] )
    led.toggle()
    time.sleep(0.1)
    led.toggle()
    time.sleep(0.1)
    led.toggle()
    time.sleep(0.1)
    led.off()
    
# Callback function to ping server
def ping_fastapi(timer):
    timestamp = machine.RTC().datetime()
    timestring="%04d-%02d-%02d %02d:%02d:%02d" %(timestamp[0:3] + timestamp[4:7])
    r = request.post("http://results.up.railway.app/pico/")
    print(timestring + " Status Code:" + str(r.status_code) )
    led.on()
    time.sleep(0.2)
    led.off()
    r.close()
    pass
    
tim = machine.Timer()
tim.init(period=10000, mode=machine.Timer.PERIODIC, callback=ping_fastapi)

[memjr]

Any reason your code is using httP instead of httpS?

2022-08-19_20-27.png (22.48 KiB) Viewed 3088 times

The server is returning an http 426 telling you to upgrade your protocol from http to https. Because you're not expecting that, your code is not looking for that status code so it can go "oh, ok, let me call you at https... instead".
.
Switch your pico code to reach out to https and it'll probably work

[danfascia]

I tried both http and https and it is no different, the version of the code I posted was my attempt with http since https didn’t work.

In both instances it returns 426.

I’ve since tried sending requests to pipedream’s simple webhook endpoint with no issue. It works.

It’s something to do with servers that protect against dodgy looking requests by discriminating headers - and urequests is not satisfying their needs.

The problem is that a lot of commercial API endpoints seem to behave this way.

[incognitum]

Reason is because urequests only does the ancient HTTP/1.0 protocol, and your server expects everyone to speak at least HTTP/1.1 at a minimum.

Easy to reproduce with curl.
If you force HTTP 1.0 it will result in 426.

Code: Select all

$ curl --http1.0 -v -X POST https://results.up.railway.app/pico/test
*   Trying 104.196.232.237:443...
* TCP_NODELAY set
* Connected to results.up.railway.app (104.196.232.237) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.up.railway.app
*  start date: Jun  3 07:59:40 2022 GMT
*  expire date: Sep  1 07:59:39 2022 GMT
*  subjectAltName: host "results.up.railway.app" matched cert's "*.up.railway.app"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> POST /pico/test HTTP/1.0
> Host: results.up.railway.app
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 426 Upgrade Required
< date: Sun, 21 Aug 2022 13:40:34 GMT
< server: railway
< connection: close
< content-length: 0
< 
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):

The server does accepts POSTs with HTTP/1.1.
(Does return 404 in this case, but that's because I don't know what exact end-point your web service is expecting)

Code: Select all

$ curl --http1.1 -v -X POST https://results.up.railway.app/pico/test
*   Trying 104.196.232.237:443...
* TCP_NODELAY set
* Connected to results.up.railway.app (104.196.232.237) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.up.railway.app
*  start date: Jun  3 07:59:40 2022 GMT
*  expire date: Sep  1 07:59:39 2022 GMT
*  subjectAltName: host "results.up.railway.app" matched cert's "*.up.railway.app"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> POST /pico/test HTTP/1.1
> Host: results.up.railway.app
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found

[ejolson]

Reason is because urequests only does the ancient HTTP/1.0 protocol, and your server expects everyone to speak at least HTTP/1.1 at a minimum.

Easy to reproduce with curl.
If you force HTTP 1.0 it will result in 426.

Code: Select all

$ curl --http1.0 -v -X POST https://results.up.railway.app/pico/test
*   Trying 104.196.232.237:443...
* TCP_NODELAY set
* Connected to results.up.railway.app (104.196.232.237) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.up.railway.app
*  start date: Jun  3 07:59:40 2022 GMT
*  expire date: Sep  1 07:59:39 2022 GMT
*  subjectAltName: host "results.up.railway.app" matched cert's "*.up.railway.app"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> POST /pico/test HTTP/1.0
> Host: results.up.railway.app
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 426 Upgrade Required
< date: Sun, 21 Aug 2022 13:40:34 GMT
< server: railway
< connection: close
< content-length: 0
< 
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):

The server does accepts POSTs with HTTP/1.1.
(Does return 404 in this case, but that's because I don't know what exact end-point your web service is expecting)

Code: Select all

$ curl --http1.1 -v -X POST https://results.up.railway.app/pico/test
*   Trying 104.196.232.237:443...
* TCP_NODELAY set
* Connected to results.up.railway.app (104.196.232.237) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.up.railway.app
*  start date: Jun  3 07:59:40 2022 GMT
*  expire date: Sep  1 07:59:39 2022 GMT
*  subjectAltName: host "results.up.railway.app" matched cert's "*.up.railway.app"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> POST /pico/test HTTP/1.1
> Host: results.up.railway.app
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found

So instead of urequests is there a library that will speak a modern version of TLS when connecting?

[incognitum]

So instead of urequests is there a library that will speak a modern version of TLS when connecting?

This is NOT about TLS version, but about the HTTP protocol version.
(TLS sockets are provided by Micropython, so all HTTP libraries will use the same, and if there was any problem with TLS it would require patching and recompiling Micropython instead.)

Don't know about proper HTTP libraries for MicroPython.
But sending "POST /endpoint HTTP/1.1" (+ HTTP request headers + the content you want to POST) manually through a socket isn't rocket science, so you may want to consider writing your own.

[ejolson]

So instead of urequests is there a library that will speak a modern version of TLS when connecting?

This is NOT about TLS version, but about the HTTP protocol version.
(TLS sockets are provided by Micropython, so all HTTP libraries will use the same, and if there was any problem with TLS it would require patching and recompiling Micropython instead.)

Don't know about proper HTTP libraries for MicroPython.
But sending "POST /endpoint HTTP/1.1" (+ HTTP request headers + the content you want to POST) manually through a socket isn't rocket science, so you may want to consider writing your own.

Thanks for the clarification. It seems this problem should be solvable without any rockets at all.

[memjr]

Can you send the request and print out the entire response and post it here? The responses are usually descriptive enough.

I know it is not what you asked for, but I think reading this might help you with some troubleshooting ideas too. For example it shows why you might get an http401 but is perfectly fine because that is what's supposed to happen using basich auth.

Read this https://docs.python.org/3/howto/urllib2.html
Then read this (which is linked to from the one above) https://web.archive.org/web/20201215133 ... tion.shtml

Knowing that back and forth flow will help deal with more things you're likely to run into when coding things for the web without having access to a full browser or another tool like curl.

[incognitum]

Can you send the request and print out the entire response and post it here? The responses are usually descriptive enough.

Note that you can see in my curl example, that this server sends 426 Upgrade required, with an EMPTY response body ("content-length: 0") if the older HTTP/1.0 protocol version is used.

And yes, urequests is using HTTP/1.0.
https://github.com/micropython/micropyt ... sts.py#L94

So while there may be other cases in which other servers report more detailed information there, the response body is unlikely to be helpful in this case.

[memjr]

Can you send the request and print out the entire response and post it here? The responses are usually descriptive enough.

Note that you can see in my curl example, that this server sends 426 Upgrade required, with an EMPTY response body ("content-length: 0") if the older HTTP/1.0 protocol version is used.

And yes, urequests is using HTTP/1.0.
https://github.com/micropython/micropyt ... sts.py#L94

So while there may be other cases in which other servers report more detailed information there, the response body is unlikely to be helpful in this case.

Right, but that's curl. I was wondering if the Pico was getting something different/extra back, because servers can be coded to send customized responses back for different clients (IE, Chrome, etc). The meaning of the 426 is the same no mater what, but the response may contain more/less info. Just a check anyway.