I am studying the urllib3 HTTP library. I am accessing my own servers written in Rust and Python. Both servers implement SSL/HTTPS for localhost using self-signed certificates. As it turns out, we need to disable SSL verification in this scenario. This post documents my attempts to understand this feature of the urllib3 HTTP library.

129-feature-image.png
The Python urllib3 HTTP Library and SSL/HTTPS for localhost

❶ It seems that there are two popular Python HTTP libraries: Requests and urllib3. Under the hood, the former uses the latter. I chose urllib3 as I don’t see Requests offering any significant advantages.

To install the urllib3 library in an active venv virtual environment, run the following command:

▶️Windows 10: .\venv\Scripts\pip.exe install urllib3
▶️Ubuntu 24.04: ./venv/bin/pip install urllib3

❷ Let’s start off with a simple script that retrieves the Google home page.

Content of urllib3_000_get_ssl.py:

import urllib3

http = urllib3.PoolManager()

resp = http.request("GET", "https://google.com")

status = resp.status
print(f"status: {status}\n")
print(f"data: {resp.data.decode('utf-8')}")

Run the script with the following command:

▶️Windows 10: .\venv\Scripts\python.exe urllib3_000_get_ssl.py
▶️Ubuntu 24.04: ./venv/bin/python urllib3_000_get_ssl.py

It should run successfully. The output should be:

status: 200

data: <!doctype html><html itemscope="" 
...
<meta content="Seasonal Holidays 2024" property="twitter:title">
...
<title>Google</title>
...
</body></html>

❸ This article This article describes the SSL/HTTPS implementation for localhost using self-signed certificates for the server written in Rust. 🦀 Here is the index for the complete series. From here on, it is referred to as the Rust server. Similarly, this article describes the Python server. 🐍 Here is the index for the complete series. From here on, it is referred to as the Python server.

🐍 Let’s first try the Python server’s login endpoint at https://localhost:5000/auth/token.

Content of urllib3_001_login_python.py:

import urllib3

http = urllib3.PoolManager()

resp = http.request(
    "POST",
    "https://localhost:5000/auth/token",
    fields={"username": "behai_nguyen@hotmail.com", "password": "password"},
    headers={
        "x-expected-format": "application/json"
    }
)

status = resp.json()

print( f"Status code: {status['status']['code']}" )
print( f"Status text: {status['status']['text']}" )

data = status['data']

print( f"Access Token: {data['access_token']}" )
print( f"Detail: {data['detail']}" )
print( f"Token Type: {data['token_type']}" )

I was expecting it to just work, but it did not. On Windows 10, the error is:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=’localhost’, port=5000): Max retries exceeded with url: /auth/token (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)’)))

And on Ubuntu, the error is:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=’localhost’, port=5000): Max retries exceeded with url: /auth/token (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1020)’)))

🦀 And so, I did not expect the Rust server’s login endpoint to work. The endpoint is https://localhost:5000/api/login.

Content of urllib3_001_login_rust.py:

import json
import urllib3

http = urllib3.PoolManager()

encoded_body = json.dumps({
    "email": "saniya.kalloufi.10008@gmail.com", 
    "password": "password",
})

# See F:/rust/actix_web/tests/test_auth_handlers.rs
#     https://github.com/behai-nguyen/rust_web_01/blob/125378410c5afa06e22646deacb68c80021a303f/tests/test_auth_handlers.rs#L178-L207
#       async fn post_login_json()
resp = http.request(
    "POST",
    "https://localhost:5000/api/login",
    body=encoded_body,
    headers={
        "content-type": "application/json"
    }    
)

status = resp.json()

print( f"Code: {status['code']}" )
print( f"Message: {status['message']}" )
print( f"Session Id: {status['session_id']}\n" )

data = status['data']

print( f"Email: {data['email']}" )
print( f"Access Token: {data['access_token']}" )
print( f"Token Type: {data['token_type']}" )

Windows 10 error:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=’192.168.0.16’, port=5000): Max retries exceeded with url: /api/login (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)’)))

Ubuntu error:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=’192.168.0.16’, port=5000): Max retries exceeded with url: /api/login (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1020)’)))

❹ According to the official page Custom SSL certificates, the pool manager should be initiated with the certificate information as follows:

>>> import urllib3
>>> http = urllib3.PoolManager(
...     cert_reqs='CERT_REQUIRED',
...     ca_certs='/path/to/your/certificate_bundle')

This article OpenSSL create Certificate Chain [Root & Intermediate CA] dated 27/07/2024 discusses certificate bundles. After completing all the certificate creation steps, we should have the following directory content:

behai@HP-Pavilion-15:~$ tree myCA/
myCA/
├── intermediateCA
│   ├── certs
│   │   ├── ca-chain.cert.pem
│   │   ├── intermediate.cert.pem
│   │   ├── intermediate.csr.pem
│   │   └── localhost.cert.pem
│   ├── crl
│   ├── crlnumber
│   ├── csr
│   │   └── localhost.csr.pem
│   ├── index.txt
│   ├── index.txt.attr
│   ├── index.txt.old
│   ├── newcerts
│   │   └── 1000.pem
│   ├── openssl_intermediate.cnf
│   ├── private
│   │   ├── intermediate.key.pem
│   │   └── localhost.key.pem
│   ├── serial
│   └── serial.old
└── rootCA
    ├── certs
    │   └── ca.cert.pem
    ├── crl
    ├── crlnumber
    ├── csr
    ├── index.txt
    ├── index.txt.attr
    ├── index.txt.old
    ├── newcerts
    │   └── 1000.pem
    ├── openssl_root.cnf
    ├── private
    │   └── ca.key.pem
    ├── serial
    └── serial.old

13 directories, 25 files
behai@HP-Pavilion-15:~$

👉 Please note that for the final, local certificate, I replaced the file name prefix www.example.com with localhost.

The following two files:

  1. /home/behai/myCA/intermediateCA/certs/localhost.cert.pem
  2. /home/behai/myCA/intermediateCA/private/localhost.key.pem

are the certificate and key files that can be used in the Python and Rust server code. And the file:

  • /home/behai/myCA/intermediateCA/certs/ca-chain.cert.pem

is the certificate bundle ca_certs='/home/behai/myCA/intermediateCA/certs/ca-chain.cert.pem'.

As discussed in the two server posts, during the certificate creation, we should answer a series of questions as listed below:

Country Name (2 letter code) []: AU
State or Province Name []: Victoria
Locality Name []: Melbourne
Organization Name []: Personal
Organizational Unit Name []: Development
Common Name []: HP-Pavilion-15/localhost/192.168.0.16
Email Address []: behai_nguyen@hotmail.com

💥 For the Common Name field, contrary to what I have discussed before, I also tried localhost and the Ubuntu IP address 192.168.0.16. All three values produce the same result.

❺ 👉 Please note that the following local modifications and testing have been carried out on Ubuntu only.

🐍 Refactor the Python Server

To use the newly generated certificate file in the main.py module, replace the last three lines:

130
131
132

if __name__ == "__main__":
    uvicorn.run("main:app", host="0.0.0.0", port=5000, \
                ssl_keyfile="./cert/key.pem", ssl_certfile="./cert/cert.pem")

with:

if __name__ == "__main__":
    uvicorn.run("main:app", host="0.0.0.0", port=5000, \
                ssl_keyfile="/home/behai/myCA/intermediateCA/private/localhost.key.pem", \
                ssl_certfile="/home/behai/myCA/intermediateCA/certs/localhost.cert.pem")

Update urllib3_001_login_python.py to use the certificate bundle:

...
http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED',
                           ca_certs="/home/behai/myCA/intermediateCA/certs/ca-chain.cert.pem")
...

This does not address the problem. The error message is slightly different:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=’localhost’, port=5000): Max retries exceeded with url: /auth/token (Caused by SSLError(SSLCertVerificationError(1, “[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for ‘localhost’. (_ssl.c:1020)”)))

🦀 Refactor the Rust Server In the src/lib.rs module, replace line 80:

80

let mut file = File::open("./cert/key-pass.pem").unwrap();

with:

let mut file = File::open("/home/behai/myCA/intermediateCA/private/localhost.key.pem").unwrap();

And replace line 98:

98

builder.set_certificate_chain_file("./cert/cert-pass.pem").unwrap();

with:

builder.set_certificate_chain_file("/home/behai/myCA/intermediateCA/certs/localhost.cert.pem").unwrap();

Then update urllib3_001_login_rust.py to use the certificate bundle:

...
http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED', 
                           ca_certs="/home/behai/myCA/intermediateCA/certs/ca-chain.cert.pem")
...

As anticipated, the error is:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=’192.168.0.16’, port=5000): Max retries exceeded with url: /api/login (Caused by SSLError(SSLCertVerificationError(1, “[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for ‘192.168.0.16’. (_ssl.c:1020)”))) IP address mismatch, certificate is not valid for ‘192.168.0.16’

❻ This article, Disable SSL Verification in Python – requests, urllib3, discusses disabling SSL verification. The pool manager should be initiated as follows:

urllib3.disable_warnings()
http = urllib3.PoolManager(
    cert_reqs='CERT_NONE',
    assert_hostname=False
)

👉 Please note, we leave the above local changes in place for both servers.

🐍 Update the Python Server Login Script

Update urllib3_001_login_python.py to:

...
urllib3.disable_warnings()
http = urllib3.PoolManager(cert_reqs='CERT_NONE', assert_hostname=False)
...

And we are now able to access the Python server. The output of the test script is:

Status code: 200
Status text:
Access Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJiZWhhaV9uZ3V5ZW5AaG90bWFpbC5jb20iLCJlbXBfbm8iOjUwMDIyMiwic2NvcGVzIjpbInVzZXI6cmVhZCIsInVzZXI6d3JpdGUiXSwiZXhwIjoxNzM0MzIzMzMzfQ.Xi4q2LbqVax06sRaxLx2R1oTN38CMyQarpd_8mQrwII
Detail:
Token Type: bearer

🐍 Python Server New HTTP GET Test Script Using the above access token in a new test script urllib3_002_admin_me_python.py to access the Python server endpoint https://localhost:5000/admin/me.

Content of urllib3_002_admin_me_python.py:

import urllib3

# Disable SSL verification and warning.
urllib3.disable_warnings()
http = urllib3.PoolManager(cert_reqs='CERT_NONE', assert_hostname=False)

resp = http.request(
    "GET",
    "https://localhost:5000/admin/me",
    headers={
        "Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJiZWhhaV9uZ3V5ZW5AaG90bWFpbC5jb20iLCJlbXBfbm8iOjUwMDIyMiwic2NvcGVzIjpbInVzZXI6cmVhZCIsInVzZXI6d3JpdGUiXSwiZXhwIjoxNzM0MzIzMzMzfQ.Xi4q2LbqVax06sRaxLx2R1oTN38CMyQarpd_8mQrwII",
		"x-expected-format": "application/json"
    }
)

print( resp.json() )

The new test script executes successfully. The response is captured in the screenshot below:


🦀 Update the Rust Server Login Script Apply the same modification to the Rust server login test code. Update urllib3_001_login_rust.py to:

...
urllib3.disable_warnings()
http = urllib3.PoolManager(cert_reqs='CERT_NONE', assert_hostname=False)
...

And the successful output is:

Code: 200
Message: None
Session Id: None

Email: saniya.kalloufi.10008@gmail.com
Access Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJlbWFpbCI6InNhbml5YS5rYWxsb3VmaS4xMDAwOEBnbWFpbC5jb20iLCJzZXNzaW9uX2lkIjoiNDlhZDQ4OWUtYjk4NC00ZTg4LTlhMWItYjJhZTM3ZDA0YzgzIiwiaWF0IjoxNzM0MzIzMTcwLCJleHAiOjE3MzQzMjQ5NzAsImxhc3RfYWN0aXZlIjoxNzM0MzIzMTcwfQ.zuZ2bIW5R65fNhEYMjkTqWaLqnKJbeTGmVBRs7ioNa8
Token Type: bearer

🦀 Rust Server New HTTP POST Test Script Using the above access token in a new test script urllib3_002_data_employees_rust.py to access the Rust server endpoint https://localhost:5000/data/employees.

Content of urllib3_002_data_employees_rust.py:

import json
import urllib3

# Disable SSL verification and warning.
urllib3.disable_warnings()
http = urllib3.PoolManager(cert_reqs='CERT_NONE', assert_hostname=False)

encoded_body = json.dumps({
    "last_name": "%chi",
    "first_name": "%ak",
})

# See F:/rust/actix_web/tests/test_handlers.rs
#     https://github.com/behai-nguyen/rust_web_01/blob/125378410c5afa06e22646deacb68c80021a303f/tests/test_handlers.rs#L65-L102
#       async fn post_employees_json1()
resp = http.request(
    "POST",
    "https://localhost:5000/data/employees",
    body=encoded_body,
    headers={
        "Authorization": "Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJlbWFpbCI6InNhbml5YS5rYWxsb3VmaS4xMDAwOEBnbWFpbC5jb20iLCJzZXNzaW9uX2lkIjoiNDlhZDQ4OWUtYjk4NC00ZTg4LTlhMWItYjJhZTM3ZDA0YzgzIiwiaWF0IjoxNzM0MzIzMTcwLCJleHAiOjE3MzQzMjQ5NzAsImxhc3RfYWN0aXZlIjoxNzM0MzIzMTcwfQ.zuZ2bIW5R65fNhEYMjkTqWaLqnKJbeTGmVBRs7ioNa8",
        "content-type": "application/json",
    }    
)

status = resp.json()
print(f"status: {status}")

It is also successful. The output is captured in the screenshot below:


👉 From Windows 10, to access the Rust and Python servers, just replace https://localhost:5000/ with https://192.168.0.16:5000/ in the test scripts.

❼ The primary purpose of implementing SSL/HTTPS for localhost using a self-signed certificate is to better manage cookies, as discussed in the two server posts. Therefore, disabling SSL verification is not a problem, as it is not about security. In hindsight, I would prefer the method of creating a self-signed certificate as discussed in this post, rather than the methods employed in the two server posts.

Thank you for reading. I hope you find the information in this post useful. Stay safe, as always.

✿✿✿

Feature image sources: