So I am a brand new user of Selenium in Python. I am really enthusiastic about using this framework, but I'm encountering some issues with certificate errors. When I Google for this issue, I noticed several users in the past year having similar issues but nobody really providing a workable solution.
I'm essentially testing out Selenium by trying out the first script in the documentation before cutting my teeth on scraping my actual target website which I plan to do after everything is smooth with the first script that acts as my baseline to ensure that everything in my development environment is setup right:
https://www.selenium.dev/documentation/webdriver/getting_started/first_script/
Note that the official documentation never actually tells me to install the Selenium web driver. It just tells me to install Selenium on my Python virtual environment which I did using "pip install selenium".
However at the time of reviewing the above documentation I was aware that a separate web driver for the web browser of choice was needed too. In my case I use Chrome. I decided I would first just follow the official documentation and see what would happen.
I pasted in the code verbatim from the Selenium github (https://github.com/SeleniumHQ/seleniumhq.github.io/blob/trunk/examples/python/tests/getting_started/first_script.py#L4) into first_script.py and then ran it in my python interpreter.
To my surprise it did spawn a browser briefly and then closed. But it's throwing up several certificate errors:
[2944:20700:1219/211513.232:ERROR:cert_issuer_source_aia.cc(34)] Error parsing cert retrieved from AIA (as DER):
ERROR: Couldn't read tbsCertificate as SEQUENCE
ERROR: Failed parsing Certificate
Upon further inspection the script does appear to submit the form and then scrape the form results page as it correctly scripts "Received!" and stores it as a string in the "text" variable.
However the certificate errors are a huge problem I need resolved. My normal Chrome installation doesn't throw up any certificate errors on https://www.selenium.dev/selenium/web/web-form.html which the above Selenium script submits and then scrapes the result. The web form url has a perfectly valid lets encrypt SSL certificate.
Remember I mentioned that I didn't install the web driver since it wasn't mentioned in the official document? I decided to rectify that since I figured maybe that could be the problem.
Since I am using Chrome 120.0.6099.110 I went to https://sites.google.com/chromium.org/driver/downloads?authuser=0 which instructed me to go here: https://googlechromelabs.github.io/chrome-for-testing/
I grabbed chromedriver.exe that matches my web browser and dropped it into my project directory. Then I modified first_script.py to ensure that it calls that specific web driver also commented out the original code:
#driver = webdriver.Chrome()
service = Service('C:\\py311seleniumbot\\chromedriver.exe')
driver = webdriver.Chrome(service=service)
Well it's using the new chromedriver.exe alright but I'm still getting the same cert error.
I'm at a loss on how to fix this. I need there to be a proper https connection to the remote website. I don't want to sweep this under the rug by using the "--ignore-certificate-errors" flag as that doesn't address the security concerns of a production environment.
Is there a kind soul in this wonderful Selenium community that can help a budding selenium enthusiast?
If I can get this pesky certificate issue resolved, I'm going to be well on my way to becoming a Selenium fanatic and very excited to start automating with this beauty.