Selenium Web Browser Automation

Noob--I'm trying to automate verification of survey completion for 600 users. I can open the page fine, but struggle putting a value into the input box. What am I missing? Any help appreciated.

Website: https://gptw.care/vi

Input: document.querySelector("#root > div > div.MuiFormControl-root.MuiTextField-root.jss16")

#SingleInstance, force
#Include, Lib\Chrome\Chrome.ahk ;https://github.com/G33kDude/Chrome.ahk/releases

; verify if the profile folder exists
; if not, create it
if !FileExist("profile")
FileCreateDir, % "profile"

; load the chrome object pointing to the folder created above
; and the link to the page (or pages) to open at startup
chrome := new chrome("profile", "https://gptw.care/vi")

; get the first page (you can use other functions to select a specific page, like GetPageByTitle() or GetPageByURL())
pg := chrome.getpage()

; wait for the page to finish loading before clicking anything.
; clicking on an nonexistant element results in an error
pg.WaitForLoad("complete")

; pass arbitrary javascript to the page. This allows you to do virtually anything you want
; with the page
pg.Evaluate("document.querySelector('#root > div > div.MuiFormControl-root.MuiTextField-root.jss16').value = '7654321'")

return

Hi! I have developed an application for parsing store products with uploading product cards to telegram. And it works, but only on a local computer. When the application is uploaded to the hosting, the program doesn't show any errors, but the product cards don't appear. Please help me.
main.py

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from congif import *
def ex(func):
   async def show_ex(*args, **kwargs):
       try:
           await func(*args, **kwargs)
       except Exception as e:
           print('Error: ', e)
   return show_ex
@ex
async def create_table(cursor, word):
   try:
       await cursor.execute(f'''
           CREATE TABLE IF NOT EXISTS {word.lower()} (
               id INTEGER PRIMARY KEY AUTOINCREMENT,
               price TEXT,
               link TEXT UNIQUE,
               img TEXT
           )
       ''')
   except sqlite3.OperationalError:
       time.sleep(10)
       await cursor.execute(f'''
                   CREATE TABLE IF NOT EXISTS {word.lower()} (
                       id INTEGER PRIMARY KEY AUTOINCREMENT,
                       price TEXT,
                       link TEXT UNIQUE,
                       img TEXT
                   )
               ''')
@ex
async def insert_data(cursor, word, new_price, link, img):
   try:
       await cursor.execute(f'''
           INSERT INTO {word.replace(" ", "").lower()} (price, link, img) VALUES (?, ?, ?)
       ''', (new_price, link, img))
   except sqlite3.IntegrityError:
       pass
   except sqlite3.OperationalError:
       time.sleep(10)
       await cursor.execute(f'''
                   INSERT INTO {word.replace(" ", "").lower()} (price, link, img) VALUES (?, ?, ?)
               ''', (new_price, link, img))
@ex
async def check_table_exists(conn, table_name):
   async with conn.cursor() as cursor:
       await cursor.execute(f"SELECT name FROM sqlite_master WHERE type='table' AND name='{table_name.lower()}';")
       return await cursor.fetchone() is not None
@ex
async def send_product_messages(user_id, card, img_url):
   try:
       await bot.send_photo(user_id, photo=img_url, caption=card)
   except Exception as e:
       print(f"Error: {e}")
@ex
async def process_user(user_id, products):
   conn = await aiosqlite.connect('/data/mercaridata.db')
   async with conn.cursor() as cursor:
       tasks = []
       for product_key, product_value in products.items():
           table_exists = await check_table_exists(conn, product_value)
           if not table_exists:
               await create_table(cursor, product_value.replace(" ", ''))
           url = f'https://jp.mercari.com/search?keyword={product_value.replace(" ", "%20")}&sort=created_time&order=desc'
           task = asyncio.create_task(process_product(user_id, cursor, product_value, url))
           tasks.append(task)
       await asyncio.gather(*tasks)
       await conn.commit()
@ex
async def process_product(user_id, cursor, product_value, url):
   options = webdriver.ChromeOptions()
   options.add_argument("--headless")
   options.add_argument("--no-sandbox")
   options.add_argument("--disable-gpu")
   options.add_argument("--remote-debugging-port=9230")
   driver = webdriver.Chrome(options=options)
   try:
       driver.maximize_window()
       driver.get(url)
       card_divs = driver.find_elements(By.XPATH, '//li[@class="sc-bcd1c877-2 gWHcGv"]')[:10]
       for card_div in card_divs:
           try:
               price = float(
                   (card_div.find_element(By.TAG_NAME, 'span').text.replace(' ', '').replace(',', '').
                    replace('\n', '')))
           except ValueError:
               price = 'SALE!'
           except Exception:
               price = '?'
           try:
               link = card_div.find_element(By.TAG_NAME, 'a').get_attribute('href')
           except Exception:
               continue
           img = card_div.find_element(By.TAG_NAME, 'img').get_attribute('src')
           await insert_data(cursor, product_value, price, link, img)
           print(price, link, img)
           card = (f"Item: {link} \n"
                   f"Price: {price} $.\n")
           if card:
               await send_product_messages(user_id, card, img)
           else:
               print('ТNo items')
               await bot.send_message(user_id, 'No items')
   finally:
       driver.stop_client()
       driver.close()
       driver.quit()
 

So I sometimes run selenium without being connected to the internet, and whenever I do that it spits out this error message:

Exception managing firefox: error sending request for url (https://github.com/mozilla/geckodriver/releases/latest): error trying to connect: dns error: No such host is known. (os error 11001)

Now of course I could wrap it in a try-catch-statement, but is there a proper way to prevent displaying this message (or disable the update check)? The code still works properly, I just don't wanna see the message, or even control the update check.

I'm behind a company proxy and am normally able to do anything if I just point to it(no user/pass required). I've tried about all the different options to set it, the last one was the example of the docs here: https://www.selenium.dev/documentation/webdriver/drivers/options/#proxy

But every time there's a Geckodriver update I run into the same NoSuchDriverException: Message: Unable to obtain geckodriver using Selenium Manager: Message: Unsuccessful command executed. Error sending request for url (https://github.com/mozilla/geckodiver/releases/latest): error trying to connect: tcp connect error: something something os error 10060.

Does setting the webdriver desiredcapabilities proxy even have an effect on the selenium manager that downloads the new driver?

I'm trying to automate a process where I log into a website, download a series of documents, process them, then re-upload. I'm having trouble downloading; the preferences don't seem to work:

chromedriver_path = "C:\\Users\\XXX\\PycharmProjects\\YYY\\util_files\\chromedriver-win64\\chromedriver.exe"
service = Service(chromedriver_path)
options = Options()
options.headless = True
options.add_argument("--incognito")
prefs = {"download.default_directory": "C:\\Users\\XXX\\Downloads",
         "download.prompt_for_download": False,
         "download.directory_upgrade": True}
options.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(service=service, options=options)

It doesn't run it headless (not a big deal) but it keeps prompting me with the download dialog. It's tough to find a solution online because it seems like Selenium has updated how it does some things over time, so I don't know of my code is the most up to date or not. TIA

I can't get selenium (on Python 3.11) to interact with the chrome's (122.0.6261.112 (Official Build) (64-bit) (cohort: Stable)) profile selection pop-up (on the browser start-up) correctly.

So far I've tried:

• finding elements by visible text (nothing is found when searching for an XPATH containing "Sign in")
• keyboard navigation (TAB switches between elements but ENTER doesn't open the login pop-up despite playing the 'click animation')
• moving the cursor to the element I got the TAB to highlight and clicking (click() action de-selects the highlighted element)
• disabling the profile selection window by parsing options arguments when launching the browser (best I got was no default profile listed on the pop-up)
• logging-in manually for the duration of the test session (chrome opens new window which messes up the 'yield' statement of my browser fixture because "window is already closed, duh"; that login doesn't persist between browser sessions so the pop-up appears again)

I'm bound to testing with the stable chrome because of the company policy. Our IT is controlling the versioning and might even force certain policies on the browser behavior. This profile selector started showing up yesterday and I really need to either disable it or get past it.

Hello, I'm attempting to screenshot tweets with selenium and some other python packages. I have it working perfectly in my local environment, but when i attempt to run the script via my ec2 instance, videos within tweets are unable to load. iIve tried adding the enable-javascript arg, which doesn't help.

has anyone encountered this issue? using python 3.11.5 with selenium 4.16, latest stable chromedriver, and im on an amazon ec2 server running the script with gunicorn.. using the following args:

chrome_options.add_argument("--headless") chrome_options.add_argument("--no-sandbox") chrome_options.add_argument("--disable-gpu") chrome_options.add_argument("--test-type") chrome_options.add_argument("--disable-logging") chrome_options.add_argument('--ignore-certificate-errors') chrome_options.add_argument("--disable-dev-shm-usage") chrome_options.add_argument(f"--window-size={ceil(1024*scale)},{ceil(1024*scale)}") chrome_options.add_argument("--remote-debugging-port=9222") chrome_options.add_argument("--incognito") chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36")

any help is much appreciated.

I'm experiencing an issue with my web scraper running on a Linux server where it fails to locate elements using Selenium WebDriver. The scraper works perfectly fine on my local machine, but when I run it on the Linux server, it throws a NoSuchElementException.

The error message I receive is:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".Crom_table__p1iZz"}

I've tried using WebDriverWait to wait for the table to load, but it doesn't seem to be working. Additionally, I've logged the page source on the Linux server and compared it to my local machine, and I've noticed significant differences in the HTML structure.

I am using a custom user-agent to prevent differences, but that hasn't resolved the issue. I'm not sure what else to try to resolve this issue.

Here's a snippet of my code:

def team_defenses(season):
    url = 'https://www.nba.com/stats/teams/defense?Season=' + season
  
    driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=get_driver_options())

    driver.get(url)

    # Wait for the table to load, if it doesn't load display content
    try:
        WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'Crom_table__p1iZz')))
    except Exception as e:
        print(f"Error: {e}")
        with open('error_page_source.txt', 'w') as f:
            f.write(driver.page_source)
        driver.quit()
        return

    table = driver.find_element(By.CLASS_NAME, 'Crom_table__p1iZz')

    # ... rest of my code ...

    driver.quit()

def get_driver_options():
    options = Options()
    options.add_argument('--headless')  # Runs Chrome in headless mode.
    options.add_argument('--no-sandbox')  # Bypass OS security model
    options.add_argument('--disable-dev-shm-usage')  # Overcome limited resource problems
    return options

I wrote a script to brute force a solution to a quiz on a google forms page There is a text field and a button
Entering a guess in the field and pressing the button results in either a text message that the solution is wrong or a new page is loaded saying the answer is correct

I successfully wrote a simple loop that changes the text of the field and presses the button

1. How can I detect the page change after the button is pressed? (when the guess is correct)
2. When I just looped through the numbers 000-999 the solution page did not load (I was expecting the script to fail at the next number since the input field is not present in the solution page). However, when I debugged the script, single-stepping the solution caused the browser to load the solution page. Do I need to add a delay (sleep) after the button click?

To run the Java Selenium application on a separate server I am attempting to build a Dockerfile install Chrome and use a headless way to run a browser. (This application provides a desired output when executed locally.) But when attempting to run the docker I receive this error.

Am I missing something in the docker build or the Java-based configurations provided.? How should chrome driver-related configurations be added?

Technologies used

• OS - Ubuntu 22.04 LTS
• Docker version - Docker version 24.0.2, build cb74dfc
• Java version - java version "17.0.4" 2022-07-19 LTS
• Selenium version - 4.11.0
• Chrome version - Google Chrome 120.0.6099.199--

Update: There also seems to be another way to run Chrome using selenium/standalone-chrome. Is it possible to integrate chrome/selenium with another docker using this method? which is the preferred option of these two methods?

>FAILED CONFIGURATION: u/BeforeClass openBrowser

org.openqa.selenium.remote.NoSuchDriverException: Unable to obtain: Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*, --headless], extensions: [], prefs: {download.default_directory: report/}}}, error Command failed with code: 65, executed: [/tmp/selenium-manager246697546082813077936780283589629/selenium-manager, --browser, chrome, --output, json]

request or response body error: operation timed out

Build info: version: '4.11.0', revision: '040bc5406b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-150-generic', java.version: '17.0.6'

Driver info: driver.version: ChromeDriver
at org.openqa.selenium.remote.service.DriverFinder.getPath(DriverFinder.java:25)
at org.openqa.selenium.remote.service.DriverFinder.getPath(DriverFinder.java:13)
at org.openqa.selenium.chrome.ChromeDriver.generateExecutor(ChromeDriver.java:99)
at org.openqa.selenium.chrome.ChromeDriver.(ChromeDriver.java:88)
at org.openqa.selenium.chrome.ChromeDriver.(ChromeDriver.java:83)
at org.openqa.selenium.chrome.ChromeDriver.(ChromeDriver.java:72)
at Infra.BasePage.openBrowser(BasePage.java:108)

Caused by: org.openqa.selenium.WebDriverException: Command failed with code: 65, executed: [/tmp/selenium-manager246697546082813077936780283589629/selenium-manager, --browser, chrome, --output, json]

request or response body error: operation timed out Build info: version: '4.11.0', revision: '040bc5406b' System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-150-generic', java.version: '17.0.6' Driver info: driver.version: ChromeDriver at org.openqa.selenium.manager.SeleniumManager.runCommand(SeleniumManager.java:151) at org.openqa.selenium.manager.SeleniumManager.getDriverPath(SeleniumManager.java:273) at org.openqa.selenium.remote.service.DriverFinder.getPath(DriverFinder.java:22)

from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options

options = Options()
options.binary_location = '/mnt/sdb1/firefox/'

driver = webdriver.Firefox(options = options, service=Service(executable_path = '/mnt/sdb1/root'))

driver.get('https://youtube.com')
driver.execute_script('alert(\'your favorite music is here\')')

I get the following error

Traceback (most recent call last):
  File "/mnt/sdb1/root/sele.py", line 8, in 
    driver = webdriver.Firefox(options = options, service=Service(executable_path = '/mnt/sdb1/root'))
  File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
    self.service.path = DriverFinder.get_path(self.service, options)
  File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/common/driver_finder.py", line 44, in get_path
    raise NoSuchDriverException(f"Unable to locate or obtain driver for {options.capabilities['browserName']}")
selenium.common.exceptions.NoSuchDriverException: Message: Unable to locate or obtain driver for firefox; For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/driver_location

I've created a library to bring the Testing Library (testing-library.com) to Selenium (Kotlin/Java) since it's an awesome library with great principles for promoting usability, accessibility, and tests that don't depend on tech details. Check it out!

The more your tests resemble the way your software is used, the more confidence they can give you. (Kent C. Dodds)

more info

Hi, so I post around 30 facebook market place ads a month, its a super tideious task and could easily be done by a computer. The issue is, is that facebook has such a weird structure when I inspect elements. the image attached is a example of it, the class names are this weird code, and I cant find any names, ids, on these elements to give to Selenium. at the moment I use py auto GUI + selenium to click around the web page, but this is super hard coded and doesn't work all the time. so please let me know what you guys recommend

So I am a brand new user of Selenium in Python. I am really enthusiastic about using this framework, but I'm encountering some issues with certificate errors. When I Google for this issue, I noticed several users in the past year having similar issues but nobody really providing a workable solution.

I'm essentially testing out Selenium by trying out the first script in the documentation before cutting my teeth on scraping my actual target website which I plan to do after everything is smooth with the first script that acts as my baseline to ensure that everything in my development environment is setup right:

https://www.selenium.dev/documentation/webdriver/getting_started/first_script/

Note that the official documentation never actually tells me to install the Selenium web driver. It just tells me to install Selenium on my Python virtual environment which I did using "pip install selenium".

However at the time of reviewing the above documentation I was aware that a separate web driver for the web browser of choice was needed too. In my case I use Chrome. I decided I would first just follow the official documentation and see what would happen.

I pasted in the code verbatim from the Selenium github (https://github.com/SeleniumHQ/seleniumhq.github.io/blob/trunk/examples/python/tests/getting_started/first_script.py#L4) into first_script.py and then ran it in my python interpreter.

To my surprise it did spawn a browser briefly and then closed. But it's throwing up several certificate errors:

[2944:20700:1219/211513.232:ERROR:cert_issuer_source_aia.cc(34)] Error parsing cert retrieved from AIA (as DER):
ERROR: Couldn't read tbsCertificate as SEQUENCE
ERROR: Failed parsing Certificate

Upon further inspection the script does appear to submit the form and then scrape the form results page as it correctly scripts "Received!" and stores it as a string in the "text" variable.

However the certificate errors are a huge problem I need resolved. My normal Chrome installation doesn't throw up any certificate errors on https://www.selenium.dev/selenium/web/web-form.html which the above Selenium script submits and then scrapes the result. The web form url has a perfectly valid lets encrypt SSL certificate.

Remember I mentioned that I didn't install the web driver since it wasn't mentioned in the official document? I decided to rectify that since I figured maybe that could be the problem.

Since I am using Chrome 120.0.6099.110 I went to https://sites.google.com/chromium.org/driver/downloads?authuser=0 which instructed me to go here: https://googlechromelabs.github.io/chrome-for-testing/

I grabbed chromedriver.exe that matches my web browser and dropped it into my project directory. Then I modified first_script.py to ensure that it calls that specific web driver also commented out the original code:

#driver = webdriver.Chrome()
service = Service('C:\\py311seleniumbot\\chromedriver.exe')
driver = webdriver.Chrome(service=service)

Well it's using the new chromedriver.exe alright but I'm still getting the same cert error.

I'm at a loss on how to fix this. I need there to be a proper https connection to the remote website. I don't want to sweep this under the rug by using the "--ignore-certificate-errors" flag as that doesn't address the security concerns of a production environment.

Is there a kind soul in this wonderful Selenium community that can help a budding selenium enthusiast?

If I can get this pesky certificate issue resolved, I'm going to be well on my way to becoming a Selenium fanatic and very excited to start automating with this beauty.

I try to use selunim to solve the capatha using ai but I have a problem in clicking the img of capatcha I don't know I try many methods like using javascript... To bypass it put it not work.. Url site : "https://algeria.blsspainglobal.com/DZA/account/login"

I use selenium in python with chromedriver to scrape my bank account balances everyday so I don't have to log into multiple websites. I don't think Chase likes this and they actively try to stop it with their use of shadow roots, etc., but I've still managed to scrape Chase up till now. As of a couple weeks ago, I'm getting a really strange behavior, where if I run the python script there is an element not found error message for an element that clearly is there. Weirdly, if I put a breakpoint in the code and then simply resume execution once it hits the breakpoint then the element is found (by ID) and my script runs, but otherwise it doesn't. Timers don't help. I've tried sleeping for up to 20 seconds and the element is still not found, but something about hitting a breakpoint in the python script allows the element to be found. Anyone else run into this?

The line is:

element = driver.find_element("id", "requestAccounts")

Anyone?

I had this question, found a privated reddit post, so I figured I'd ask again :)

I'm basically trying to figure out a minimal container that can load pages via selenium and take a screenshot.

"I am presently using a firefox gecko driver with python in ubuntu for my auto testing. The problem is it uses a very high memory about 150-350 mb and also cpu for a single site. Is there any lightweight selenium web drivers which uses less system resources? Can you recommend any? "

Manual QA here want to start learning Test Automation. Coming from a non-tech background will make it a bit of a challenge but I'm looking to "test" myself.

Out of the three languages (Java, JS, Py), which should I focus on learning to get the best knowledge to apply when eventually using Selenium WebDriver?

I would like to comprehend all three later on in the future but for now which is the best to start off or continue with. Many have mentioned Python is beginner-friendly approach to learning programming and less stressful, but I see a lot of tutorials for Selenium webdriver being taught using Java, Also I heard JS is being implemented a lot more and more nowadays.