Selenium Web Browser Automation

101 readers

4 users here now

Users forum for selenium browser testing. Do not advertise here.

founded 2 years ago

MODERATORS

Terevos

[email protected]

Weird behavior trying to scrape chase.com (self.selenium)

submitted 1 year ago* (last edited 1 year ago) by joystick to c/selenium

2 comments fedilink hide all child comments

I use selenium in python with chromedriver to scrape my bank account balances everyday so I don't have to log into multiple websites. I don't think Chase likes this and they actively try to stop it with their use of shadow roots, etc., but I've still managed to scrape Chase up till now. As of a couple weeks ago, I'm getting a really strange behavior, where if I run the python script there is an element not found error message for an element that clearly is there. Weirdly, if I put a breakpoint in the code and then simply resume execution once it hits the breakpoint then the element is found (by ID) and my script runs, but otherwise it doesn't. Timers don't help. I've tried sleeping for up to 20 seconds and the element is still not found, but something about hitting a breakpoint in the python script allows the element to be found. Anyone else run into this?

The line is:

element = driver.find_element("id", "requestAccounts")

top 2 comments

sorted by: hot top controversial new old

[–] ShunkW 4 points 1 year ago (1 children)

It sounds like there's some js running after the page is fully loaded, but you haven't interacted with the page yet. If there's something inserting a DOM element in that gray area, then the element wouldn't be matched correctly. Perhaps try scrolling the page if possible. I haven't used selenium in a while so I'm not sure how to solve the exact problem you're seeing.

[–] joystick 2 points 1 year ago

Oh, wow, that worked. Thanks!