I'm experiencing an issue with my web scraper running on a Linux server where it fails to locate elements using Selenium WebDriver. The scraper works perfectly fine on my local machine, but when I run it on the Linux server, it throws a NoSuchElementException.
The error message I receive is:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".Crom_table__p1iZz"}
I've tried using WebDriverWait to wait for the table to load, but it doesn't seem to be working. Additionally, I've logged the page source on the Linux server and compared it to my local machine, and I've noticed significant differences in the HTML structure.
I am using a custom user-agent to prevent differences, but that hasn't resolved the issue. I'm not sure what else to try to resolve this issue.
Here's a snippet of my code:
def team_defenses(season):
url = 'https://www.nba.com/stats/teams/defense?Season=' + season
driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=get_driver_options())
driver.get(url)
# Wait for the table to load, if it doesn't load display content
try:
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'Crom_table__p1iZz')))
except Exception as e:
print(f"Error: {e}")
with open('error_page_source.txt', 'w') as f:
f.write(driver.page_source)
driver.quit()
return
table = driver.find_element(By.CLASS_NAME, 'Crom_table__p1iZz')
# ... rest of my code ...
driver.quit()
def get_driver_options():
options = Options()
options.add_argument('--headless') # Runs Chrome in headless mode.
options.add_argument('--no-sandbox') # Bypass OS security model
options.add_argument('--disable-dev-shm-usage') # Overcome limited resource problems
return options
I had an issue like this once. For me it was because the element wasn't "on screen". I would specify your resolution in your options and also there is another option to maximize the screen when the driver is instantiated.
I don't think this would be the reason, but make sure you are using the Linux version of the driver.
Hopefully you find the issue!