This error occurs because you are getting the reference of a dom element which no longer exists.
Every time a page is reloaded or refreshed, or the dom has somewhat changed, every reference you have previously obtained is useless.
So, in order to work, you have to keep a reference of what you have already processed, and renew your dom reference every time the page changes.
# City loop
managed_cities = []
while True:
cities_table = driver.find_element_by_xpath('/html/body/div[1]/div[1]/div[1]/div[2]/div[1]/div')
cities = cities_table.find_elements_by_tag_name('div')
for city in cities:
# Here you have to find and ID or some unique text that identifies the
# already processed city in order to skip it
if city.text in managed_cities:
continue
city = city.find_element_by_tag_name('a') # Buttons have tags "a"
city.click()
time.sleep(3)
driver.back()
break
# Now, if you have managed all the cities, you can exit from the while
# otherwise, keep loop until all cities have been processed
if len(managed_cities) == len(cities):
break
I am trying to iterate clicking through the North American cities in the Uber Movement data.
I have setup Python code to select the table of all cities (cities_table), make it into a list, and identify the button for each city to be clicked (the button has a tag name ‘a’).
My problem is that I am able to select the first city in the table (Atlanta), but when I try to go back to the page with the table of all cities I do not manage to click on the next city on the list, which should be Boston.
After I successfully enter the Atlanta page, I setup a driver.back() to go back to the cities table, but when trying to move on to the next item on the list, I get the following error message:
Message: stale element reference: element is not attached to the page document
How can I correctly click on a city, go back to the cities table page and click on the NEXT item?
Here is my code so far:
# Load Chrome driver and movement.uber.com/cities website
PATH = 'C:Program Files (x86)chromedriver.exe'
driver = webdriver.Chrome(PATH)
driver.get('https://movement.uber.com/cities?lang=en-US')
# City loop
cities_table = driver.find_element_by_xpath('/html/body/div[1]/div[1]/div[1]/div[2]/div[1]/div')
cities = cities_table.find_elements_by_tag_name('div')
for city in cities:
city = city.find_element_by_tag_name('a') # Buttons have tags "a"
city.click()
time.sleep(3)
driver.back()
Full error message below:
StaleElementReferenceException Traceback (most recent call last)
<ipython-input-25-a148ef184a6b> in <module>
6 cities = cities_table.find_elements_by_tag_name('div')
7 for city in cities:
----> 8 city = city.find_element_by_tag_name('a') # Buttons have tags "a"
9 city.click()
10 time.sleep(3)
~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremotewebelement.py in find_element_by_tag_name(self, name)
303 element = element.find_element_by_tag_name('h1')
304 """
--> 305 return self.find_element(by=By.TAG_NAME, value=name)
306
307 def find_elements_by_tag_name(self, name):
~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremotewebelement.py in find_element(self, by, value)
657
658 return self._execute(Command.FIND_CHILD_ELEMENT,
--> 659 {"using": by, "value": value})['value']
660
661 def find_elements(self, by=By.ID, value=None):
~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremotewebelement.py in _execute(self, command, params)
631 params = {}
632 params['id'] = self._id
--> 633 return self._parent.execute(command, params)
634
635 def find_element(self, by=By.ID, value=None):
~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremotewebdriver.py in execute(self, driver_command, params)
319 response = self.command_executor.execute(driver_command, params)
320 if response:
--> 321 self.error_handler.check_response(response)
322 response['value'] = self._unwrap_value(
323 response.get('value', None))
~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremoteerrorhandler.py in check_response(self, response)
240 alert_text = value['alert'].get('text')
241 raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)
243
244 def _value_or_default(self, obj, key, default):
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=84.0.4147.89)
Why dod you want to iterate clicking through the North American cities? What is your usecase?
Within each city, there is a dataset I want to download. So the plan is: click on the city, select the specific time ranges for the data I want to download (which will be another loop entirely, because I will have to download separate csv files for each day because Uber doesn’t provide time series data, only monohronic data), download the csv data and go back to the North America cities page, so I can repeat the process for the next city.
Sorry, what do you mean by “code trials”? The city loop is the one on this thread, I just don’t understand why python/Selenium can’t recognize the next city in the loop. It says the element is not attached. Sorry if I didn’t get your question correctly.
Oh my god, thanks a lot! One thing though: when you say I need to find an ID or some text that identifies the already managed city, how would I do that if 1) the elements in the cities variable are now list items and 2) there is no way to find a unique id or text for each one, because the only thing differentiating them is the name of the city itself. Maybe something that might help is that if you print city.text, it will output the name of the city + United States, so for example, the string would be: “Atlanta nUnited States”.
The identifier really depends on your subsequent logic: 1) what are the available informations on DOM? 2) under what circumstances you can consider a city processed? My suggest is to find a reliable enough text content or attribute (eg “class”) and use it.