Solution 1 :

This error occurs because you are getting the reference of a dom element which no longer exists.
Every time a page is reloaded or refreshed, or the dom has somewhat changed, every reference you have previously obtained is useless.

So, in order to work, you have to keep a reference of what you have already processed, and renew your dom reference every time the page changes.

# City loop

managed_cities = []
while True:
    cities_table = driver.find_element_by_xpath('/html/body/div[1]/div[1]/div[1]/div[2]/div[1]/div')
    cities = cities_table.find_elements_by_tag_name('div')
    for city in cities:
        # Here you have to find and ID or some unique text that identifies the
        # already processed city in order to skip it
        if city.text in managed_cities:
            continue

        city = city.find_element_by_tag_name('a') # Buttons have tags "a"
        city.click()
        time.sleep(3)
        driver.back()
        break

    # Now, if you have managed all the cities, you can exit from the while
    # otherwise, keep loop until all cities have been processed
    if len(managed_cities) == len(cities):
        break

Problem :

I am trying to iterate clicking through the North American cities in the Uber Movement data.

I have setup Python code to select the table of all cities (cities_table), make it into a list, and identify the button for each city to be clicked (the button has a tag name ‘a’).

My problem is that I am able to select the first city in the table (Atlanta), but when I try to go back to the page with the table of all cities I do not manage to click on the next city on the list, which should be Boston.

After I successfully enter the Atlanta page, I setup a driver.back() to go back to the cities table, but when trying to move on to the next item on the list, I get the following error message:

Message: stale element reference: element is not attached to the page document

How can I correctly click on a city, go back to the cities table page and click on the NEXT item?

Here is my code so far:

# Load Chrome driver and movement.uber.com/cities website
PATH = 'C:Program Files (x86)chromedriver.exe'
driver = webdriver.Chrome(PATH)
driver.get('https://movement.uber.com/cities?lang=en-US')

# City loop
cities_table = driver.find_element_by_xpath('/html/body/div[1]/div[1]/div[1]/div[2]/div[1]/div')
cities = cities_table.find_elements_by_tag_name('div')
for city in cities:
    city = city.find_element_by_tag_name('a') # Buttons have tags "a"
    city.click()
    time.sleep(3)
    driver.back()

Full error message below:

StaleElementReferenceException            Traceback (most recent call last)
<ipython-input-25-a148ef184a6b> in <module>
      6 cities = cities_table.find_elements_by_tag_name('div')
      7 for city in cities:
----> 8     city = city.find_element_by_tag_name('a') # Buttons have tags "a"
      9     city.click()
     10     time.sleep(3)

~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremotewebelement.py in find_element_by_tag_name(self, name)
    303             element = element.find_element_by_tag_name('h1')
    304         """
--> 305         return self.find_element(by=By.TAG_NAME, value=name)
    306 
    307     def find_elements_by_tag_name(self, name):

~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremotewebelement.py in find_element(self, by, value)
    657 
    658         return self._execute(Command.FIND_CHILD_ELEMENT,
--> 659                              {"using": by, "value": value})['value']
    660 
    661     def find_elements(self, by=By.ID, value=None):

~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremotewebelement.py in _execute(self, command, params)
    631             params = {}
    632         params['id'] = self._id
--> 633         return self._parent.execute(command, params)
    634 
    635     def find_element(self, by=By.ID, value=None):

~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremotewebdriver.py in execute(self, driver_command, params)
    319         response = self.command_executor.execute(driver_command, params)
    320         if response:
--> 321             self.error_handler.check_response(response)
    322             response['value'] = self._unwrap_value(
    323                 response.get('value', None))

~AppDataRoamingPythonPython37site-packagesseleniumwebdriverremoteerrorhandler.py in check_response(self, response)
    240                 alert_text = value['alert'].get('text')
    241             raise exception_class(message, screen, stacktrace, alert_text)
--> 242         raise exception_class(message, screen, stacktrace)
    243 
    244     def _value_or_default(self, obj, key, default):

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=84.0.4147.89)

Comments

Comment posted by undetected Selenium

Why dod you want to iterate clicking through the North American cities? What is your usecase?

Comment posted by Luiz Scheuer

Within each city, there is a dataset I want to download. So the plan is: click on the city, select the specific time ranges for the data I want to download (which will be another loop entirely, because I will have to download separate csv files for each day because Uber doesn’t provide time series data, only monohronic data), download the csv data and go back to the North America cities page, so I can repeat the process for the next city.

Comment posted by undetected Selenium

Your relevant code trials?

Comment posted by Luiz Scheuer

Sorry, what do you mean by “code trials”? The city loop is the one on this thread, I just don’t understand why python/Selenium can’t recognize the next city in the loop. It says the element is not attached. Sorry if I didn’t get your question correctly.

Comment posted by undetected Selenium

I mean code trials for

Comment posted by Luiz Scheuer

Oh my god, thanks a lot! One thing though: when you say I need to find an ID or some text that identifies the already managed city, how would I do that if 1) the elements in the cities variable are now list items and 2) there is no way to find a unique id or text for each one, because the only thing differentiating them is the name of the city itself. Maybe something that might help is that if you print city.text, it will output the name of the city + United States, so for example, the string would be: “Atlanta nUnited States”.

Comment posted by Mattia Galati

The identifier really depends on your subsequent logic: 1) what are the available informations on DOM? 2) under what circumstances you can consider a city processed? My suggest is to find a reliable enough text content or attribute (eg “class”) and use it.

Comment posted by Luiz Scheuer

Each city only has class information and it is the same for every city in the cities_table. Inside each city, though, there is the button () for each. Each button has the following DOM info: class and href (which is a link/url). A city is considered processed once I run another huge loop which downloads .csv data from each day for 4 years of travel time data. The reason for all this is that I’m trying to create a time series dataset out of monochronic data.

Comment posted by Mattia Galati

You should check if, among cities, the text contained in the tag is different enough to keep a unique consistency. Another option could be using the href as an ID, since even a city with the same name, should be reachable through a unique url.

By