Solution 1 :

when i checked the div shows as:

<div class="some random code" data-id="search-assist-input-sugglst">

try:

.//div[@data-id='search-assist-input-sugglst']/ul[1]/li

i tried inspecting it its data-id not data-test, and and extra div before ul.

enter image description here

Problem :

I posted this question earlier, sorry for a repost but I can’t figure out why the code works for the person who answered and not for me.

How to get values from search suggestions after keying in text using python selenium?

I’ll retype the question and update it with the code he posted.

When you enter something for example apple into the search bar at https://finance.yahoo.com/ there is a search suggestions menu that appears.

enter image description here

I am trying to get it to return a list, dictionary or dataframe of the values in that drop down box.

For example

{'AAPL':['Apple Inc.','Equity - NMS','https://finance.yahoo.com/quote/AAPL?p=AAPL&.tsrc=fin-srch'],
 'AAPL.BA':['Apple Inc.','Equity - BUE','https://finance.yahoo.com/quote/AAPL.BA?p=AAPL.BA&.tsrc=fin-srch'],
  .....}

or

['AAPL','Apple Inc.','Equity - NMS','https://finance.yahoo.com/quote/AAPL?p=AAPL&.tsrc=fin-srch']
['APPL.BA','Apple Inc.','Equity - BUE','https://finance.yahoo.com/quote/AAPL.BA?p=AAPL.BA&.tsrc=fin-srch']

The last value is the hyperlink from clicking the link.

Someone posted this code below earlier,

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import time
import pandas as pd

options=webdriver.ChromeOptions()
options.add_argument('start-maximized')

driver = webdriver.Chrome(executable_path=r'C:Program Fileschromedriverchromedriver.exe',options=options)
url = "https://finance.yahoo.com/"
driver.get(url)

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="yfin-usr-qry"]'))).send_keys('apple')

WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div//*[contains(text(),'Symbols')]")))
web_elem_list = driver.find_elements_by_xpath(".//div[@data-test='search-assist-input-sugglst']/div/ul[1]/li/div")
results = pd.DataFrame()
for web_elem in web_elem_list:
    suggests=[]
    suggests.append(web_elem.find_element_by_xpath("./div/div").text)
    suggests.append(web_elem.find_element_by_xpath("./div/div/following-sibling::div").text)
    suggests.append(web_elem.find_element_by_xpath("./div/following-sibling::div").text)
    results=results.append(pd.Series(suggests),ignore_index=True)

print(results)

driver.close()

But keep getting blank values in the list. I don’t understand why, the person who posted says the code works for him, but I tried everything I could think of to troubleshoot. I think the xpath of the elements might not be right.

Does any one have any idea?

Comments

Comment posted by Danila Manturov

Have you tried to use a Firefox driver?

Comment posted by anarchy

i specifically said that question didnt answer my question

Comment posted by anarchy

I get this error InvalidSelectorException: Message: invalid selector: Unable to locate an element with the xpath expression ./.div/div because of the following error: TypeError: Failed to execute ‘evaluate’ on ‘Document’: The result is not a node set, and therefore cannot be converted to the desired type. (Session info: chrome=81.0.4044.138)

Comment posted by HardBoiledPotato

edited the answer. there is an extra div on the end. sorry

Comment posted by anarchy

ok now the first ./div part works! the next part gives an error, NoSuchElementException: Message: no such element: Unable to locate element: {“method”:”xpath”,”selector”:”./following-sibling::div”} (Session info: chrome=81.0.4044.138)

Comment posted by HardBoiledPotato

are you still using suggests.append(web_elem.find_element_by_xpath(“./div/div”).text) suggests.append(web_elem.find_element_by_xpath(“./div/div/following-sibling::div”).text) suggests.append(web_elem.find_element_by_xpath(“./div/following-sibling::div”).text) i tested them and they should work with .//div[@data-id=’search-assist-input-sugglst’]/ul[1]/li

Comment posted by anarchy

I tried both ways, it finally works now, I had to add some sleep timers

By

Leave a Reply

Your email address will not be published. Required fields are marked *