No not really. Web crawlers and scrapers can click through link permutations or scrape data available in the page, but when you have complex UI making additional calls there’s really no general way to do this easily. You would need to iterate through the UI for what you want to scrape or hook into their APIs if they allow that.

”’
I had a code for the webpage but on a different link. But I am unable to scrape this one due to the drop-down tabs. Is there a universal way to scrape a web page with this sort of structure or do we have to look at the structure and scrape?
here is the link I wish to scrape: https://www.moneycontrol.com/markets/fno-market-snapshot

from bs4 import BeautifulSoup
import requests
import pandas as pd
import time
r=requests.get('https://www.moneycontrol.com/india/stockpricequote/')

#We want to view the data in text format
data=r.text

#we can check some part of this html data
print(data[:1000])

soup=BeautifulSoup(data,'html.parser')

mydivs = soup.findAll("a", {"class": "bl_12"})
values=[]
for link in mydivs:
    value=link.get('href')
    values.append(value)


values= [ link.get('href') for link in mydivs]
values[0:5]
#create a temp Df
pd.set_option('max_colwidth', 800)
stock_data = pd.DataFrame({'LINK': values})

print(stock_data.shape)
stock_data.head(5)