We have 3 kinds of infinite scrolling to bottom while using selenium.
- Simply scrolling – Stop until there is no more content.
- Meet scrolling – Stop until we meet a speciy element.
- Partial scrolling – Scrolling content in specify html container, stop until there are no more content.
All these 3 scenses will use Javascript. Let’s assume we have a driver created like this:
driver = webdriver.Chrome()
Note: here we all use
smooth
to scroll the windows to make it more like a humman action.
1 Simply scrolling
def scroll_to_bottom(driver):
old_height = driver.execute_script("return document.body.scrollHeight;")
while True:
driver.execute_script('window.scrollTo({top: document.body.scrollHeight, behavior: "smooth"})')
# sleep randomly
time.sleep(random.randint(1, 3))
new_height = driver.execute_script("return document.body.scrollHeight;")
if new_height == old_height:
break
old_height = new_height
Here, we use new_height == old_height
to decide if we meet the bottom, which means the content doesn’t expand anymore.
2 Meet scrolling
def scroll_to_bottom(driver):
# start to scroll until DOM <div id="content"> is displayed.
# Will throw an exception after 20 seconds
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, '//div[@id="content"]')))
# start to scroll
while True:
driver.execute_script('window.scrollTo({top: document.body.scrollHeight, behavior: "smooth"})')
# sleep randomly
time.sleep(random.randint(1, 3))
try:
# stop scrolling until we meet the DOM <span class="bottom">
driver.find_element(By.XPATH, '//span[@class="bottom""]')
break
except NoSuchElementException as e:
# Keep scrolling
continue
Partial scrolling
Here we only scroll the sidebar container, not the whole page.
def scroll_to_bottom(driver):
sidebar_element = 'document.querySelector(".sidebar")'
old_height = driver.execute_script(f"return {sidebar_element}.scrollHeight;")
while True:
driver.execute_script(f'{sidebar_element}.scrollTo({top: {sidebar_element}.scrollHeight, behavior: "smooth"})')
# sleep randomly
time.sleep(random.randint(1, 3))
new_height = driver.execute_script(f"return {sidebar_element}.scrollHeight;")
if new_height == old_height:
break
old_height = new_height