Python Automation: Web Scraping with BeautifulSoup

Web scraping is a powerful technique for extracting data from websites. Python makes it easy with BeautifulSoup.

from bs4 import BeautifulSoup
import requests

url = "https://example.com"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

titles = soup.find_all("h2")
for title in titles:
    print(title.text)

Most websites paginate their content. Here's how to handle it:

for page in range(1, 11):
    url = f"https://example.com/page/{page}"
    response = requests.get(url)
    # Process each page

Always respect robots.txt and rate limit your requests when scraping.