当前位置：首页 > news >正文

数学建模-爬虫入门

news 2025/6/27 15:12:11

Python快速入门

简单易懂Python入门

爬虫流程

获取网页内容：HTTP请求
解析网页内容：Requst库、HTML结果、Beautiful Soup库
储存和分析数据

什么是HTTP请求和响应

在这里插入图片描述

如何用Python Requests发送请求

下载pip
macos系统下载：pip3 install requests

在这里插入图片描述
通过第二行进行伪装为浏览器请求

实践

import requests
headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.1 Safari/605.1.15"
}
response = requests.get("https://movie.douban.com/top250",headers=headers)print(response.text)

什么是HTML网页结构？

在这里插入图片描述

HTML常见标签

:链接

![在这里插入图片描述](https://img-blog.csdnimg.cn/48567ae1276e494e8f03b3035aa9aa56.png) # Beautiful Soup

pip3 install bs4

from bs4 import BeautifulSoup
import requests
content = requests.get("http://books.toscrape.com/").textsoup = BeautifulSoup(content,"html.parser")
all_prices = soup.findAll("p",attrs={"class","price_color"})
for price in all_prices:print(price.string[2:])

实战

import requests
from bs4 import BeautifulSoup
headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.1 Safari/605.1.15"
}
for start_num in range(0,250,25):response = requests.get(f"https://movie.douban.com/top250?start={start_num}", headers=headers)html = response.textsoup = BeautifulSoup(html, "html.parser")all_titles = soup.findAll("span", attrs={"class", "title"})for title in all_titles:title_string = title.stringif "/" not in title_string:print(title_string)