python爬虫-bs4
python爬虫-bs4
目录
- python爬虫-bs4
- 说明
- 安装
- 导入
- 基础用法
- 解析对象
- 获取文本
- Tag对象
- 获取HTML中的标签内容
- find参数
- 获取标签属性
- 获取所有标签
- 获取标签名
- 嵌套获取
- 子节点和父节点
说明
BeautifulSoup 是一个HTML/XML的解析器，主要的功能也是如何解析和提取 HTML/XML 数据
在爬虫项目中经常会遇到不规范、及其复杂的HTML代码
BeautifulSoup4提供了强大的方法来遍历文档的节点以及根据各种条件搜索和过滤文档中的元素。你可以使用CSS选择器、正则表达式等灵活的方式来定位和提取所需的数据
安装
pip install BeautiifulSoup4
导入
from bs4 import BeautifulSoup
基础用法
解析对象
soup = BeautifulSoup('目标数据','解析器')
目前有三种主流解析器
html.parserlxml(推荐)html5lib
获取文本
获取文本的方法两种方式text和contents
contents：
from bs4 import BeautifulSoupdata = """
<h1>Welcome to BeautifulSoup Practice</h1><div class="article"><h2>Article Title</h2><p>This is a paragraph of text for practicing BeautifulSoup.</p><a href="https://www.example.com">Link to Example Website</a>
"""
soup = BeautifulSoup(data, 'lxml')
print(soup.contents)
# 输出：
"""
[<html><body><h1>Welcome to BeautifulSoup Practice</h1>
<div class="article">
<h2>Article Title</h2>
<p>This is a paragraph of text for practicing BeautifulSoup.</p>
<a href="https://www.example.com">Link to Example Website</a>
</div></body></html>]
"""
text：
print(soup.text)
"""
Welcome to BeautifulSoup PracticeArticle Title
This is a paragraph of text for practicing BeautifulSoup.
Link to Example Website
"""
Tag对象
获取HTML中的标签内容
比如<p> <div>
示例：
print(soup.h2)
# <h2>Article Title</h2>print(soup.h2.text)
# Article Title
find参数
获取class要加下划线，因为在python中它属于关键字，除了class还可以换成任意属性名
data = """
<h1>Welcome to BeautifulSoup Practice</h1><div class="article"><p>This is a paragraph of text for practicing BeautifulSoup.</p></div><div class="ex2"><p>This is a abcd.</p></div>
"""
soup = BeautifulSoup(data, 'lxml')
print(soup.find('div', class_='article'))
获取标签属性
data = ' <p id = "apple">This is a paragraph of text for practicing BeautifulSoup.</p>'
soup = BeautifulSoup(data, 'lxml')
tag = soup.find('p')
print(tag.get('id'))
# apple
获取所有标签
soup = BeautifulSoup(data, 'lxml')
print(soup.find_all('p'))
# [<p>This is a paragraph of text for practicing BeautifulSoup.</p>, <p>This is a abcd.</p>]print(len(soup.find_all('p')))
# 2
括号为空则获取全部标签
获取标签名
print(soup.div.name)
# div
嵌套获取
示例HTML如下
html = '''
<div class="article"><h2>Article Title</h2><p>This is a paragraph of text for practicing BeautifulSoup.</p><p>This is a abcd.</p><a href="https://www.example.com">Link to Example Website</a>
</div>
'''
目标：获取div下的所有p标签内容
print(soup.find('div', class_='article').find_all('p'))
子节点和父节点
soup = BeautifulSoup(data, 'lxml')
# 遍历获取所有父节点
for item in soup.p.parents:print(item)# 遍历获取所有子节点
for i in soup.p.children:print(soup.p.children)