🌟 从零构建另类数据采集管线：API 请求、爬虫框架、以及数据伦理红线。中小学生色系 · 快乐学习！
429 处理import requests
url = "https://api.binance.com/api/v3/ticker/price"
params = {"symbol": "BTCUSDT"}
resp = requests.get(url, params=params)
data = resp.json()
print(f"BTC 价格: {data['price']}")import requests
from bs4 import BeautifulSoup
url = "https://example.com/data"
r = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
soup = BeautifulSoup(r.text, "html.parser")
for item in soup.select(".data-row"):
print(item.get_text(strip=True))robots.txt| 特性 | Requests + BeautifulSoup | Scrapy 框架 |
|---|---|---|
| 上手难度 | ⭐ 低 (适合快速原型) | ⭐⭐⭐ 中高 (结构完善) |
| 并发/异步 | 需自行实现 (ThreadPool) | 内置异步 Twisted, 高性能 |
| 中间件/扩展 | 无 | 丰富的中间件、Pipeline、扩展 |
| 数据去重 | 手动 | 内置去重 (RFPDupeFilter) |
| 适用场景 | 小规模、单页面、API 调试 | 大规模、多页面、生产级爬虫 |
# news_spider.py
import scrapy
class NewsSpider(scrapy.Spider):
name = "financial_news"
start_urls = ["https://example-finance-news.com/latest"]
def parse(self, response):
for article in response.css("div.news-item"):
yield {
"title": article.css("h2::text").get(),
"date": article.css("span.date::text").get(),
"sentiment": self.estimate_sentiment(article.css("p::text").get())
}
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield scrapy.Request(url=next_page, callback=self.parse)🧑⚖️ 数据采集伦理 · 黄金法则
robots.txt 及 Crawl-delay