加快Yahoo Finance上的网页抓取

最后发布: 2018-08-30 12:52:31


问题

我正在使用Yahoo Finance和带beautifulsoup lib的python在Yahoo Finance上进行网络抓取测试,但运行速度非常慢。 我如何加快速度? 报废网站是否合法?

quotes = []
quote = {}
symbolList = ['SPY', 'AAPL']

for symbol in symbolList:
    url = ('http://finance.yahoo.com/quote/%s?p=%s' % (symbol, symbol))
    page = requests.get(url)
    html = page.text
    soup = BeautifulSoup(html, 'html.parser')

    span_latest_price = soup.find("span", {"class": "Trsdu(0.3s) Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(b)"})
    span_quote_name = soup.find('h1', {'class': 'D(ib) Fz(16px) Lh(18px)'})

    last_price = span_latest_price.text if span_latest_price else ''
    name = span_quote_name.text if span_quote_name else ''
    quote = {'name': name, 'lastPrice': last_price}
    quotes.append(quote)
python web-scraping yahoo-finance