requests.get 的获取的网页源代码和 chrome 上的不一致

DavidLee
  • 4
新手上路,请多包涵

我使用 Mac 的终端编写 Python 文件,使用 requeusts.get 函数访问豆瓣读书网址,但获取的网页源代码和chrome 显示的不一致,后访问了百度等页面,但还是不一致,希望各位大神能帮我解决这个问题。代码如下:

#!usr/bin/env python3
#-*- coding: utf-8 -*-
import requests
import re
content = requests.get('https://book.douban.com/').text
print(content)

获取的网址信息如下

,最终发生不幸。一时间,抑郁症、自杀倾向等敏感词汇被推到...
                <a href="https://book.douban.com/review/8696260/">(全文)</a>
            </div>
        </div>
    </div>



    <div class="review ">
        <div class="review-hd">
            <a href="https://book.douban.com/review/8664663/">
              <img class="lazy"
              data-original="https://img3.doubanio.com/mpic/s28041493.jpg"
              src="https://img3.doubanio.com/f/book/54a69da17972770faffce1f758c820ecb8e440af/pics/book/grey.gif"
              alt="西方对中国抗战的认识,这是一个正确的开始" />
            </a>
        </div>
        <div class="review-bd">
            <h3><a href="https://book.douban.com/review/8664663/">西方对中国抗战的认识,这是一个正确的开始</a></h3>
            <div class="review-meta">
                <a href="https://www.douban.com/people/138228322/">遥远的星空</a> 评论
                <a href="https://book.douban.com/review/8664663/">《中国,被遗忘的盟友》</a>

                <span class="allstar50"></span>
            </div>
            <div class="review-content">
                假如问一个问题:中国的抗日战争一共打了多少年?不少人头脑中曾经最熟悉的一个数字一定是8年,是从1937年的“七·七”事变开始至1945年8月15日日本天皇宣布无条件投降止。但这个数字,之前应该加一个限定语——...
                <a href="https://book.douban.com/review/8664663/">(全文)</a>
            </div>
        </div>
    </div>



    <div class="review last">
        <div class="review-hd">
            <a href="https://book.douban.com/review/8720532/">
              <img class="lazy"
              data-original="https://img3.doubanio.com/mpic/s29482391.jpg"
              src="https://img3.doubanio.com/f/book/54a69da17972770faffce1f758c820ecb8e440af/pics/book/grey.gif"
              alt="要什么自行车,一言不合就流学" />
            </a>
        </div>
        <div class="review-bd">
            <h3><a href="https://book.douban.com/review/8720532/">要什么自行车,一言不合就流学</a></h3>
            <div class="review-meta">
                <a href="https://www.douban.com/people/136030628/">阮宁的林林</a> 评论
                <a href="https://book.douban.com/review/8720532/">《流学的一年》</a>

                <span class="allstar40"></span>
            </div>
            <div class="review-content">
                身体和心灵,总有一个在路上,既然选择了远方,便只顾风雨兼程。和我一起,看《流学的一年》,领略一下浪迹天涯的梦想和一颗在路上的心。 拿到这本书,我开始怀疑我自己,是不是看的书太少,这种形式的书,记忆中...
                <a href="https://book.douban.com/review/8720532/">(全文)</a>
            </div>
        </div>
    </div>

          </div>
      </div>




</div>
      <div class="aside">




  <!-- douban ad begin -->
  <div id="dale_book_home_top_right" class="s ad-placeholder"
    data-dstat-areaid="51" data-dstat-mode="click,expose"
    style="margin-top: 30px;"></div>
  <!-- douban ad end -->

  <!-- douban ad begin -->
  <div id="dale_book_home_top_right2" class="ad-placeholder"></div>
  <!-- douban ad end -->






  <h2 class=''>
    <span class="">热门标签</span>
      <span class="link-more">
        <a class="" href="/tag/?view=type&amp;icn=index-sorttags-all"
          >所有热门标签»</a>
      </span>
  </h2>

  <ul class="hot-tags-col5 s" data-dstat-areaid="54" data-dstat-mode="click,expose">



    <li>
        <ul class="clearfix">
            <li class="tag_title">
                文学
            </li>
            <li>
                <a href="/tag/小说" class="tag">小说</a>
            </li>
            <li>
                <a href="/tag/随笔" class="tag">随笔</a>
            </li>
            <li>
                <a href="/tag/日本文学" class="tag">日本文学</a>
            </li>
            <li class="last">
                  <a href="/tag/散文" class="tag">散文</a>
            </li>
            <li>
                <a href="/tag/诗歌" class="tag">诗歌</a>
            </li>
            <li>
                <a href="/tag/童话" class="tag">童话</a>
            </li>
            <li>
                <a href="/tag/名著" class="tag">名著</a>
            </li>
            <li class="last">
                  <a href="/tag/港台" class="tag">港台</a>
            </li>
            <li class="last">
                  <a href="/tag/?view=type&amp;icn=index-sorttags-hot#文学" class="tag more_tag">更多》</a>
            </li>
        </ul>
    </li>




    <li>
        <ul class="clearfix">
            <li class="tag_title">
                流行
            </li>
            <li>
                <a href="/tag/漫画" class="tag">漫画</a>
            </li>
            <li>
                <a href="/tag/推理" class="tag">推理</a>
            </li>
            <li>
                <a href="/tag/绘本" class="tag">绘本</a>
            </li>
            <li class="last">
                  <a href="/tag/青春" class="tag">青春</a>
            </li>
            <li>
                <a href="/tag/科幻" class="tag">科幻</a>
            </li>
            <li>
                <a href="/tag/言情" class="tag">言情</a>
            </li>
            <li>
                <a href="/tag/武侠" class="tag">武侠</a>
            </li>
            <li class="last">
                  <a href="/tag/奇幻" class="tag">奇幻</a>
            </li>
            <li class="last">
                  <a href="/tag/?view=type&amp;icn=index-sorttags-hot#流行" class="tag more_tag">更多》</a>
            </li>
        </ul>
    </li>




    <li>
        <ul class="clearfix">
            <li class="tag_title">
                文化
            </li>
            <li>
                <a href="/tag/历史" class="tag">历史</a>
            </li>
            <li>
                <a href="/tag/哲学" class="tag">哲学</a>
            </li>
            <li>
                <a href="/tag/传记" class="tag">传记</a>
            </li>
            <li class="last">
                  <a href="/tag/设计" class="tag">设计</a>
            </li>
            <li>
                <a href="/tag/建筑" class="tag">建筑</a>
            </li>
            <li>
                <a href="/tag/电影" class="tag">电影</a>
            </li>
            <li>
                <a href="/tag/回忆录" class="tag">回忆录</a>
            </li>
            <li class="last">
                  <a href="/tag/音乐" class="tag">音乐</a>
            </li>
            <li class="last">
                  <a href="/tag/?view=type&amp;icn=index-sorttags-hot#文化" class="tag more_tag">更多》</a>
            </li>
        </ul>
    </li>




    <li>
        <ul class="clearfix">
            <li class="tag_title">
                生活
            </li>
            <li>
                <a href="/tag/旅行" class="tag">旅行</a>
            </li>
            <li>
                <a href="/tag/励志" class="tag">励志</a>
            </li>
            <li>
                <a href="/tag/职场" class="tag">职场</a>
            </li>
            <li class="last">
                  <a href="/tag/美食" class="tag">美食</a>
            </li>
            <li>
                <a href="/tag/教育" class="tag">教育</a>
            </li>
            <li>
                <a href="/tag/灵修" class="tag">灵修</a>
            </li>
            <li>
                <a href="/tag/健康" class="tag">健康</a>
            </li>
            <li class="last">
                  <a href="/tag/家居" class="tag">家居</a>
            </li>
            <li class="last">
                  <a href="/tag/?view=type&amp;icn=index-sorttags-hot#生活" class="tag more_tag">更多》</a>
            </li>
        </ul>
    </li>




    <li>
        <ul class="clearfix">
            <li class="tag_title">
                经管
            </li>
            <li>
                <a href="/tag/经济学" class="tag">经济学</a>
            </li>
            <li>
                <a href="/tag/管理" class="tag">管理</a>
            </li>
            <li>
                <a href="/tag/商业" class="tag">商业</a>
            </li>
            <li class="last">
                  <a href="/tag/金融" class="tag">金融</a>
            </li>
            <li>
                <a href="/tag/营销" class="tag">营销</a>
            </li>
            <li>
                <a href="/tag/理财" class="tag">理财</a>
            </li>
            <li>
                <a href="/tag/股票" class="tag">股票</a>
            </li>
            <li class="last">
                  <a href="/tag/企业史" class="tag">企业史</a>
            </li>
            <li class="last">
                  <a href="/tag/?view=type&amp;icn=index-sorttags-hot#经管" class="tag more_tag">更多》</a>
            </li>
        </ul>
    </li>




    <li>
        <ul class="clearfix">
            <li class="tag_title">
                科技
            </li>
            <li>
                <a href="/tag/科普" class="tag">科普</a>
            </li>
            <li>
                <a href="/tag/互联网" class="tag">互联网</a>
            </li>
            <li>
                <a href="/tag/编程" class="tag">编程</a>
            </li>
            <li class="last">
                  <a href="/tag/交互设计" class="tag">交互设计</a>
            </li>
            <li>
                <a href="/tag/算法" class="tag">算法</a>
            </li>
            <li>
                <a href="/tag/通信" class="tag">通信</a>
            </li>
            <li>
                <a href="/tag/神经网络" class="tag">神经网络</a>
            </li>
            <li class="last">
                  <a href="/tag/?view=type&amp;icn=index-sorttags-hot#科技" class="tag more_tag">更多》</a>
            </li>
        </ul>
    </li>

  </ul>


  <!-- douban ad begin -->
  <div id="dale_book_homepage_right_bottom" class="ad-placeholder"></div>
  <!-- douban ad end -->


<div class="section weekly-top">
    <div class="hd">
        <h2>畅销图书榜</h2>
    </div>
    <div class="bd">
        <ul class="nav-vendor">
            <li class="on book-chart-hd" id="jd-book-chart-hd">
                <img src="https://img3.doubanio.com/f/book/7fd9bf017a2b6c0349981f25700e2b71c12df7d0/pics/book/partner/jd_chart_hover.png" width="18" height="18"/>
                <span>京东</span>
            </li>
            <li class="book-chart-hd" id="dangdang-book-chart-hd">
                <img src="https://img3.doubanio.com/f/book/31b437b71ced2d15e1e110d6be26b070da8c0d4a/pics/book/partner/dangdang_chart.png" width="18" height="18"/>
                <span>当当</span>
            </li>
        </ul>
            <ul class="list list-ranking">

                <li class="item">
                    <span class="rank-num">1.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/27084262/" class="name" target="_blank">我的前半生</a>
                        <div class="author">亦舒</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=2690&amp;pos=1&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEQRVG14RByJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQERB1UeXxAdS0IJRmtAZ0VeLFopC2BrXwtIIRBqSQUsHz9lDh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sRAxMFVRpTFgMQA1MrXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsWARIHUB9eCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=27084262" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">2.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/25862578/" class="name" target="_blank">解忧杂货店</a>
                        <div class="author">[日] 东野圭吾</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=2730&amp;pos=2&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEgJRGFIQAyJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQIXA1YSXhQdS0IJRmt3X0RALUYbTmFGVzRTE0VQSUBLUF5TDh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sQCxsDUBpfEQAXD1YrXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsVBxYEXB5aCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=25862578" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">3.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/26610864/" class="name" target="_blank">浮生六记</a>
                        <div class="author">沈复</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=2200&amp;pos=3&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEgFRHVgUByJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQIUA1MYWhAdS0IJRmsSVEljCGxZE2BmBAtcAWV1G1ssZVtDDh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sQCxsDUBpeEgMUBl0rXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsVBBYBVhpeCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=26610864" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">4.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/25747921/" class="name" target="_blank">从你的全世界路过</a>
                        <div class="author">张嘉佳</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=2320&amp;pos=4&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEgVXEl0TAiJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQIQBVwdXRUdS0IJRmt2WmZsE30rXWAXeTZmImlgYmYyGztDDh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sQCxsDUBpcFAsQD10rXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsVABAOUx1bCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=25747921" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">5.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/27084268/" class="name" target="_blank">幸运星</a>
                        <div class="author">亦舒</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=2690&amp;pos=5&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEQZdHFISCiJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQETD1ISXB0dS0IJRmt1Q2F3D209E2d3R09nImJwahk0YTNTDh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sRAxIDVhpeEAoUB1QrXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsWAxoAXBxTCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=27084268" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">6.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/27101705/" class="name" target="_blank">2018 肖秀荣考研政治命题人1000题(套装上下册)</a>
                        <div class="author">肖秀荣</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=3300&amp;pos=6&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEQdcHlsSASJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQESDlAbXBYdS0IJRmtrdkZRVk8zQGdtVzNAWWZybUwDGx1lDh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sRAxICXBtSEAUbAVwrXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsWAhsCVRxYCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=27101705" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">7.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/26356948/" class="name" target="_blank">摆渡人</a>
                        <div class="author">[英]克莱儿·麦克福尔</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=1800&amp;pos=7&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEgBcHVIRCyJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQIVDlMSXxwdS0IJRmt1YGJUDkcmC2dQHQcTMF5hb34AeVxTDh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sQCxsDUBpeHQcVAVcrXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsVBRsBXB9SCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=26356948" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">8.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/27084283/" class="name" target="_blank">人淡如菊</a>
                        <div class="author">亦舒</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=2620&amp;pos=8&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEQZdHFITAiJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQETD1ISXRUdS0IJRmtzYWViCE9BFGBAQ09%252FC0NaSQ8LGTJ1Dh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sRAxMDUhJdFQEQAFwrXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsWAxoAXB1bCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=27084283" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">9.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/26974162/" class="name" target="_blank">孤独是生命的礼物</a>
                        <div class="author">余光中/林清玄/白先勇</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=2590&amp;pos=9&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEQZdGF8VCyJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQETD1YfWxwdS0IJRmt0BHBcAkIrfGFgYStcGEFBG1kObl9DDh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sQCxsDUBtYEgIbA1IrXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsWAxoEURtSCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=26974162" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">10.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/26852422/" class="name" target="_blank">古汉语常用字字典</a>
                        <div class="author">蒋绍愚/唐作藩/张万起/宋绍年/李树青</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=jingdong&amp;srcpage=bestseller&amp;price=2790&amp;pos=10&amp;url=http%3A%2F%2Funion-click.jd.com%2Fjdc%3Fe%3D%26p%3DAyIOZR5aEQISA1AYUyUCEg9WGl8WAiJDCkMFSjJLQhBaUAscSkIBR0ROVw1VC0dFFQIaBFQfWBUdS0IJRmt9e21jUk04HGdLW1FgHUFAEHkpGVNTDh43Uh5cEwUbBFIrWxEDFgNlK1sUMkBpja3tzaejG4Gx1MCKhTdUK1sQCxsDUBtbHQEVAlIrXCXbkrCDucnMnJjS3YxrJTIi%26t%3DW1dCFBBFC1pXUwkEAEAdQFkJBVsVChEGURhbCltXWwg%253D&amp;srcsubj=&amp;type=bkbuy&amp;subject=26852422" target="_blank"><span class="buy-button">去购买</span></a>
                </li>
            </ul>
            <ul class="list list-ranking">

                <li class="item">
                    <span class="rank-num">1.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/26647769/" class="name" target="_blank">人间失格</a>
                        <div class="author">太宰治</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=1730&amp;pos=1&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s26647769%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D23761145&amp;srcsubj=&amp;type=bkbuy&amp;subject=26647769" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">2.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/25862578/" class="name" target="_blank">解忧杂货店</a>
                        <div class="author">[日] 东野圭吾</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=2730&amp;pos=2&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s25862578%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D23464478&amp;srcsubj=&amp;type=bkbuy&amp;subject=25862578" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">3.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/1770782/" class="name" target="_blank">追风筝的人</a>
                        <div class="author">[美] 卡勒德·胡赛尼</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=1689&amp;pos=3&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s1770782%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D23274638&amp;srcsubj=&amp;type=bkbuy&amp;subject=1770782" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">4.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/10554308/" class="name" target="_blank">白夜行</a>
                        <div class="author">东野圭吾</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=2730&amp;pos=4&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s10554308%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D22935553&amp;srcsubj=&amp;type=bkbuy&amp;subject=10554308" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">5.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/4913064/" class="name" target="_blank">活着</a>
                        <div class="author">余华</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=1100&amp;pos=5&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s4913064%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D22864990&amp;srcsubj=&amp;type=bkbuy&amp;subject=4913064" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">6.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/26356948/" class="name" target="_blank">摆渡人</a>
                        <div class="author">[英]克莱儿·麦克福尔</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=1700&amp;pos=6&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s26356948%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D23694647&amp;srcsubj=&amp;type=bkbuy&amp;subject=26356948" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">7.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/26926101/" class="name" target="_blank">我的第一本地理启蒙书</a>
                        <div class="author">郑利强 著/段虹 绘</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=2350&amp;pos=7&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s26926101%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D23967348&amp;srcsubj=&amp;type=bkbuy&amp;subject=26926101" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">8.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/26666472/" class="name" target="_blank">天才在左 疯子在右(完整版)</a>
                        <div class="author">高铭</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=2750&amp;pos=8&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s26666472%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D23822200&amp;srcsubj=&amp;type=bkbuy&amp;subject=26666472" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">9.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/26865749/" class="name" target="_blank">所谓情商高,就是会说话</a>
                        <div class="author">佐佐木圭一</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=2300&amp;pos=9&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s26865749%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D24026269&amp;srcsubj=&amp;type=bkbuy&amp;subject=26865749" target="_blank"><span class="buy-button">去购买</span></a>
                </li>

                <li class="item">
                    <span class="rank-num">10.</span>
                    <div class="book-info">
                        <a href="https://book.douban.com/subject/11524204/" class="name" target="_blank">围城</a>
                        <div class="author">钱锺书</div>
                    </div>
                    <a href="https://book.douban.com/link2/?pre=0&amp;vendor=dangdang&amp;srcpage=bestseller&amp;price=2200&amp;pos=10&amp;url=http%3A%2F%2Funion.dangdang.com%2Ftransfer.php%3Ffrom%3DP-306226-0-s11524204%26backurl%3Dhttp%3A%2F%2Fproduct.dangdang.com%2Fproduct.aspx%3Fproduct_id%3D22819334&amp;srcsubj=&amp;type=bkbuy&amp;subject=11524204" target="_blank"><span class="buy-button">去购买</span></a>
                </li>
            </ul>
    </div>
</div>






  <div class="block5">


  <h2 class=''>
    <span class="">豆瓣图书250</span>
      <span class="link-more">
        <a class="" href="/top250?icn=index-book250-all"
          >更多»</a>
      </span>
  </h2>


    <div class="content clearfix s" id="book_rec"
      data-dstat-areaid="58"
      data-dstat-mode="click,expose">


    <dl>
      <dt>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/25985021/?icn=index-book250-subject">
          <img src="https://img3.doubanio.com/spic/s27814883.jpg" class="m_sub_img">
        </a>
      </dt>
      <dd>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/25985021/?icn=index-book250-subject">
          人类简史
        </a>
        <p class="extra-info">


  <span class="meta-label">
    有电子书
  </span>

        </p>
      </dd>
    </dl>

    <dl>
      <dt>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/2022979/?icn=index-book250-subject">
          <img src="https://img1.doubanio.com/spic/s5813979.jpg" class="m_sub_img">
        </a>
      </dt>
      <dd>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/2022979/?icn=index-book250-subject">
          喜宝
        </a>
        <p class="extra-info">


        </p>
      </dd>
    </dl>

    <dl>
      <dt>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/2382705/?icn=index-book250-subject">
          <img src="https://img3.doubanio.com/spic/s2868780.jpg" class="m_sub_img">
        </a>
      </dt>
      <dd>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/2382705/?icn=index-book250-subject">
          不朽
        </a>
        <p class="extra-info">


        </p>
      </dd>
    </dl>
      <div class="clearfix rr" style="width:100%"></div>

    <dl>
      <dt>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/3506871/?icn=index-book250-subject">
          <img src="https://img1.doubanio.com/spic/s3635269.jpg" class="m_sub_img">
        </a>
      </dt>
      <dd>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/3506871/?icn=index-book250-subject">
          曾有一个人,爱我如生命
        </a>
        <p class="extra-info">


        </p>
      </dd>
    </dl>

    <dl>
      <dt>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/3259440/?icn=index-book250-subject">
          <img src="https://img3.doubanio.com/spic/s4610502.jpg" class="m_sub_img">
        </a>
      </dt>
      <dd>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/3259440/?icn=index-book250-subject">
          白夜行
        </a>
        <p class="extra-info">


        </p>
      </dd>
    </dl>

    <dl>
      <dt>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/1016523/?icn=index-book250-subject">
          <img src="https://img3.doubanio.com/spic/s1650313.jpg" class="m_sub_img">
        </a>
      </dt>
      <dd>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/1016523/?icn=index-book250-subject">
          告别薇安
        </a>
        <p class="extra-info">


        </p>
      </dd>
    </dl>
      <div class="clearfix rr" style="width:100%"></div>

    <dl>
      <dt>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/1013502/?icn=index-book250-subject">
          <img src="https://img1.doubanio.com/spic/s2241708.jpg" class="m_sub_img">
        </a>
      </dt>
      <dd>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/1013502/?icn=index-book250-subject">
          钢铁是怎样炼成的
        </a>
        <p class="extra-info">


        </p>
      </dd>
    </dl>

    <dl>
      <dt>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/1045818/?icn=index-book250-subject">
          <img src="https://img3.doubanio.com/spic/s2153661.jpg" class="m_sub_img">
        </a>
      </dt>
      <dd>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/1045818/?icn=index-book250-subject">
          苏菲的世界
        </a>
        <p class="extra-info">


        </p>
      </dd>
    </dl>

    <dl>
      <dt>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/1016300/?icn=index-book250-subject">
          <img src="https://img1.doubanio.com/spic/s1513378.jpg" class="m_sub_img">
        </a>
      </dt>
      <dd>
        <a onclick="moreurl(this, {from:'top'})" href="https://book.douban.com/subject/1016300/?icn=index-book250-subject">
          梦里花落知多少
        </a>
        <p class="extra-info">


        </p>
      </dd>
    </dl>
      <div class="clearfix rr" style="width:100%"></div>

    </div>
  </div>


  <div class="contact">
    <h2>联系我们</h2>
    <ul class="apply-links">
      <li>
        合作联系:<img src="https://img3.doubanio.com/f/book/6a8e1160fbccbaa8c13ddf72e22145056bda0ba4/pics/book/email_book.png" />
      </li>
    </ul>
  </div>


  <div class="contact mod">
      <h2>关注我们</h2>
      <ul class="embassy-list clearfix">
          <li>
              <a href="https://site.douban.com/book/" target="_blank" class="icon-embassy-site"></a>
              <a href="https://site.douban.com/book/" target="_blank" class="primary-link">豆瓣小站</a>
          </li>
          <li>
              <a href="http://weibo.com/doubandushu" target="_blank" class="icon-embassy-weibo"></a>
              <a href="http://weibo.com/doubandushu" target="_blank" class="primary-link">微博</a>
          </li>
          <li>
              <a class="icon-embassy-weixin">
                  <div class="hover"><img class="home-qrcode" src="https://img3.doubanio.com/f/book/ee2d0bf8adc46ee0e1a53c28abf41a7a3b589e54/pics/book/[email protected]"></div>
              </a>
              <a class="primary-link" href="javascript:;">微信</a>
          </li>
      </ul>
  </div>
  <!-- douban app begin -->
  <div class="s" data-dstat-areaid="60" data-dstat-mode="click,expose">
    <!-- douban ad begin -->
    <div id="dale_book_home_inner_middle"></div>
    <div id="dale_book_home_download_middle"></div>
    <!-- douban ad end -->
  </div>
  <!-- douban app end -->

  <!-- douban ad begin -->
  <div id="dale_book_home_bottem_right" class="ad-placeholder"></div>
  <!-- douban ad end -->

      </div>
      <div class="extra">

      </div>
    </div>
  </div>


  <div id="footer">

<span id="icp" class="fleft gray-link">
    &copy; 2005-2017 douban.com, all rights reserved 北京豆网科技有限公司
</span>

<a href="https://www.douban.com/hnypt/variformcyst.py" style="display: none;"></a>

<span class="fright">
    <a href="https://www.douban.com/about">关于豆瓣</a>
    · <a href="https://www.douban.com/jobs">在豆瓣工作</a>
    · <a href="https://www.douban.com/about?topic=contactus">联系我们</a>
    · <a href="https://www.douban.com/about?policy=disclaimer">免责声明</a>

    · <a href="https://help.douban.com/?app=book" target="_blank">帮助中心</a>
    · <a href="https://book.douban.com/library_invitation">图书馆合作</a>
    · <a href="https://www.douban.com/doubanapp/">移动应用</a>
    · <a href="https://www.douban.com/partner/">豆瓣广告</a>
</span>

    <script src="https://img3.doubanio.com/f/book/7495d157b1564be6aae1d21c79280b5ea374400a/js/book/lib/do/init.js"
      data-cfg-corelib="https://img3.doubanio.com/f/book/0495cb173e298c28593766009c7b0a953246c5b5/js/book/lib/jquery/jquery.js">
    </script>
  </div>

  </div>
  <script type="text/javascript" src="https://img3.doubanio.com/misc/mixed_static/33734cec318e690e.js"></script>
  <!-- mako -->

  <!-- douban ad begin -->







<script type="text/javascript">
    (function (global) {
        var newNode = global.document.createElement('script'),
            existingNode = global.document.getElementsByTagName('script')[0],
            adSource = '//erebor.douban.com/',
            userId = '',
            browserId = 'uofiXAAIhYo',
            criteria = '3:/',
            preview = '',
            debug = false,
            adSlots = ['dale_book_home_top_right', 'dale_book_home_top_right2', 'dale_book_home_left_middle', 'dale_book_home_bottem_right', 'dale_book_home_top_super_banner', 'dale_book_home_left_top', 'dale_book_homepage_right_bottom', 'dale_book_home_inner_middle', 'dale_book_home_download_middle'];

        global.DoubanAdRequest = {src: adSource, uid: userId, bid: browserId, crtr: criteria, prv: preview, debug: debug};
        global.DoubanAdSlots = (global.DoubanAdSlots || []).concat(adSlots);

        newNode.setAttribute('type', 'text/javascript');
        newNode.setAttribute('src', 'https://img3.doubanio.com/f/shire/a01a8a95c80c60b76f4d29cd788399ff7ac5e00e/js/ad.js');
        newNode.setAttribute('async', true);
        existingNode.parentNode.insertBefore(newNode, existingNode);
    })(this);
</script>










  <!-- douban ad end -->
  <script>
    Do(function() {
      $('.click-track').on('click', function(e) {
        e.preventDefault()
        var name = $(this).data('name')
        moreurl(this, { from: name })
      })
    })
  </script>





<script type="text/javascript">
  var _paq = _paq || [];
  _paq.push(['trackPageView']);
  _paq.push(['enableLinkTracking']);
  (function() {
    var p=(('https:' == document.location.protocol) ? 'https' : 'http'), u=p+'://fundin.douban.com/';
    _paq.push(['setTrackerUrl', u+'piwik']);
    _paq.push(['setSiteId', '100001']);
    var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
    g.type='text/javascript';
    g.defer=true;
    g.async=true;
    g.src=p+'://s.doubanio.com/dae/fundin/piwik.js';
    s.parentNode.insertBefore(g,s);
  })();
</script>

<script type="text/javascript">
var setMethodWithNs = function(namespace) {
  var ns = namespace ? namespace + '.' : ''
    , fn = function(string) {
        if(!ns) {return string}
        return ns + string
      }
  return fn
}

var gaWithNamespace = function(fn, namespace) {
  var method = setMethodWithNs(namespace)
  fn.call(this, method)
}

var _gaq = _gaq || []
  , accounts = [
      { id: 'UA-7019765-1', namespace: 'douban' }
    , { id: 'UA-7019765-16', namespace: '' }
    ]
  , gaInit = function(account) {
      gaWithNamespace(function(method) {
        gaInitFn.call(this, method, account)
      }, account.namespace)
    }
  , gaInitFn = function(method, account) {
      _gaq.push([method('_setAccount'), account.id])


  _gaq.push([method('_addOrganic'), 'google', 'q'])
  _gaq.push([method('_addOrganic'), 'baidu', 'wd'])
  _gaq.push([method('_addOrganic'), 'soso', 'w'])
  _gaq.push([method('_addOrganic'), 'youdao', 'q'])
  _gaq.push([method('_addOrganic'), 'so.360.cn', 'q'])
  _gaq.push([method('_addOrganic'), 'sogou', 'query'])
  if (account.namespace) {
    _gaq.push([method('_addIgnoredOrganic'), '豆瓣'])
    _gaq.push([method('_addIgnoredOrganic'), 'douban'])
    _gaq.push([method('_addIgnoredOrganic'), '豆瓣网'])
    _gaq.push([method('_addIgnoredOrganic'), 'www.douban.com'])
  }

      if (account.namespace === 'douban') {
        _gaq.push([method('_setDomainName'), '.douban.com'])
      }

        _gaq.push([method('_setCustomVar'), 1, 'responsive_view_mode', 'desktop', 3])

        _gaq.push([method('_setCustomVar'), 2, 'login_status', '0', 2]);

      _gaq.push([method('_trackPageview')])
    }

for(var i = 0, l = accounts.length; i < l; i++) {
  var account = accounts[i]
  gaInit(account)
}


;(function() {
    var ga = document.createElement('script');
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    ga.setAttribute('async', 'true');
    document.documentElement.firstChild.appendChild(ga);
})()
</script>









  <!-- sindar23d-docker-->

  </body>
</html>

我不懂为什么会这样子,Google 和百度后也并没有发现相关问题,希望各位大神帮忙解决一下。谢谢,感激不尽。

回复
阅读 14.2k
2 个回答
防暴队大盾
  • 2
新手上路,请多包涵

我也是新手,今天正好也碰到差不多的疑惑,给你一个我的解决办法,P.S.我是在get百度的时候遇到的问题

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) Chrome/50.0.2661.102'}
payload = {'wd': 'GitHub'}  # 搜索的关键字是GitHub
url = 'https://www.baidu.com/s'

res = requests.get(url, params=payload, headers=headers)

需要加headers和payload参数。如果还是不明白,可以去查查什么是HTTP Headers
在Chrome浏览器中查看HTTP报文的头部信息

首先。没有搞懂你所谓的不一致是什么不一致
首先要考虑有一些网站是经过请求,然后js处理再渲染出的页面
再者,要考虑你的requests请求头部。是否少了。(这得你自己去尝试)
我以前也曾经采集过豆瓣图书资源,使用代理ip多线程多进程直接跑的。
头部是多头部随机使用。接下来就是分析页面元素提取信息。
对于一些需要js渲染之后处理后才出现的网页,可以考虑抓取它的数据接口,或者使用PhantomJS Selenium 这类的。将它伪装成一个真正的"浏览器行为"
希望回答能帮到你

撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进
宣传栏