Web Spider -- 做一个简单的爬虫 （愿给您启示）

Web Spider

一个简单的爬虫

代码已托管

这里有一个简单的例子

public class SpiderZhiHuDemo {


    private Logger logger = LoggerFactory.getLogger(SpiderZhiHuDemo.class);


    /**
     * 根据提供的种子爬取数据
     */
    public void initSeedDemo() {
        String seed = "https://www.zhihu.com/question/28629300";
        new Spider() {
            {
                downloader = new ProxyDownloader(new JSoupDownloader());
                extractPattern = "https://www.zhihu.com/question/.*"; // 指定对应的抓取规则
            }
        }.init(seed).start();
    }

    /**
     * 自己定义 抓取的url链接 (简单的)
     */
    public void easyStreamDemo(){
        String url = "https://www.zhihu.com/question/";
        new Spider() {
            {
                downloader = new ProxyDownloader(new JSoupDownloader());
                pipeline = (title, page) -> {
                    logger.info("url: {} , title : {} ", page.getRequest(), title);
                };
            }
        }.init(Stream.iterate(28629300, i -> i + 1).limit(60).map(i -> url + i).parallel()).start();
    }
} 

控制台打印结果

18:35:54.278 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629326 , title : 建筑工地上的青年如何自我成长？ - 知乎 
18:35:54.525 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629319 , title : 国内专做进口行业的公司多不？（不包括货代公司） - 知乎 
18:35:54.553 [main] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629338 , title : 如何有效地进行后天性丰胸？ - 知乎 
18:35:54.881 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629328 , title : Android开发中list和适配器是怎样关联的？适配器具体作用是什么？list可以做什么？ - 知乎 
18:35:55.168 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629321 , title : 互联网公司，粉丝运营如何做？ - 知乎 
18:35:55.317 [ForkJoinPool.commonPool-worker-2] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629311 , title : 为什么一些大公司的特喜欢装X？ - 知乎 
18:35:55.506 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629315 , title : 大二中 刚开学 一闲下来却总想给家里打电话 无法控制 这是为什么？ - 知乎 
18:35:55.664 [main] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629342 , title : 中国雷达哪家强？ - 知乎 
18:35:56.113 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629325 , title : 美丽加芬蜗牛系列怎么样? - 知乎 
18:35:56.472 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629300 , title : "截石位"的说法是从哪里来的或者怎么翻译来的？ - 知乎 
18:35:56.713 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629304 , title : 我想考南京大学的历史系中国史的其中一组 就是01～09方向 请问 是按组命题的么? - 知乎 
18:35:56.803 [ForkJoinPool.commonPool-worker-2] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629353 , title : K线的精髓在哪里？ - 知乎 
18:35:57.022 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629301 , title : 华为加班到底有多恐怖？ - 知乎 
18:35:57.143 [main] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629335 , title : 人体的哪些疾病（症状）可以自行诊断？ - 知乎 
18:35:57.167 [ForkJoinPool.commonPool-worker-2] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629354 , title : 既然ISIS这么罪大恶极，那么联合国为什么还不派兵剿灭? - 知乎 
18:35:57.333 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629302 , title : 谁能帮我起个专门针对声乐培训，演出，小明星，小歌手教学工作室名字？能让人很快记住大气的名字？谢谢！？ - 知乎 
18:35:58.014 [ForkJoinPool.commonPool-worker-2] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629357 , title : 大学毕业后留有一大堆书不舍得丢掉，每到搬家时就会发愁了，请问如何处理不会觉得可惜而且便捷？ - 知乎 
18:35:58.022 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629349 , title : 华为研发员工的发展，待遇，福利？和互联网公司比呢？ - 知乎 
18:35:58.209 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629332 , title : 朋友圈疯传: 求扩散！一个小孩的生命结束了就是因为她把手机充电接口放到嘴里，而另一头还在插线板上。 求解? - 知乎