Web Spider
一个简单的爬虫
代码已托管
这里有一个简单的例子
public class SpiderZhiHuDemo {
private Logger logger = LoggerFactory.getLogger(SpiderZhiHuDemo.class);
/**
* 根据提供的种子爬取数据
*/
public void initSeedDemo() {
String seed = "https://www.zhihu.com/question/28629300";
new Spider() {
{
downloader = new ProxyDownloader(new JSoupDownloader());
extractPattern = "https://www.zhihu.com/question/.*"; // 指定对应的抓取规则
}
}.init(seed).start();
}
/**
* 自己定义 抓取的url链接 (简单的)
*/
public void easyStreamDemo(){
String url = "https://www.zhihu.com/question/";
new Spider() {
{
downloader = new ProxyDownloader(new JSoupDownloader());
pipeline = (title, page) -> {
logger.info("url: {} , title : {} ", page.getRequest(), title);
};
}
}.init(Stream.iterate(28629300, i -> i + 1).limit(60).map(i -> url + i).parallel()).start();
}
}
控制台打印结果
18:35:54.278 [ForkJoinPool.commonPool-worker-3] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629326 , title : 建筑工地上的青年如何自我成长? - 知乎
18:35:54.525 [ForkJoinPool.commonPool-worker-1] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629319 , title : 国内专做进口行业的公司多不?(不包括货代公司) - 知乎
18:35:54.553 [main] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629338 , title : 如何有效地进行后天性丰胸? - 知乎
18:35:54.881 [ForkJoinPool.commonPool-worker-3] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629328 , title : Android开发中list和适配器是怎样关联的?适配器具体作用是什么?list可以做什么? - 知乎
18:35:55.168 [ForkJoinPool.commonPool-worker-1] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629321 , title : 互联网公司,粉丝运营如何做? - 知乎
18:35:55.317 [ForkJoinPool.commonPool-worker-2] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629311 , title : 为什么一些大公司的特喜欢装X? - 知乎
18:35:55.506 [ForkJoinPool.commonPool-worker-1] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629315 , title : 大二中 刚开学 一闲下来却总想给家里打电话 无法控制 这是为什么? - 知乎
18:35:55.664 [main] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629342 , title : 中国雷达哪家强? - 知乎
18:35:56.113 [ForkJoinPool.commonPool-worker-3] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629325 , title : 美丽加芬蜗牛系列怎么样? - 知乎
18:35:56.472 [ForkJoinPool.commonPool-worker-3] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629300 , title : "截石位"的说法是从哪里来的或者怎么翻译来的? - 知乎
18:35:56.713 [ForkJoinPool.commonPool-worker-1] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629304 , title : 我想考南京大学的历史系中国史的其中一组 就是01~09方向 请问 是按组命题的么? - 知乎
18:35:56.803 [ForkJoinPool.commonPool-worker-2] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629353 , title : K线的精髓在哪里? - 知乎
18:35:57.022 [ForkJoinPool.commonPool-worker-3] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629301 , title : 华为加班到底有多恐怖? - 知乎
18:35:57.143 [main] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629335 , title : 人体的哪些疾病(症状)可以自行诊断? - 知乎
18:35:57.167 [ForkJoinPool.commonPool-worker-2] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629354 , title : 既然ISIS这么罪大恶极,那么联合国为什么还不派兵剿灭? - 知乎
18:35:57.333 [ForkJoinPool.commonPool-worker-3] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629302 , title : 谁能帮我起个专门针对声乐培训,演出,小明星,小歌手教学工作室名字?能让人很快记住大气的名字?谢谢!? - 知乎
18:35:58.014 [ForkJoinPool.commonPool-worker-2] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629357 , title : 大学毕业后留有一大堆书不舍得丢掉,每到搬家时就会发愁了,请问如何处理不会觉得可惜而且便捷? - 知乎
18:35:58.022 [ForkJoinPool.commonPool-worker-3] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629349 , title : 华为研发员工的发展,待遇,福利?和互联网公司比呢? - 知乎
18:35:58.209 [ForkJoinPool.commonPool-worker-1] INFO c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629332 , title : 朋友圈疯传: 求扩散!一个小孩的生命结束了就是因为她把手机充电接口放到嘴里,而另一头还在插线板上。 求解? - 知乎
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用
。你还可以使用@
来通知其他用户。