网页快照实现-Java探险之旅

引言

在监控系统、网页爬虫、审核流程等领域，经常需要保存系统在某一时刻的状态-网页快照。实现网页快照主要有phantomjs、selenium,但是网上资料都是零碎的，本文主要采用Selenium,详细阐述从编码到部署的整个流程.

一、添加Selenium依赖

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>3.11.0</version>
</dependency>

二、实现网页快照

(1) 创建WebDriver

a.手机设备

public WebDriver createWebDriver(String device) {
    Map<String, String> mobileEmulation = new HashMap<String, String>();
    //设置设备,例如:Google Nexus 7/Apple iPhone 6
    //mobileEmulation.put("deviceName", "Google Nexus 7");
    if(StringUtils.isBlank(device))
        device = "iPhone 6";
    mobileEmulation.put("deviceName", device);   //这里是要使用的模拟器名称，就是浏览器中模拟器中的顶部型号
    Map<String, Object> chromeOptions = new HashMap<String, Object>();
    chromeOptions.put("mobileEmulation", mobileEmulation);
    DesiredCapabilities capabilities = DesiredCapabilities.chrome();
    capabilities.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
    WebDriver driver = new ChromeDriver(capabilities);
    Dimension dimension = new Dimension(420, 900);
    driver.manage().window().setSize(dimension);
    return driver;
}

b.PC设备

public WebDriver createWebDriver(String device) {
    //暂时采用特定分辨率
    ChromeOptions options = new ChromeOptions();
    options.addArguments("window-size=1200,800");
    WebDriver driver = new ChromeDriver(options);
    driver.manage().window();
    return driver;
}

(2) 绘制图像

    public String execute(String uri, String device) {
        PerfCounter.count("miui_ad_schedule_capture.execute.total", 1L);
        long start = System.currentTimeMillis();
        System.setProperty("webdriver.chrome.driver", "/home/rd/chromedriver");
        //windows: System.setProperty("webdriver.chrome.driver", "src/main/resources/driver/chromedriver.exe");
        WebDriver driver = createWebDriver(device);
        String cdnUrl = "";
        try {
            driver.get(uri);
            cdnUrl = savePage(driver, url, "capture_" + System.currentTimeMillis() + ".png");
        } catch (Exception e) {
            logger.error("capture fail: [{},{}],{} !", uri ,device,e);
            throw new RuntimeException(String.format("save page failed, uri=%s, device=%s"));
        } finally {
            driver.close();
            driver.quit();
        }
        PerfCounter.count("miui_ad_schedule_capture.execute.success", 1L, System.currentTimeMillis() - start);
        return cdnUrl;
    }

    private String savePage(WebDriver driver, String filePath, String fileName) throws IOException, InterruptedException, NoSuchAlgorithmException {
        JavascriptExecutor jsExecutor = (JavascriptExecutor) driver;
        TakesScreenshot takesScreenshot = (TakesScreenshot) driver;
        jsExecutor.executeScript("window.scrollTo(0,0)");
        BufferedImage imageOriginal = new BufferedImage(1, 1, BufferedImage.TYPE_INT_RGB);// 创建全屏截图
        int lastScroll = -1;
        int currentScroll = 0;
        Thread.sleep(1000);
        while (lastScroll != currentScroll) {
            byte[] bytesScroll = takesScreenshot.getScreenshotAs(OutputType.BYTES);
            BufferedImage imageScroll = ImageIO.read(new ByteArrayInputStream(bytesScroll));
            int screenHeight = imageScroll.getHeight();
            int screenWidth = imageScroll.getWidth();

            BufferedImage combined = new BufferedImage(screenWidth, currentScroll + screenHeight, BufferedImage.TYPE_INT_RGB);
            Graphics g = combined.getGraphics();
            g.drawImage(imageOriginal, 0, 0, null);
            g.drawImage(imageScroll, 0, currentScroll, null);
            imageOriginal = combined;

            logger.info("lastScroll:" + lastScroll + "    currentScroll:" + currentScroll + "    screenHeight:" + screenHeight);
            int scrollTo = currentScroll + screenHeight;
            lastScroll = currentScroll;
            jsExecutor.executeScript("window.scrollTo(0," + scrollTo + ")");
            currentScroll = Double.valueOf(jsExecutor.executeScript("return document.documentElement.scrollTop").toString()).intValue();
            if (lastScroll > 5000) {
                break;
            }
            //网络加载
            Thread.sleep(1000);
        }

        File path = new File(filePath);
        if (!path.exists() || !path.isDirectory()) {
            path.mkdirs();
        }
        File file = new File(path.getAbsolutePath() + File.separatorChar + fileName);
        ImageIO.write(imageOriginal, "png", file);
        String uploadPath = FileStorageUtils.upload(file.getAbsolutePath(), CHANNEL_ID);
        if(file.exists()){
            file.delete();
        }
        return uploadPath;
    }

三、部署(ubantu16.04)

(1) 安装chrome

sudo apt-get install libxss1 libappindicator1 libindicator7 #install dependency
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome*.deb  
sudo apt-get install -f

安装完成后测试：

google-chrome --headless --remote-debugging-port=9222 https://chromium.org --disable-gpu

这里是使用headless模式进行远程调试，ubuntu上大多没有gpu，所以--disable-gpu以免报错。之后使用另一个命令行访问本地的9222端口：

curl http://localhost:9222

能够看到调试信息应该就是安装成功了

(2) 下载chromedriver

chromedriver提供了操作chrome的api，是selenium控制chrome的桥梁。
可以到：https://sites.google.com/a/ch...。
下载并解压：

wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip

Chrome与WebDriver的版本对应关系：

chromedriver版本	支持的Chrome版本
v2.37	v64-66
v2.36	v63-65
v2.35	v62-64
v2.34	v61-63
v2.33	v60-62
v2.32	v59-61
v2.31	v58-60
v2.30	v58-60
v2.29	v56-58
v2.28	v55-57
v2.27	v54-56
v2.26	v53-55
v2.25	v53-55
v2.24	v52-54
v2.23	v51-53
v2.22	v49-52
v2.21	v46-50
v2.20	v43-48
v2.19	v43-47
v2.18	v43-46
v2.17	v42-43
v2.13	v42-45
v2.15	v40-43
v2.14	v39-42
v2.13	v38-41
v2.12	v36-40
v2.11	v36-40
v2.10	v33-36
v2.9	v31-34
v2.8	v30-33
v2.7	v30-33
v2.6	v29-32
v2.5	v29-32
v2.4	v29-32

(3) 安装Xfvb

如果不安装Xfvb,java -jar xxx.jar 将报一下错误：

Starting ChromeDriver 2.30.477691 (6ee44a7247c639c0703f291d320bdf05c1531b57) on port 14103
Only local connections are allowed.
Exception in thread "main" org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally
  (Driver info: chromedriver=2.30.477691 (6ee44a7247c639c0703f291d320bdf05c1531b57),platform=Linux 4.9.15-x86_64-linode81 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 60.13 seconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'localhost', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.15-x86_64-linode81', java.version: '1.8.0_131'
Driver info: driver.version: ChromeDriver
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    ...

安装方式：
a.安装Xvfb

sudo apt install Xvfb

b.启动Xvfb服务

Xvfb -ac :7 -screen 0 1280x1024x8 &

c.连接服务7

export DISPLAY=:7

关于Xvfb的具体应用请参考:https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml

参考链接:
https://blog.csdn.net/codebat...
https://jiayi.space/post/zai-...

博客编写不易，如果觉得写得可以，请帮忙点个赞