Data visualization can be easily realized without writing code. This SpringBoot-based artifact is simply amazing!

Some readers asked me if I have any useful BI (Business Intelligence) tools? BI tool is simply a data visualization tool. Today I recommend an open source data visualization tool DataEase , based on SpringBoot implementation, integrated with Apache Doris + Kettle, which can support very large data volume second-level queries, I hope it will help you!

SpringBoot actual combat e-commerce project mall (50k+star) address: https://github.com/macrozheng/mall

Introduction

DataEase is an available to everyone. 4.1K+Star is already available on Github. Committed to helping users quickly analyze data and gain insights into business trends, so as to achieve business improvement and optimization. DataEase supports rich data source connections, can quickly create charts by dragging and dropping, and can share them with others.

The following is a large visualization screen generated by DataEase, which is quite cool.

Architecture

As a data visualization tool, DataEase uses the popular big data technologies Apache Doris and Kettle at this stage. If you want to learn these two technologies, this project is a good choice.

system structure

The technology stack used by DataEase is as follows:

Technology	instruction
SpringBoot	Back-end infrastructure
MySQL	data storage
Apache Doris	A modern MPP analytical database product. Only sub-second response time is required to obtain query results, effectively supporting real-time data analysis.
Kettle	An open source ETL (that is, the process of data extraction, conversion, and loading), written in pure Java, can achieve efficient and stable data extraction.
Docker	Containerized deployment
Vue	Front-end infrastructure
Element	Front-end UI framework

The usage scenarios of various technologies in DataEase are as follows:

Functional architecture

Below is the functional architecture diagram of DataEase, from which we can easily see what we can do with DataEase.

Install

DataEase provides an installation package. Download the installation package and use the installation script install.sh to complete the installation. If your server has already installed MySQL, some additional configuration is required.

• First, we need to download the installation package, here is the v1.5.2 version, download address: https://github.com/dataease/dataease/releases

• After the download is complete, upload it to the Linux server and use the following command to decompress it to the specified directory;

tar -zxvf dataease-v1.5.2-online.tar.gz

• After the extraction is completed the following directory structure, attention dataease have docker-compose deployment scripts folder;

• Next, modify the installation configuration install.conf , mainly modify the service operation port DE_PORT and MySQL configuration;

# 基础配置
## 安装目录
DE_BASE=/opt
## Service 端口（默认80，大概率冲突）
DE_PORT=8010

# 数据库配置
## 是否使用外部数据库
DE_EXTERNAL_MYSQL=false
## 数据库地址（默认mysql，之前如果用docker安装过mysql建议修改）
DE_MYSQL_HOST=mysql-de
## 数据库端口（默认3306，之前如果用docker安装过mysql建议修改）
DE_MYSQL_PORT=3307
## DataEase 数据库库名
DE_MYSQL_DB=dataease
## 数据库用户名
DE_MYSQL_USER=root
## 数据库密码
DE_MYSQL_PASSWORD=Password123@mysql

• Modify the docker-compose file of dataease/docker-compose.yml , the path is 061d502b7f410c, modify the MySQL dependency name and network configuration, the default network configuration may cause conflicts;

services:

  dataease:
    image: registry.cn-qingdao.aliyuncs.com/dataease/dataease:v1.5.2
    container_name: dataease
    ports:
      - ${DE_PORT}:8081
    mem_limit: 4096m
    volumes:
      - ${DE_BASE}/dataease/conf:/opt/dataease/conf
      - ${DE_BASE}/dataease/logs:/opt/dataease/logs
      - ${DE_BASE}/dataease/plugins/thirdpart:/opt/dataease/plugins/thirdpart
      - ${DE_BASE}/dataease/data/kettle:/opt/dataease/data/kettle
    depends_on:
      # 如果之前使用Docker安装过mysql，修改名称
      mysql-de:
        condition: service_healthy
    networks:
      - dataease-network

networks:
  dataease-network:
    driver: bridge
    ipam:
      driver: default
      # 默认网段配置可能会冲突，建议修改
      config:
        - subnet: 172.33.0.0/16
          gateway: 172.33.0.1

• Modify the docker-compose file of Doris, the path is dataease/docker-compose-kettle-doris.yml , mainly modify the network configuration;

version: '2.1'
services:

  doris-fe:
    image: registry.cn-qingdao.aliyuncs.com/dataease/doris:0.15
    container_name: doris-fe
    networks:
      # 修改为33网段防止冲突
      dataease-network :
        ipv4_address: 172.33.0.198
    restart: always

  doris-be:
    image: registry.cn-qingdao.aliyuncs.com/dataease/doris:0.15
    networks:
      # 修改为33网段防止冲突
      dataease-network :
        ipv4_address: 172.33.0.199
    restart: always

• Modify the MySQL docker-compose file, the path is dataease/docker-compose-mysql.yml , just modify the container name;

version: '2.1'
services:

  mysql-de:
    image: registry.cn-qingdao.aliyuncs.com/dataease/mysql:5.7.36
    # 之前使用Docker安装过mysql，需要修改容器名称
    container_name: mysql-de
    env_file:
      - ${DE_BASE}/dataease/conf/mysql.env
    ports:
      - ${DE_MYSQL_PORT}:3306
    volumes:
      - ${DE_BASE}/dataease/conf/my.cnf:/etc/mysql/conf.d/my.cnf
      - ${DE_BASE}/dataease/bin/mysql:/docker-entrypoint-initdb.d/
      - ${DE_BASE}/dataease/data/mysql:/var/lib/mysql
    networks:
      - dataease-network

• If you enable the firewall, you must also open port 8010

firewall-cmd --zone=public --add-port=8010/tcp --permanent
firewall-cmd --reload

• Everything is ready, directly run the install.sh file in the installation directory to install;

./install.sh

• The installation process involves downloading the image. It takes a long time to wait patiently. After the installation is successful, it will be displayed as follows;

➜  dataease-v1.5.2-online ./install.sh 

Stopping doris-fe ... done
Stopping doris-be ... done
Stopping kettle   ... done
Removing doris-fe ... done
Removing doris-be ... done
Removing kettle   ... done
Removing network dataease_dataease-network
======================= 开始安装 =======================
[DATAEASE Log]: 拷贝配置文件模板文件  -> /opt/dataease/conf 
[DATAEASE Log]: 根据安装配置参数调整配置文件 
time: Wed Dec 22 10:59:39 CST 2021
/usr/sbin/getenforce
[DATAEASE Log]: 检测到 Docker 已安装，跳过安装步骤 
[DATAEASE Log]: 启动 Docker  
Redirecting to /bin/systemctl start docker.service
[DATAEASE Log]: 检测到 Docker Compose 已安装，跳过安装步骤 
[DATAEASE Log]: 拉取镜像 
Pulling doris-be ... done
Pulling kettle   ... done
Pulling mysql-de ... done
Pulling dataease ... done
Pulling doris-fe ... done
...省略若干日志
  Name                Command                       State                         Ports              
-----------------------------------------------------------------------------------------------------
dataease   /deployments/run-java.sh         Up (health: starting)   0.0.0.0:8010->8081/tcp           
doris-be   /entrypoint.sh                   Up (healthy)                                             
doris-fe   /entrypoint.sh                   Up (health: starting)                                    
kettle     /opt/kettle/carte.sh kettl ...   Up                                                       
mysql-de   docker-entrypoint.sh mysqld      Up (healthy)            0.0.0.0:3306->3306/tcp, 33060/tcp
[DATAEASE Log]: 服务启动中，请稍候 ... 
[DATAEASE Log]: 服务启动中，请稍候 ... 
[DATAEASE Log]: 【警告】服务在等待时间内未完全启动！请稍后使用 dectl status 检查服务运行状况。 
======================= 安装完成 =======================

请通过以下方式访问:
 URL: http://$LOCAL_IP:8010
 用户名: admin
 初始密码: dataease

• Since we modified the MySQL configuration, we also need to modify the MySQL connection configuration under the /opt /opt/dataease/conf/dataease.properties , changed to mysql-de ;

# 数据库配置
spring.datasource.url=jdbc:mysql://mysql-de:3306/dataease?autoReconnect=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=convertToNull&useSSL=false

• Then restart the dataease container;

docker restart dataease

• docker logs -f dataease view the log when restarting, and the project will start successfully after the database import is completed;

• Since DateEase will automatically register the dataease service in the system after successful installation, we can use the following commands to operate it.

# 查看服务状态
systemctl status dataease
# 启动服务
systemctl start dataease
# 停止服务
systemctl stop dataease

use

DataEase can be used to easily realize data visualization. Next, let's take the data in Excel and MySQL as examples to experience its functions.

basic concepts

Before using DataEase, we have to understand some of its basic concepts, which will be very helpful for using it.

• Data source: It is the data source for subsequent data analysis, referring to various database connection information, supporting common data sources such as MySQL, Elasticsearch, MongoDB, etc.;
• Data set: A collection of data, which can be Excel data, database table data, custom SQL query data, and is the data source of the view;
• View: The smallest unit of visual display, which is the basic element of the dashboard, which can be a line chart, a bar chart, a pie chart, etc.;
• Dashboard: large visual screen, combined view interface;
• Templates: Data and style templates that can be used to quickly build dashboards.

Excel data analysis

Next, we will get data from Excel and implement a dashboard to experience the data visualization function of DataEase.

• After DataEase is started successfully, you can log in admin:dataease http://192.168.3.105:8010/

• Since we modified the name of the MySQL container before, we also need to modify data source here;

• Next, we need to create a data set, just use the official sample Excel. After the download is complete, you can open it to see a product sales report. The download address is: https://dataease.io/docs/manual_demo/sales_dashboard. xlsx

• Then choose to add the data set;

• Upload Excel when creating a new file, and finally select Confirm to import;

• Because the network segment of Doris has been modified before, the imported Excel data will not be displayed, and the following error message will pop up;

• Enter the mysql-de container and enter the following command to solve it;

# 进入内置 MySQL 容器内
docker exec -it mysql-de sh
# 进入 MySQL 容器后，连接 doris-fe
mysql -uroot -h doris-fe -P 9030
# 由于修改了doris的网段，此处也要修改
ALTER SYSTEM ADD BACKEND "172.33.0.199:9050";
SET PASSWORD FOR 'root' = PASSWORD('Password123@doris');
CREATE DATABASE dataease;

• After the data is imported successfully, you can start to create the view, and select the data set we just imported;

• Then select the type of view , where the pie chart representing the distribution is selected;

• Drag to select dimensions and indicators, then change the style, and finally save it, and a view is completed;

• Create a few more views, and then you can create a dashboard, and then drag and drop the edit form to complete the dashboard, is it very convenient!

Database data analysis

Of course, DataEase also supports importing data from the database, and can even customize SQL queries. Let's experience these functions.

• First of all, we have to create a new data source, you can choose various types of data sources, there are quite a lot of support, here choose MySQL;

• Then create a data set, choose to add a data set from the database;

• Then create a view and use the data set created above;

• Of course, you can also customize SQL queries to add data sets;

• DataEase also has a more powerful function, you can set each view to directly link based on a certain field. For example, in the official example, we select a province, and the data of other views becomes that province;

• There is also a more interesting feature called drill down. For example, if we select a province to drill down, we can view the relevant data of cities in that province.

Summarize

In general, DataEase is a very good data visualization tool. It allows us to easily realize some data visualization requirements without writing code, and supports analyzing data from various data sources and Excel. And it uses the popular big data analysis technologies Apache Doris and Kettle, and friends who are interested in these technologies can also try it.

Reference

• Project address: https://github.com/dataease/dataease
• Official document: https://dataease.io/docs/

This article GitHub https://github.com/macrozheng/mall-learning has been included, welcome to Star!