DataEase入门

1、源码编译

前提 安装 JDK 21、apache-maven-3.9.6(我用3.9.3也是可以的)

qiaozhanwei@192 target % mvn -v
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /Users/qiaozhanwei/Downloads/InstallProgram/apache-maven-3.6.3
Java version: 21.0.4, vendor: Oracle Corporation, runtime: /Users/qiaozhanwei/Downloads/jdk-21.0.4.jdk/Contents/Home
Default locale: zh_CN_#Hans, platform encoding: UTF-8
OS name: "mac os x", version: "14.5", arch: "aarch64", family: "mac"

1.1、编译步骤

cd dataease
mvn clean install

cd core
mvn clean package -Pstandalone -U -Dmaven.test.skip=true

1.2、问题解决

[INFO] --------------------------------[ jar ]---------------------------------
Downloading from alimaven: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/plugins/maven-resources-plugin/3.3.1/maven-resources-plugin-3.3.1.pom
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/plugins/maven-resources-plugin/3.3.1/maven-resources-plugin-3.3.1.pom (8.2 kB at 15 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/plugins/maven-resources-plugin/3.3.1/maven-resources-plugin-3.3.1.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/plugins/maven-resources-plugin/3.3.1/maven-resources-plugin-3.3.1.jar (31 kB at 139 kB/s)
Downloading from fit2cloud-public: https://repository.fit2cloud.com/repository/fit2cloud-public/org/apache/calcite/calcite-core/1.35.14/calcite-core-1.35.14.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for dataease 2.10.1:
[INFO] 
[INFO] dataease ........................................... SUCCESS [  0.290 s]
[INFO] sdk ................................................ SUCCESS [  0.042 s]
[INFO] extensions ......................................... SUCCESS [  0.111 s]
[INFO] extensions-datasource .............................. FAILURE [01:07 min]
[INFO] extensions-view .................................... SKIPPED
[INFO] common ............................................. SKIPPED
[INFO] api ................................................ SKIPPED
[INFO] api-permissions .................................... SKIPPED
[INFO] api-base ........................................... SKIPPED
[INFO] api-sync ........................................... SKIPPED
[INFO] distributed ........................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:07 min
[INFO] Finished at: 2024-09-27T14:15:32+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project extensions-datasource: Could not resolve dependencies for project io.dataease:extensions-datasource:jar:2.10.1: Failed to collect dependencies at org.apache.calcite:calcite-core:jar:de:1.35.14: Failed to read artifact descriptor for org.apache.calcite:calcite-core:jar:de:1.35.14: Could not transfer artifact org.apache.calcite:calcite-core:pom:1.35.14 from/to fit2cloud-public (https://repository.fit2cloud.com/repository/fit2cloud-public/): Transfer failed for https://repository.fit2cloud.com/repository/fit2cloud-public/org/apache/calcite/calcite-core/1.35.14/calcite-core-1.35.14.pom: Unsupported or unrecognized SSL message -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :extensions-datasource

解决 settings.xml 进行如下配置:

<mirror>
   <id>alimaven</id>
   <name>aliyun maven</name>
   <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
   <mirrorOf>*,!fit2cloud-public</mirrorOf>
</mirror>

1.3、创建database

CREATE DATABASE dataease CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

1.4、创建运行目录

mkdir -p /opt/dataease2.0/config /opt/dataease2.0/drivers/ /opt/dataease2.0/cache/ /opt/dataease2.0/data/map /opt/dataease2.0/data/static-resource
进入到 DataEase 源码目录，将 DataEase 源码目录下的 drivers 目录的驱动复制到 /opt/dataease2.0/drivers。
cp -rp /opt/dataease/drivers/* /opt/dataease2.0/drivers/

将源码 mapFiles 目录下的地图文件复制到 /opt/dataease2.0/data/map 目录下。
cp -rp /opt/dataease/mapFiles/* /opt/dataease2.0/data/map/

将源码staticResource 目录下的资源文件复制到 /opt/dataease2.0/data/static-resource 目录下。
cp -rp /opt/dataease/staticResource/* /opt/dataease2.0/data/static-resource/

进入 DataEase 配置文件目录。
cd /opt/dataease2.0/config

在配置文件目录下创建 DataEase 配置文件 application.yml，YOUR_IP:PORTS/YOUR_DATABASE/YOUR_USER/YOUR_PASSWORD 部分请修改成实际数据库配置。
server:
    tomcat:
        connection-timeout: 70000
spring:
    servlet:
        multipart:
            max-file-size: 500MB
            max-request-size: 500MB
    datasource:
        url: jdbc:mysql://10.253.26.85:3306/dataease?autoReconnect=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=convertToNull&useSSL=false&allowPublicKeyRetrieval=true
        username: root
        password: root@123

注意 : dataease2.0 这个目录是必须的，源码中写死的

1.5、启动访问

启动
cd /opt/dataease2.0/dataease
nohup java -jar CoreApplication.jar &

访问
http://10.22.334.85:8100/
使用默认用户名 admin 密码 DataEase@123456 进行登录。

1.5、问题解决

v2.10.1 版本在程序启动的时候，flyway报错

Caused by: java.sql.SQLSyntaxErrorException: Table 'dataease.xpack_report_info' doesn't exist
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
        at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:767)
        at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:652)
        at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94)
        at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
        at org.flywaydb.core.internal.jdbc.JdbcTemplate.executeStatement(JdbcTemplate.java:210)
        at org.flywaydb.core.internal.sqlscript.ParsedSqlStatement.execute(ParsedSqlStatement.java:89)
        at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:206)

启动的时候会报，这个是官网的一个BUG，https://github.com/dataease/dataease/issues/12433；这条语句删除就好

/Users/qiaozhanwei/IdeaProjects/dataease/core/core-backend/src/main/resources/db/migration/V2.10.1__ddl.sql

2、IDEA启动

2.1、IDEA中配置本地Maven

依次选择File -> Settings -> Build,Execution,Deployment -> Build Tools -> Maven

2.2、手动添加core目录为pom

如果core文件夹没有高亮显示，则说明IDEA没有将该文件夹识别为一个有效的maven子项目，此刻需要手动添加Maven Project，等待IDEA扫描完成

2.3、编译/打包dataease模块

选择IDEA右侧的Maven按钮，选择dataease -> package按钮进行打包，打包成功后点击install将各个子模块安装到本地Maven仓库中

2.4、编译/打包core模块

选择IDEA右侧的Maven按钮，选择core -> package按钮进行打包

2.5、MySql中创建本地数据库dataease

CREATE DATABASE dataease CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

2.6、修改配置

修改数据源连接信息

/Users/qiaozhanwei/IdeaProjects/dataease/core/core-backend/src/main/resources/application-standalone.yml

spring:
  datasource:
    url: jdbc:mysql://localhost:3306/dataease?autoReconnect=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=convertToNull&useSSL=false&allowPublicKeyRetrieval=true
    username: root
    password: root@123
  messages:
    basename: i18n/lic,i18n/core,i18n/permissions,i18n/xpack
  flyway:
    enabled: true
    table: de_standalone_version
    validate-on-migrate: false
    locations: classpath:db/migration
    baseline-on-migrate: true
    out-of-order: true

mybatis-plus:
  mapper-locations: classpath:mybatis/*.xml

2.7、启动

启动成功后：访问localhost:8100 账号:admin 密码:DataEase@123456；密码是假的，只是前端校验而已

3、跑官网实例

CREATE TABLE demo_gdp_by_city (
    id INT AUTO_INCREMENT PRIMARY KEY,
    province varchar(20),
    city varchar(20),
    gdp decimal(13,8)
);


INSERT INTO demo_gdp_by_city VALUES
(1,'广东省','深圳市',14324.47000000),  
(2,'广东省','广州市',5324.78000000), 
(3,'广东省','佛山市',64324.12000000),  
(4,'广东省','东莞市',74324.47000000), 
(5,'广东省','惠州市',34324.37000000),  
(6,'广东省','珠海市',44324.46000000), 
(7,'广东省','茂名市',64324.40000000),  
(8,'广东省','江门市',14724.27000000), 
(9,'广东省','中山市',14824.44000000),  
(10,'广东省','湛江市',13324.37000000), 
(11,'广东省','汕头市',14924.17000000);



INSERT INTO demo_gdp_by_city VALUES
(12,'河北省','石家庄',14224.47000000),  
(13,'河北省','邢台市',5624.78000000), 
(14,'河北省','邯郸市',69324.12000000),  
(15,'河北省','承德市',74124.47000000), 
(16,'河北省','秦皇岛市',14324.37000000),  
(17,'河北省','廊坊市',24324.46000000), 
(18,'河北省','沧州市',34324.40000000),  
(19,'河北省','保定市',4724.27000000), 
(20,'河北省','张家口市',1824.44000000);

4、核心源码实现逻辑

代码阅读方向 :
数据源表 : core_datasource、core_de_engine(引擎表)

1、数据集；涉及的表 core_dataset_group、core_dataset_table和core_dataset_table_field
2、图 data_visualization_info、core_chart_view

底层使用了Apache Calcite生成SQL，如果是单数据源直接就自定义ClassLoader 加载对应的驱动来查询结果数据；前提会把Apache Calcite中的SQL进行替换，比如说把schema去掉；最后filter、group by、limit等使用 io.dataease.engine.sql.SQLProvider#createQuerySQL，是根据 antlr 渲染
如果是跨数据源，就使用 Apache Calcite，里面设置了不同的schema信息(自己做的标识)；可以实现联邦查询


1、版本SQL升级 springboot flyway
2、io.dataease.config.DeMvcConfig，API开头拦截器(de2api)，WebMvcConfigurer
3、core_de_engine引擎元数据存储configuration加密拦截器，io.dataease.config.MybatisConfig#dbInterceptor

5、前端打包提示node权限不足

参考文章 : https://github.com/thingsboard/thingsboard/issues/1335

修改 :