Spark Standalone 模式部署案例

头像
北冥有鱼
    阅读 6 分钟

    先决条件：

    环境

    准备 master 主机和 worker 分机

    • server1 机器：10.8.26.197，master

    • server2 机器：10.8.26.196，worker

    • server3 机器：10.8.26.195，worker

    修改 host

    [root@server1 ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.8.26.197   server1
10.8.26.196   server2
10.8.26.195   server3

    关闭所有节点机防火墙

    # systemctl status firewalld
# systemctl stop firewalld
# systemctl disable firewalld

    启动集群

    主节点

    ./sbin/start-master.sh

    查看输出日志：

    cat logs/spark....
[root@server1 spark-2.0.2-bin-hadoop2.7]# cat logs/spark-root-org.apache.spark.deploy.master.Master-1-server1.out
Spark Command: /usr/local/jdk1.8.0_102/bin/java -cp /usr/local/spark-2.0.2-bin-hadoop2.7/conf/:/usr/local/spark-2.0.2-bin-hadoop2.7/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host server1 --port 7077 --webui-port 8080
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/12/26 10:35:34 INFO Master: Started daemon with process name: 8671@server1
16/12/26 10:35:34 INFO SignalUtils: Registered signal handler for TERM
16/12/26 10:35:34 INFO SignalUtils: Registered signal handler for HUP
16/12/26 10:35:34 INFO SignalUtils: Registered signal handler for INT
16/12/26 10:35:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/12/26 10:35:35 INFO SecurityManager: Changing view acls to: root
16/12/26 10:35:35 INFO SecurityManager: Changing modify acls to: root
16/12/26 10:35:35 INFO SecurityManager: Changing view acls groups to:
16/12/26 10:35:35 INFO SecurityManager: Changing modify acls groups to:
16/12/26 10:35:35 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
16/12/26 10:35:36 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
16/12/26 10:35:36 INFO Master: Starting Spark master at spark://server1:7077
16/12/26 10:35:36 INFO Master: Running Spark version 2.0.2
16/12/26 10:35:36 INFO Utils: Successfully started service 'MasterUI' on port 8080.
16/12/26 10:35:36 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://10.8.26.197:8080
16/12/26 10:35:36 INFO Utils: Successfully started service on port 6066.
16/12/26 10:35:36 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
16/12/26 10:35:37 INFO Master: I have been elected leader! New state: ALIVE

    通过 master-ip:8080 访问 master 的 web UI

    spark-setup

    各 worker 节点

    ./sbin/start-slave.sh spark://server1:7077

    节点输出日志：

    cat logs/spark....
[root@server2 spark-2.0.2-bin-hadoop2.7]# cat logs/spark-root-org.apache.spark.deploy.worker.Worker-1-server2.out
Spark Command: /usr/local/jdk1.8.0_102/bin/java -cp /usr/local/spark-2.0.2-bin-hadoop2.7/conf/:/usr/local/spark-2.0.2-bin-hadoop2.7/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://server1:7077
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/12/26 10:43:04 INFO Worker: Started daemon with process name: 7466@server2
16/12/26 10:43:04 INFO SignalUtils: Registered signal handler for TERM
16/12/26 10:43:04 INFO SignalUtils: Registered signal handler for HUP
16/12/26 10:43:04 INFO SignalUtils: Registered signal handler for INT
16/12/26 10:43:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/12/26 10:43:05 INFO SecurityManager: Changing view acls to: root
16/12/26 10:43:05 INFO SecurityManager: Changing modify acls to: root
16/12/26 10:43:05 INFO SecurityManager: Changing view acls groups to:
16/12/26 10:43:05 INFO SecurityManager: Changing modify acls groups to:
16/12/26 10:43:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
16/12/26 10:43:06 INFO Utils: Successfully started service 'sparkWorker' on port 47422.
16/12/26 10:43:06 INFO Worker: Starting Spark worker 10.8.26.196:47422 with 1 cores, 1024.0 MB RAM
16/12/26 10:43:06 INFO Worker: Running Spark version 2.0.2
16/12/26 10:43:06 INFO Worker: Spark home: /usr/local/spark-2.0.2-bin-hadoop2.7
16/12/26 10:43:06 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
16/12/26 10:43:06 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://10.8.26.196:8081
16/12/26 10:43:06 INFO Worker: Connecting to master server1:7077...
16/12/26 10:43:07 INFO TransportClientFactory: Successfully created connection to server1/10.8.26.197:7077 after 109 ms (0 ms spent in bootstraps)
16/12/26 10:43:07 INFO Worker: Successfully registered with master spark://server1:7077

    通过 master-ip:8080 访问 master 的 web UI

    spark-setup

    通过 worker-ip:8081 访问 worker 的 web UI

    spark-setup

    提交应用程序到集群

    集成 shell 测试环境

    切换至 bin 目录

    [root@server1 spark-2.0.2-bin-hadoop2.7]# cd bin

    进入运行在集群上的 spark 的 集成调试环境。

    Python

    [root@server1 bin]# ./pyspark --master spark://server1:7077
Python 2.7.5 (default, Nov  6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
16/12/26 10:59:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.0.2
      /_/

Using Python version 2.7.5 (default, Nov  6 2016 00:28:07)
SparkSession available as 'spark'.

    以统计文本行数为例：

    >>> textFile=sc.textFile("../README.md")
>>> textFile.count()
99

    输出 README.md 中共有 99 行

    scala

    [root@server1 spark-2.0.2-bin-hadoop2.7]# ./bin/spark-shell --master spark://server1:7077
scala> val textFile=sc.textFile("README.md")
textFile: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[9] at textFile at <console>:24

scala> textFile.count()
res0: Long = 99

    可以在 master 的 web 界面里面看到任务执行情况

    spark-setup

    也可以在 worker 的 web 界面里面看单个 worker 的情况

    spark-setup

    spark
    北冥有鱼
    11 声望2 粉丝

    认真就输了

    « 上一篇
    在 Linux 上部署 Docker
    下一篇 »
    Spark on YARN 部署案例

    引用和评论

    推荐阅读
    头像
    KAFKA 连接器发布（Microsoft SQL Server)

    北冥有鱼阅读 2.6k

    头像
    阿里云 EMR Serverless Spark 版正式开启商业化

    阿里云大数据AI阅读 325

    头像
    Spark 写入 Elasticsearch 报错地址被占用（qbit）

    qbit阅读 270

    头像
    Spark与Hive的完美结合：如何在Spark上部署Hive

    企业任务调度平台阅读 246

    头像
    EMR Serverless Spark：一站式全托管湖仓分析利器

    阿里云大数据AI阅读 200

    头像
    经典梅开二度，我把IDEA版本的SparkSQL插件也撸好了

    泊浮目阅读 64

    0 条评论
    得票最新