SPark On Mac

SPark On Mac

安装

可以使用 brew 直接安装

brew install apache-spark

使用

然后执行

timgerdeMac-mini:flume timger$ spark-shell 
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
14/07/22 17:16:28 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/07/22 17:16:28 INFO SecurityManager: Changing view acls to: timger
14/07/22 17:16:28 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(timger)
14/07/22 17:16:28 INFO HttpServer: Starting HTTP Server
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.0.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_11)
Type in expressions to have them evaluated.
Type :help for more information.
14/07/22 17:16:33 INFO SecurityManager: Changing view acls to: timger
14/07/22 17:16:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(timger)
14/07/22 17:16:34 INFO Slf4jLogger: Slf4jLogger started
14/07/22 17:16:34 INFO Remoting: Starting remoting
14/07/22 17:16:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@172.28.55.78:57188]
14/07/22 17:16:34 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@172.28.55.78:57188]
14/07/22 17:16:34 INFO SparkEnv: Registering MapOutputTracker
14/07/22 17:16:34 INFO SparkEnv: Registering BlockManagerMaster
14/07/22 17:16:34 INFO DiskBlockManager: Created local directory at /var/folders/34/v8cydr991yb_xmw9jwn5wd1h0000gn/T/spark-local-20140722171634-1047
14/07/22 17:16:34 INFO MemoryStore: MemoryStore started with capacity 294.6 MB.
14/07/22 17:16:34 INFO ConnectionManager: Bound socket to port 57189 with id = ConnectionManagerId(172.28.55.78,57189)
14/07/22 17:16:34 INFO BlockManagerMaster: Trying to register BlockManager
14/07/22 17:16:34 INFO BlockManagerInfo: Registering block manager 172.28.55.78:57189 with 294.6 MB RAM
14/07/22 17:16:34 INFO BlockManagerMaster: Registered BlockManager
14/07/22 17:16:34 INFO HttpServer: Starting HTTP Server
14/07/22 17:16:34 INFO HttpBroadcast: Broadcast server started at http://172.28.55.78:57190
14/07/22 17:16:34 INFO HttpFileServer: HTTP File server directory is /var/folders/34/v8cydr991yb_xmw9jwn5wd1h0000gn/T/spark-cc9df9f4-f0e2-4201-a29d-02d73bc6569f
14/07/22 17:16:34 INFO HttpServer: Starting HTTP Server
14/07/22 17:16:35 INFO SparkUI: Started SparkUI at http://172.28.55.78:4040
14/07/22 17:16:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/22 17:16:37 INFO Executor: Using REPL class URI: http://172.28.55.78:57166
14/07/22 17:16:37 INFO SparkILoop: Created spark context..
Spark context available as sc.

scala> 

浏览器打开 http://172.28.55.78:4040/executors/