Louyj的博客

记录所思，分享所得。

Louyj的博客

主页
安装部署
Tags

Search this blog

Spark Installation

2019-05-06 14:51:53 环境搭建 hadoop Views:57 Likes:0 Comments:0

## Install Scala

wget http://downloads.lightbend.com/scala/2.10.6/scala-2.10.6.tgz
    tar zxvf scala-2.10.6.tgz
    
vi /etc/profile

export SCALA_HOME=/home/hadoop/scala-2.10.6
    export PATH=$PATH:$SCALA_HOME/bin
    
    source /etc/profile
    scala -version

## Install Spark

wget http://d3kbcqa49mib13.cloudfront.net/spark-1.6.2-bin-hadoop2.6.tgz
    tar zxvf spark-1.6.2-bin-hadoop2.6.tgz
    cd spark-1.6.2-bin-hadoop2.6/conf
    cp spark-env.sh.template spark-env.sh
    
vi spark-env.sh

export JAVA_HOME=/opt/jdk1.8.0_91
    export SCALA_HOME=/home/hadoop/scala-2.10.6
    export HADOOP_HOME=/home/hadoop/hadoop-2.6.4
    export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
    export YARN_HOME=/home/hadoop/hadoop-2.6.4
    export YARN_CONF_DIR=${YARN_HOME}/etc/hadoop
    export SPARK_HOME=/home/hadoop/spark-1.6.2-bin-hadoop2.6
    export SPARK_LOCAL_DIRS=/home/hadoop/spark-1.6.2-bin-hadoop2.6
    export SPARK_LIBARY_PATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME/lib/native
    export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin
    export SPARK_MASTER_IP=x.x.x.x
    export SPARK_DRIVER_MEMORY=1G

vi slaves

cp slaves.template slaves
    
    linode01
    linode02
    linode03

vi /etc/profile

export SPARK_HOME=/home/hadoop/spark-1.6.2-bin-hadoop2.6
    export PATH=$PATH:/opt/jdk1.8.0_91/bin:$SCALA_HOME/bin:$SPARK_HOME/bin
    
    source /etc/profile

## Start Spark

sbin/start-all.sh
    
    http://x.x.x.x:8080

Loading article…

Previous Mount HDFS by NFS

Next Impala Installation

57 views

Tencent Weibo
QQ Zone
RenRen

Sign in to leave a comment.
No Leanote account? Sign up now.

0 comments

More comments

Explore

主页
Tags

Links

My Note
Leanote Home
Leanote Comunity
lea++
Leanote Github

Proudly powered by Leanote