Install

Debian / Ubuntu:

apt-get install python-pip
pip install shadowsocks

CentOS:

yum install python-setuptools && easy_install pip
#curl -o get-pip.py https://bootstrap.pypa.io/pip/2.7/get-pip.py && python get-pip.py
pip install shadowsocks

Windows:

See [Install Server on Windows]

Usage

ssserver -p 443 -k password -m aes-256-cfb

To run in the background:

sudo ssserver -p 443 -k password -m aes-256-cfb --user nobody -d start
ssserver --log-file /var/log/shadowsocks-12300.log --pid-file /var/run/shadowsocks-12300.pid --user nobody -p 12300 -k password -m aes-256-cfb -s 0.0.0.0 -d start

To stop:

sudo ssserver -d stop

To check the log:

sudo less /var/log/shadowsocks.log

Check all the options via -h. You can also use a [Configuration] file
instead.

Client

• [Windows] / [OS X]
• [Android] / [iOS]
• [OpenWRT]

Use GUI clients on your local PC/phones. Check the README of your client
for more information.

Documenta

Install Jdk

参考Jdk安装

Configure SSH

参考SSH免密码登陆

Install Ntp

yum install ntp
vi /etc/ntp.conf
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp3.aliyun.com iburst
systemctl enable ntpd
systemctl start ntpd

Install Hadoop

Download Hadoop

wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.5/hadoop-2.7.5.tar.gz
tar zxvf hadoop-2.7.5.tar.gz 
cd hadoop-2.7.5/

Configure Hadoop

Config Env

vi .bashrc
export JAVA_HOME=/opt/jdk1.8.0_202
export HADOOP_PID_DIR=/data/hadooptemp

Configure slaves

vi slaves
test01
test02
test03

Configure core-site.xml

mkdir /data/hadoop
mkdir /data/hadooptemp

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/hadoop</value>
        <description>
            A base for other temporary directories.
        </description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://cluster01</value>
        <descr

Before You Begin

SSH Configuration

vi /etc/hosts
x.x.x.x linode01
x.x.x.x linode02
x.x.x.x linode03

hostnamectl set-hostname linode01

ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub linode01
ssh-copy-id -i ~/.ssh/id_rsa.pub linode02
ssh-copy-id -i ~/.ssh/id_rsa.pub linode03

Disable Firewall

systemctl stop firewalld.service
systemctl disable firewalld.service

Dependency

yum install psmisc -y
yum install libxslt-devel -y
yum install chkconfig bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb -y
yum install python-psycopg2 -y 
yum install snappy snappy-devel  -y

#NFS
yum install rpcbind -y
service rpcbind start

Install the Oracle JDK

cd /opt/
sudo wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn/java/jdk/7u80-b15/jdk-7u80-linux-x64

Before You Begin

SSH Configuration

vi /etc/hosts
x.x.x.x linode01
x.x.x.x linode02
x.x.x.x linode03

hostnamectl set-hostname linode01

ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub linode01
ssh-copy-id -i ~/.ssh/id_rsa.pub linode02
ssh-copy-id -i ~/.ssh/id_rsa.pub linode03

Disable Firewall

systemctl stop firewalld.service
systemctl disable firewalld.service

Dependency

yum install psmisc -y
yum install libxslt-devel -y
yum install chkconfig bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb -y
yum install python-psycopg2 -y 
yum install snappy snappy-devel  -y

#NFS
yum install rpcbind -y
service rpcbind start

Install and Configure External Databases

sudo yum install postgresql-server postgresql -y
sudo su - postgres
initdb -D /var/lib/pgsql/data

#remote access
vi /var/lib/pgsql/data/postgresql.conf
listen_addresses ='*'

vi /var/lib/pgsql/data/pg_h

Overview

The NFS Gateway supports NFSv3 and allows HDFS to be mounted as part of the client’s local file system. Currently NFS Gateway supports and enables the following usage patterns:

• Users can browse the HDFS file system through their local file system on NFSv3 client compatible operating systems.
• Users can download files from the the HDFS file system on to their local file system.
• Users can upload files from their local file system directly to the HDFS file system.
• Users can stream data directly to HDFS through the mount point. File append is supported but random write is not supported.

The NFS gateway machine needs the same thing to run an HDFS client like Hadoop JAR files, HADOOP_CONF directory. The NFS gateway can be on the same host as DataNode, NameNode, or any HDFS client.

Configuration

in core-site.xml of the namenode, the following must be set( in non-secure mode)

<property>
  <name>hadoop.proxyuser.nfs

Install Scala

wget http://downloads.lightbend.com/scala/2.10.6/scala-2.10.6.tgz
tar zxvf scala-2.10.6.tgz

vi /etc/profile

export SCALA_HOME=/home/hadoop/scala-2.10.6
export PATH=$PATH:$SCALA_HOME/bin

source /etc/profile
scala -version

Install Spark

wget http://d3kbcqa49mib13.cloudfront.net/spark-1.6.2-bin-hadoop2.6.tgz
tar zxvf spark-1.6.2-bin-hadoop2.6.tgz
cd spark-1.6.2-bin-hadoop2.6/conf
cp spark-env.sh.template spark-env.sh

vi spark-env.sh

export JAVA_HOME=/opt/jdk1.8.0_91
export SCALA_HOME=/home/hadoop/scala-2.10.6
export HADOOP_HOME=/home/hadoop/hadoop-2.6.4
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
    export YARN_HOME=/home/hadoop/hadoop-2.6.4
    export YARN_CONF_DIR=${YARN_HOME}/etc/hadoop
export SPARK_HOME=/home/hadoop/spark-1.6.2-bin-hadoop2.6
export SPARK_LOCAL_DIRS=/home/hadoop/spark-1.6.2-bin-hadoop2.6
export SPARK_LIBARY_PATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME/lib/native
    expo