Config Metastore

Install the Hive metastore somewhere in your cluster, see hive installation.

As part of this process, you configure the Hive metastore to use an external database as a metastore. Impala uses this same database for its own table metadata. You can choose either a MySQL or PostgreSQL database as the metastore.

It is recommends setting up a Hive metastore service rather than connecting directly to the metastore database; this configuration is required when running Impala under CDH 4.1. Make sure the /etc/impala/conf/hive-site.xml file contains the following setting, substituting the appropriate hostname for metastore_server_host:

<property>
<name>hive.metastore.uris</name>
<value>thrift://metastore_server_host:9083</value>
</property>
<property>
<name>hive.metastore.client.socket.timeout</name>
<value>3600</value>
<description>MetaStore Client socket timeout in seconds</description>
</property>

Install Imp

By default Hadoop HTTP web-consoles (JobTracker, NameNode, TaskTrackers and DataNodes) allow access without any form of authentication.

The next section describes how to configure Hadoop HTTP web-consoles to require user authentication.

Configuration

The following properties should be in the core-site.xml of all the nodes in the cluster.

 <property>
      <name>hadoop.http.filter.initializers</name>
      <value>org.apache.hadoop.security.AuthenticationFilterInitializer</value>
      <description>
                    Authentication for Hadoop HTTP web-consoles
                    add to this property the org.apache.hadoop.security.AuthenticationFilterInitializer initializer class.
      </description>
    </property>
    <property>
      <name>hadoop.http.authentication.type</name>
      <value>pers.louyj.utils.hadoop.auth.ext.StandardAuthenticationHandler</value>
      <description>
                    Defines authentic

Install Hive

wget http://mirror.bit.edu.cn/apache/hive/hive-2.1.0/apache-hive-2.1.0-bin.tar.gz
tar zxvf apache-hive-2.1.0-bin.tar.gz
mv apache-hive-2.1.0-bin hive-2.1.0

Install Postgresql

sudo -u postgres psql

CREATE ROLE hive LOGIN PASSWORD 'hive_password';
CREATE DATABASE metastore OWNER hive ENCODING 'UTF8';
GRANT ALL PRIVILEGES ON DATABASE metastore TO hive;


cd /home/hadoop/hive-2.1.0/lib
wget http://central.maven.org/maven2/org/postgresql/postgresql/9.4.1211.jre7/postgresql-9.4.1211.jre7.jar

Configuration

cd /home/hadoop/hive-2.1.0/conf

vi hive-site.xml

<configuration>
<property>
    <name>hive.exec.scratchdir</name>
    <value>hdfs://linode01.touchworld.link:9000/hive/scratchdir</value>
</property>
<property>
    <name>hive.metastore.warehouse.dir</name>
    <value>hdfs://linode01.touchworld.link:9000/hive/warehousedir</value>
</property>
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <val

Install database

install postgresql

sudo yum install postgresql-server postgresql

init database

sudo su - postgres
initdb -D /var/lib/pgsql/data

start service

systemctl status postgresql.service
systemctl start postgresql.service
systemctl stop postgresql.service

remote access

vi /var/lib/pgsql/data/postgresql.conf
listen_addresses ='*'
vi /var/lib/pgsql/data/pg_hba.conf
host all all 0.0.0.0/0 trust

restart service

systemctl restart postgresql.service

set password

su - postgres
psql
\password postgres
xxxpgxxx

create cloudera-manager database

Connect to PostgreSQL:

sudo -u postgres psql

If you are not using the Cloudera Manager installer, create a database for the Cloudera Manager Server. The database name, user name, and password can be any value. Record the names chosen because you will need them later when running the scm_prepare_database.sh script.

CREATE ROLE scm LOGIN PASSWORD 'scm';
CREATE DATABA

Install MongoDB Community Edition

Download the binary files for the desired release of MongoDB.

curl -O https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.2.9.tgz

Extract the files from the downloaded archive.

tar -zxvf mongodb-linux-x86_64-3.2.9.tgz

Copy the extracted archive to the target directory

mkdir -p ~/mongodb-3.2.9
mv mongodb-linux-x86_64-3.2.9 ~/mongodb-3.2.9

Ensure the location of the binaries is in the PATH variable.

export PATH=<mongodb-install-directory>/bin:$PATH

Run MongoDB Community Edition

Create the data directory

Before you start MongoDB for the first time, create the directory to which the mongod process will write data.

By default, the mongod process uses the /data/db directory.

If you create a directory other than this one, you must specify that directory in the dbpath option when starting the mongod process later in this procedure.

mkdir -p /data/mongodb

Set permission

Down and build source

https://github.com/eleme/corvus/archive/0.2.5.1.zip
unzip corvus-0.2.5.1
https://github.com/jemalloc/jemalloc/archive/4.5.0.tar.gz
tar zxvf jemalloc-4.5.0.tar.gz 
rm -r corvus-0.2.5.1/deps/jemalloc
mv jemalloc-4.5.0 corvus-0.2.5.1/deps/jemalloc
cd corvus-0.2.5.1/deps/jemalloc
autoconf
cd ../../
make

Configuration

bind 5101
node 192.168.1.202:5001,192.168.1.202:5002,192.168.1.203:5001,192.168.1.203:5002,192.168.1.204:5001,192.168.1.202:5002
thread 4

Start and Stop

 /root/upload/corvus-0.2.5.1/src/corvus /root/upload/corvus-0.2.5.1/corvus.conf > corvus.log 2>&1 &

Benchmark

/root/upload/redis-3.2.8/src/redis-benchmark -p 5101 -q -t get,set -r 1000000 -n 2000000  -P 16 -c 10