hadoop HA+Federation(高可用联邦)搭建配置(一)

摘要:
集群中的所有机器环境变量/etc/profile都已同步JAVA_HOME=/app/jdk/CLASSPATH=.:$JAVA_ HOME/lib/tools.jar:$ANT_ HOME/bin:$HBASE_HOME/bin:
hadoop HA+Federation(高可用联邦)搭建配置(一)

标签(空格分隔): 未分类


介绍

hadoop 集群一共有4种部署模式,详见《hadoop 生态圈介绍》。
HA联邦模式解决了单纯HA模式的性能瓶颈(主要指Namenode、ResourceManager),将整个HA集群划分为两个以上的集群,不同的集群之间通过Federation进行连接,使得HA集群拥有了横向扩展的能力。理论上,在该模式下,能够通过增加计算节点以处理无限增长的数据。

搭建准备环境

Centos
Hadoop 2.7.3
8台机器

ip主机名角色所属集群
10.111.0.1VECS00001zk nn1 jn zkfc rm1flash1
10.111.0.2VECS00002zk nn2 jn zkfc rm2flash1
10.111.0.3VECS00003zk jn
10.111.0.4VECS00004zk nn1 jn zkfcflash2
10.111.0.5VECS00005zk nn2 jn zkfcflash2
10.111.0.6VECS00006dn nmall
10.111.0.7VECS00007dn nmall
10.111.0.8VECS00008dn nmall

Tips:此次搭建并不涉及Resourcemanager的多集群

基础集群安装假设已经完毕:例如zookeeper,集群内所有机器hosts 映射全部已经完成。JDK已经安装完毕,

集群中所有机器环境变量 /etc/profile 全部已同步

JAVA_HOME=/app/jdk/
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar

# HADOOP CONFIG
export HADOOP_HOME=/app/hadoop
export HADOOP_PREFIX=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib/native"
export YARN_CONF_DIR=$HADOOP_CONF_DIR
export SQOOP_HOME=/app/sqoop
export HIVE_HOME=/app/hive
export PRESTO_HOME=/app/presto
export SCALA_HOME=/usr/local/scala-2.12.4
export SPARK_HOME=/app/spark
export ZOOKEEPER_HOME=/app/zookeeper
export HBASE_HOME=/app/hbase

PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin:$ANT_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$PRESTO_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$HBASE_HOME/bin:$ZOOKEEPER_HOME/bin:$PATH

export JAVA_HOME CLASSPATH PATH

集群中软件安装标准全部软连接化:

drwxr-xr-x 24 root   root         4096 Feb 26 19:34 anaconda3
drwxr-xr-x 10 deploy deploy       4096 Nov  2  2017 apache-hive-2.1.1-bin
drwxr-xr-x 12 root   root         4096 Feb 14 13:44 apache-hive-2.1.1-bin-test
drwxrwxr-x  5 deploy deploy       4096 Nov 27 17:15 apache-tez-0.9.0
drwxr-xr-x  5 deploy deploy       4096 Feb 14 11:19 apache-tez-0.9.0-bin
drwxrwxr-x  4 deploy deploy      12288 May 14 21:25 datacloud
lrwxrwxrwx  1 deploy deploy         10 Sep 18  2018 datax -> datax-3.0/
drwxrwxr-x 11 deploy deploy       4096 Sep 18  2018 datax-3.0
drwxrwxr-x  9 deploy deploy       4096 Feb 12 00:06 flink-1.7.2
-rw-r--r--  1 root   root    299718134 Feb 12 01:37 flink-1.7.2-bin-hadoop27-scala_2.11.tgz
lrwxrwxrwx  1 root   root           32 Oct 10  2018 hadoop -> hadoop-2.7.3-snappy-32core12disk
drwxrwxrwx 11 deploy deploy       4096 Oct 11  2018 hadoop-2.7.3
drwxr-xr-x 11 hdfs   hadoop       4096 Oct 10  2018 hadoop-2.7.3-snappy-32core12disk
lrwxrwxrwx  1 deploy deploy         27 Feb 14 13:53 hive -> apache-hive-2.1.1-bin-test/
lrwxrwxrwx  1 deploy deploy         12 Jun 25  2018 jdk -> jdk1.8.0_92/
drwxr-xr-x  8 deploy deploy       4096 Apr  1  2016 jdk1.8.0_92
lrwxrwxrwx  1 root   root           21 Nov 27 11:33 presto -> /app/presto-cli-0.207
drwxr-xr-x  7 root   root         4096 Nov 27 11:32 presto-cli-0.207
lrwxrwxrwx  1 deploy deploy         13 Jun 25  2018 scala -> scala-2.11.12
drwxrwxr-x  6 deploy deploy       4096 Nov 10  2017 scala-2.11.12
lrwxrwxrwx  1 deploy deploy         12 Jun 25  2018 spark -> spark-2.1.1/
drwxr-xr-x 15 deploy deploy       4096 Jun 25  2018 spark-2.1.1
-rw-r--r--  1 root   root   1938711032 Apr 10 13:34 spark-2.1.1.tar.gz
drwxrwxr-x 16 deploy deploy       4096 Apr  4 16:57 spark-2.3.1
lrwxrwxrwx  1 deploy deploy         12 Jun 25  2018 sqoop -> sqoop-1.4.6/
drwxrwxr-x  9 deploy deploy       4096 Jun 25  2018 sqoop-1.4.6
lrwxrwxrwx  1 deploy deploy         21 Feb 14 11:23 tez -> apache-tez-0.9.0-bin/
lrwxrwxrwx  1 deploy deploy         17 Jun 25  2018 zookeeper -> zookeeper-3.4.10/
drwxr-xr-x 10 deploy deploy       4096 Jun 25  2018 zookeeper-3.4.10
hadoop 各角色环境变量,JVM 参数配置文件

hadoop-env.sh

改配置文件为 hadoop namenode datanode 及两角色的log目录等设置环境

#Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.
# echo java home:${JAVA_HOME}
# export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME="/app/jdk"
export HADOOP_LOG_DIR=${HADOOP_HOME}/log
# The jsvc implementation to use. Jsvc is required to run secure datanodes
# that bind to privileged ports to provide authentication of data transfer
# protocol.  Jsvc is not required if SASL is configured for authentication of
# data transfer protocol using non-privileged ports.
#export JSVC_HOME=${JSVC_HOME}

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

# Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done

# The maximum amount of heap to use, in MB. Default is 1000.
#export HADOOP_HEAPSIZE="2048"
#export HADOOP_NAMENODE_INIT_HEAPSIZE="2048"

# Extra Java runtime options.  Empty by default.
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
#export HADOOP_OPTS="$HADOOP_OPTS -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=8"

# Command specific options appended to HADOOP_OPTS when specified
# export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xmx64G -Xms64G -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/hadoop.log"

export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS -XX:+UseG1GC -Xmx64g -Xms64g  -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:SurvivorRatio=8 -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=16 -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=60 -XX:G1MixedGCLiveThresholdPercent=85 -XX:G1NewSizePercent=1 -XX:G1MaxNewSizePercent=30 -verbose:gc -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1 -Xloggc:/app/hadoop/log/gc/hadoop.log"

export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xmx6144m -Xms6144m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/hadoop_dn.log"

export HADOOP_SECONDARYNAMENODE_OPTS=$HADOOP_NAMENODE_OPTS
#export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"

export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx1024m $HADOOP_PORTMAP_OPTS"

# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx4096m $HADOOP_CLIENT_OPTS"
#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"

# On secure datanodes, user to run the datanode as after dropping privileges.
# This **MUST** be uncommented to enable secure HDFS if using privileged ports
# to provide authentication of data transfer protocol.  This **MUST NOT** be
# defined if SASL is configured for authentication of data transfer protocol
# using non-privileged ports.
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}

# Where log files are stored.  $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR="/workspace/logs/hadoop-logs"

# Where log files are stored in the secure data environment.
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}

###
# HDFS Mover specific parameters
###
# Specify the JVM options to be used when starting the HDFS Mover.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_MOVER_OPTS=""

###
# Advanced Users Only!
###

# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by 
#       the user that will run the hadoop daemons.  Otherwise there is the
#       potential for a symlink attack.
#export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_PID_DIR=${HADOOP_HOME}/tmp
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}

# A string representing this instance of hadoop. $USER by default.
export HADOOP_IDENT_STRING=$USER

yarn-env.sh

此配置文件 决定 Yarn 集群的 各角色的jvm 参数,log 日志等等

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# User for YARN daemons
export HADOOP_YARN_USER=${HADOOP_YARN_USER:-yarn}
# resolve links - $0 may be a softlink
export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"
export YARN_PID_DIR="$HADOOP_HOME/tmp"
# some Java parameters
# export JAVA_HOME=/home/y/libexec/jdk1.6.0/
if [ "$JAVA_HOME" != "" ]; then
  #echo "run java in $JAVA_HOME"
  JAVA_HOME=$JAVA_HOME
fi
  
if [ "$JAVA_HOME" = "" ]; then
  echo "Error: JAVA_HOME is not set."
  exit 1
fi

JAVA=$JAVA_HOME/bin/java
JAVA_HEAP_MAX=-Xmx1000m 

# For setting YARN specific HEAP sizes please use this
# Parameter and set appropriately
# YARN_HEAPSIZE=1000

# check envvars which might override default args
if [ "$YARN_HEAPSIZE" != "" ]; then
  JAVA_HEAP_MAX="-Xmx""$YARN_HEAPSIZE""m"
fi
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=-Xmx2048m
# Resource Manager specific parameters
# export YARN_RESOURCEMANAGER_OPTS="-XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xmx20480m -Xms20480m -XX:NewRatio=2 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/yarn-resourcemanager.log -Dcom.sun.management.jmxremote ${YARN_RESOURCEMANAGER_OPTS}"


export YARN_RESOURCEMANAGER_OPTS="-XX:+UseG1GC -Xmx48g -Xms48g  -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=24 -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPercent=85 -XX:G1NewSizePercent=60 -XX:G1MaxNewSizePercent=75 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/yarn-resourcemanager-`hostname`.log   -Dcom.sun.management.jmxremote ${YARN_RESOURCEMANAGER_OPTS}"



export YARN_NODEMANAGER_OPTS="-XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xmx2048m -Xms2048m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/app/hadoop/log/gc/yarn-nodemanager.log -Dcom.sun.management.jmxremote ${YARN_NODEMANAGER_OPTS}"
# Specify the max Heapsize for the ResourceManager using a numerical value
# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set
# the value to 1000.
# This value will be overridden by an Xmx setting specified in either YARN_OPTS
# and/or YARN_RESOURCEMANAGER_OPTS.
# If not specified, the default value will be picked from either YARN_HEAPMAX
# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.
#export YARN_RESOURCEMANAGER_HEAPSIZE=1000

# Specify the max Heapsize for the timeline server using a numerical value
# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set
# the value to 1000.
# This value will be overridden by an Xmx setting specified in either YARN_OPTS
# and/or YARN_TIMELINESERVER_OPTS.
# If not specified, the default value will be picked from either YARN_HEAPMAX
# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.
#export YARN_TIMELINESERVER_HEAPSIZE=1000

# Specify the JVM options to be used when starting the ResourceManager.
# These options will be appended to the options specified as YARN_OPTS
# and therefore may override any similar flags set in YARN_OPTS
#export YARN_RESOURCEMANAGER_OPTS=

# Node Manager specific parameters

# Specify the max Heapsize for the NodeManager using a numerical value
# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set
# the value to 1000.
# This value will be overridden by an Xmx setting specified in either YARN_OPTS
# and/or YARN_NODEMANAGER_OPTS.
# If not specified, the default value will be picked from either YARN_HEAPMAX
# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.
#export YARN_NODEMANAGER_HEAPSIZE=1000

# Specify the JVM options to be used when starting the NodeManager.
# These options will be appended to the options specified as YARN_OPTS
# and therefore may override any similar flags set in YARN_OPTS
#export YARN_NODEMANAGER_OPTS=

# so that filenames w/ spaces are handled correctly in loops below
IFS=


# default log directory & file
if [ "$YARN_LOG_DIR" = "" ]; then
  YARN_LOG_DIR="$HADOOP_YARN_HOME/log"
fi
if [ "$YARN_LOGFILE" = "" ]; then
  YARN_LOGFILE='yarn.log'
fi

# default policy file for service-level authorization
if [ "$YARN_POLICYFILE" = "" ]; then
  YARN_POLICYFILE="hadoop-policy.xml"
fi

# restore ordinary behaviour
unset IFS


YARN_OPTS="$YARN_OPTS -Dhadoop.log.dir=$YARN_LOG_DIR"
YARN_OPTS="$YARN_OPTS -Dyarn.log.dir=$YARN_LOG_DIR"
YARN_OPTS="$YARN_OPTS -Dhadoop.log.file=$YARN_LOGFILE"
YARN_OPTS="$YARN_OPTS -Dyarn.log.file=$YARN_LOGFILE"
YARN_OPTS="$YARN_OPTS -Dyarn.home.dir=$YARN_COMMON_HOME"
YARN_OPTS="$YARN_OPTS -Dyarn.id.str=$YARN_IDENT_STRING"
YARN_OPTS="$YARN_OPTS -Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
YARN_OPTS="$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
  YARN_OPTS="$YARN_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
fi  
YARN_OPTS="$YARN_OPTS -Dyarn.policy.file=$YARN_POLICYFILE"

#tez config
export TEZ_CONF_DIR=/app/tez/conf
export TEZ_HOME=/app/tez
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${TEZ_HOME}/*:${TEZ_CONF_DIR}:${TEZ_HOME}/lib/*

marped-site.xml

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# export JAVA_HOME=/home/y/libexec/jdk1.6.0/

export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=4000

export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA

#export HADOOP_JOB_HISTORYSERVER_OPTS=
#export HADOOP_MAPRED_LOG_DIR="" # Where log files are stored.  $HADOOP_MAPRED_HOME/logs by default.
export HADOOP_MAPRED_LOG_DIR="$HADOOP_MAPRED_HOME/log"
#export HADOOP_JHS_LOGGER=INFO,RFA # Hadoop JobSummary logger.
#export HADOOP_MAPRED_PID_DIR= # The pid files are stored. /tmp by default.
#export HADOOP_MAPRED_IDENT_STRING= #A string representing this instance of hadoop. $USER by default
#export HADOOP_MAPRED_NICENESS= #The scheduling priority for daemons. Defaults to 0.

#tez config
export TEZ_CONF_DIR=/app/tez/conf
export TEZ_HOME=/app/tez
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${TEZ_HOME}/*:${TEZ_CONF_DIR}:${TEZ_HOME}/lib/*

hadoop 各角色本身工作参数配置文件

hadoop 一般常用的用于设置集群内各个角色本身的文件总共有4各:

core-site.xml(namespace 设置)
hdfs-site.xml(hdfs文件系统存储设置,包括nn,dn角色的通用,特殊设置)
yarn-site.xml(Yarn 资源管理集群设置)
mapred-site.xml(yarn 调度资源运行设置)

待续:...

免责声明:文章转载自《hadoop HA+Federation(高可用联邦)搭建配置(一)》仅用于学习参考。如对内容有疑问,请及时联系本站处理。

上篇bigNumber.js的简单使用PHP函数stream_context_create()模拟POST/GET下篇

宿迁高防,2C2G15M,22元/月;香港BGP,2C5G5M,25元/月 雨云优惠码:MjYwNzM=

相关文章

Linux下shell通用脚本启动jar(微服务)

Linux下shell通用脚本启动jar(微服务) vim app_jar.sh #!/bin/bash #source /etc/profile # Auth:https://www.cnblogs.com/liucx/ # Please change these parameters according to your real env. # set...

阿里云-容器服务之集群服务 k8s(Jenkins+gitlab+k8s的devops)- 01

由于docker官方停止更新Swarm,另外swarm在使用期间出现了很多bug,所以阿里云也在2019年7月发布公告:于2019年12月31日起停止技术支持,请您尽快迁移至容器服务Kubernetes版。 swarm这里不再赘述; 现在阿里云推荐Kubernetes集群服务; 本人从2019年7月开始学习使用,目前服务还是比较稳定,除了有些小问题,在此期...

案例:Oracle 10g RAC 集群无法启动

环境:RHEL 5.7 + Oracle 10.2.0.5 RAC 很多年前的一套测试环境,今天发现集群无法启动。手工尝试启动crs,集群日志也无任何输出。进一步检查集群配置: [oracle@rac1-server rac1-server]$ ocrcheck Status of Oracle Cluster Registry is as follows...

双击启动tomcat中的startup.bat闪退原因及解决方法

免安装的tomcat双击startup.bat后,启动窗口一闪而过,而且tomcat服务未启动,下面与大家分享下原因及解决方法 免安装的tomcat双击startup.bat后,启动窗口一闪而过,而且tomcat服务未启动。 原因是:在启动tomcat是,需要读取环境变量和配置信息,缺少了这些信息,就不能登记环境变量,导致了tomcat的闪退。 解决办法...

误删节点或集群怎么办?这里有一颗后悔药

本文来自Rancher Labs 作者介绍 王海龙,Rancher中国社区技术经理,负责Rancher中国技术社区的维护和运营。拥有6年的云计算领域经验,经历了OpenStack到Kubernetes的技术变革,无论底层操作系统Linux,还是虚拟化KVM或是Docker容器技术都有丰富的运维和实践经验。 在实际使用Rancher过程中,偶尔会因为误操...

如何在 Serverless K8s 集群中低成本运行 Spark 数据计算?

作者 | 柳密 阿里巴巴阿里云智能 ** 本文整理自《Serverless 技术公开课》,关注“Serverless”公众号,回复“入门”,即可获取 Serverless 系列文章 PPT。 导读:本节课主要介绍如何在 Serverless Kubernetes 集群中低成本运行 Spark 数据计算。首先简单介绍下阿里云 Serverless Kuber...