此页面上的内容需要较新版本的 Adobe Flash Player。

获取 Adobe Flash Player

您现在的位置: 智可网 - 新技术 - Hadoop - 正文
CentOS 6.4下Hadoop2.3.0安装(4)
教程录入:李隆权    责任编辑:quan 作者:佚名 文章来源:linuxidc
3.4.2.5,关闭Hadoop 的命令是stop-all.sh,如下所示:
[hadoop@name01 src]$ /home/hadoop/src/hadoop-2.3.0/sbin/stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [name01]
name01: stopping namenode
data01: stopping datanode
data02: stopping datanode
Stopping secondary namenodes [name01]
name01: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
data02: stopping nodemanager
data01: stopping nodemanager
no proxyserver to stop
[hadoop@name01 src]$ 

上面的命令是最简单的,可以一次性把所有节点都启动、关闭。

3.4.3,第二种,分别启动HDFS 和yarn:
执行命令start-dfs.sh,是单独启动hdfs。执行完该命令后,通过jps 能够看到NameNode、DataNode、SecondaryNameNode 三个进程启动了,该命令适合于只执行hdfs
存储不使用yarn做资源管理。关闭的命令就是stop-dfs.sh 了。

3.4.3.1 先启动HDFS
[hadoop@name01 sbin]$ jps
3687 Jps
[hadoop@name01 sbin]$ pwd
/home/hadoop/src/hadoop-2.3.0/sbin
[hadoop@name01 sbin]$ start-dfs.sh
Starting namenodes on [name01]
name01: starting namenode, logging to /home/hadoop/src/hadoop-2.3.0/logs/hadoop-hadoop-namenode-name01.out
data01: starting datanode, logging to /home/hadoop/src/hadoop-2.3.0/logs/hadoop-hadoop-datanode-data01.out
data02: starting datanode, logging to /home/hadoop/src/hadoop-2.3.0/logs/hadoop-hadoop-datanode-data02.out
Starting secondary namenodes [name01]
name01: starting secondarynamenode, logging to /home/hadoop/src/hadoop-2.3.0/logs/hadoop-hadoop-secondarynamenode-name01.out

在name01节点下,查看后台的jps进程如下:
[hadoop@name01 sbin]$ jps
4081 Jps
3800 NameNode
3977 SecondaryNameNode
[hadoop@name01 sbin]$
[root@hadoop03 src]# jps
13894 Jps
13859 DataNode

去data01节点看下,后台的jps进程如下:
[hadoop@data01 ~]$ jps
3194 Jps
2863 DataNode
[hadoop@data01 ~]$ 

3.4.3.2 再启动yarn
执行命令start-yarn.sh,可以单独启动资源管理器的服务器端和客户端进程,关闭的命令就是stop-yarn.sh 

[hadoop@name01 sbin]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/src/hadoop-2.3.0/logs/yarn-hadoop-resourcemanager-name01.out
data01: starting nodemanager, logging to /home/hadoop/src/hadoop-2.3.0/logs/yarn-hadoop-nodemanager-data01.out
data02: starting nodemanager, logging to /home/hadoop/src/hadoop-2.3.0/logs/yarn-hadoop-nodemanager-data02.out

在name01节点下,查看后台的jps进程,多了一个ResourceManager进程,如下所示:
[hadoop@name01 sbin]$ jps
4601 ResourceManager
3800 NameNode
4853 Jps
3977 SecondaryNameNode
[hadoop@name01 sbin]$ 

去data01节点看下,后台的jps进程多了一个NodeManager进程,如下所示:
[hadoop@data01 ~]$ jps
3249 NodeManager
2863 DataNode
3365 Jps
[hadoop@data01 ~]$

3.4.3.3 依次关闭,先关闭yarn再关闭HDFS
[hadoop@name01 sbin]$ stop-yarn.sh
stopping yarn daemons
stopping resourcemanager
data01: stopping nodemanager
data02: stopping nodemanager
no proxyserver to stop
[hadoop@name01 sbin]$ 
[hadoop@name01 sbin]$ stop-dfs.sh
Stopping namenodes on [name01]
name01: stopping namenode
data01: stopping datanode
data02: stopping datanode
Stopping secondary namenodes [name01]
name01: stopping secondarynamenode
[hadoop@name01 sbin]$ 

PS:当然,也可以先启动MapReduce,再启动HDFS。说明HDFS 和MapReduce的进程之间是互相独立的,没有依赖关系。

3.4.4,第三种,分别启动各个进程:
[root@book0 bin]# jps
14821 Jps
[root@book0 bin]# hadoop-daemon.sh start namenode
[root@book0 bin]# hadoop-daemon.sh start datanode
[root@book0 bin]# hadoop-daemon.sh start secondarynamenode
[root@book0 bin]# hadoop-daemon.sh start jobtracker
[root@book0 bin]# hadoop-daemon.sh start tasktracker
[root@book0 bin]# jps
14855 NameNode
14946 DataNode
15043 SecondaryNameNode
15196 TaskTracker
15115 JobTracker
15303 Jps
执行的命令是“hadoop-daemon.sh start [进程名称]”,这种启动方式适合于单独增加、删除节点的情况,在安装集群环境的时候会看到

3.5,另外一种检查状态hadoop集群的状态
:用"hadoop dfsadmin -report"来查看hadoop集群的状态
[hadoop@name01 sbin]$ "hadoop dfsadmin -report"
-bash: hadoop dfsadmin -report: command not found
[hadoop@name01 sbin]$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Configured Capacity: 58695090176 (54.66 GB)
Present Capacity: 48441020416 (45.11 GB)
DFS Remaining: 48440971264 (45.11 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)

Live datanodes:
Name: 192.168.52.130:50010 (data02)
Hostname: data02
Decommission Status : Normal
Configured Capacity: 29347545088 (27.33 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 5127024640 (4.77 GB)
DFS Remaining: 24220495872 (22.56 GB)
DFS Used%: 0.00%
DFS Remaining%: 82.53%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Last contact: Fri Aug 22 00:04:58 PDT 2014

Name: 192.168.52.129:50010 (data01)
Hostname: data01
Decommission Status : Normal
Configured Capacity: 29347545088 (27.33 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 5127045120 (4.77 GB)
DFS Remaining: 24220475392 (22.56 GB)
DFS Used%: 0.00%
DFS Remaining%: 82.53%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Last contact: Fri Aug 22 00:04:58 PDT 2014


3.5,hadoop集群测试
3.5.1 运行简单的 MapReduce 计算
在$HADOOP_HOME 下有个jar 包,叫hadoop-example-2.2.0.jar,没有的话找其他版本的测试jar包;
执行如下命令,命令用法如下:hadoop jar hadoop-example-1.1.2.jar,
[root@name01 ~]# find / -name hadoop-example-1.1.2.jar
[root@name01 ~]# 
jar包不存在,需要找出用来测试的jar包,用模糊*搜索find / -name hadoop-*examp*.jar,如下所示:
[root@name01 ~]# find / -name hadoop-*examp*.jar
/home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.3.0-sources.jar
/home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.3.0-test-sources.jar
/home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar
[root@name01 ~]#
hadoop jar /home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar
[root@name01 ~]# su - hadoop
[hadoop@name01 ~]$ hadoop jar /home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the Words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the Words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
dbcount: An example job that count the pagevIEw counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts Words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the Words in the input files.
wordmean: A map/reduce program that counts the average length of the Words in the input files.
wordmedian: A map/reduce program that counts the median length of the Words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the Words in the input files.
[hadoop@name01 ~]$

验证是否可以登录:
hadoop fs -ls hdfs://192.168.52.128:9000/
hadoop fs -mkdir hdfs://192.168.1.201:9000/testfolder 


测试计算文本字符数目:
hadoop jar hadoop-examples-0.20.2-cdh3u5.jar wordcount /soft/BUILDING.txt /Wordcountoutput
[hadoop@hadoop01 hadoop-2.3.0]$ hadoop jar hadoop-examples-0.20.2-cdh3u5.jar wordcount /soft/hadoop-2.3.0/release-2.3.0/BUILDING.txt /Wordcountoutput

查看执行结果:
[hadoop@hadoop01 hadoop-2.2.0]$ hadoop fs -ls /Wordcountoutput
Found 2 items
-rw-r--r-- 3 hadoop supergroup 0 2014-04-02 11:30 /Wordcountoutput/_SUCCESS
-rw-r--r-- 3 hadoop supergroup 5733 2014-04-02 11:30 /Wordcountoutput/part-r-00000

[hadoop@hadoop01 hadoop-2.2.0]$ hadoop fs -text /Wordcountoutput/part-r-00000
"PLATFORM" 1
"Platform", 1
"platform". 1
$ 10
& 2
'-nsu' 1
'deploy' 1
'install', 1
分享
打赏我
打开支付宝"扫一扫" 打开微信"扫一扫"
客户端
"扫一扫"下载智可网App
意见反馈
CentOS 6.4下Hadoop2.3.0安装(4)
作者:佚名 来源:linuxidc
3.4.2.5,关闭Hadoop 的命令是stop-all.sh,如下所示:
[hadoop@name01 src]$ /home/hadoop/src/hadoop-2.3.0/sbin/stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [name01]
name01: stopping namenode
data01: stopping datanode
data02: stopping datanode
Stopping secondary namenodes [name01]
name01: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
data02: stopping nodemanager
data01: stopping nodemanager
no proxyserver to stop
[hadoop@name01 src]$ 

上面的命令是最简单的,可以一次性把所有节点都启动、关闭。

3.4.3,第二种,分别启动HDFS 和yarn:
执行命令start-dfs.sh,是单独启动hdfs。执行完该命令后,通过jps 能够看到NameNode、DataNode、SecondaryNameNode 三个进程启动了,该命令适合于只执行hdfs
存储不使用yarn做资源管理。关闭的命令就是stop-dfs.sh 了。

3.4.3.1 先启动HDFS
[hadoop@name01 sbin]$ jps
3687 Jps
[hadoop@name01 sbin]$ pwd
/home/hadoop/src/hadoop-2.3.0/sbin
[hadoop@name01 sbin]$ start-dfs.sh
Starting namenodes on [name01]
name01: starting namenode, logging to /home/hadoop/src/hadoop-2.3.0/logs/hadoop-hadoop-namenode-name01.out
data01: starting datanode, logging to /home/hadoop/src/hadoop-2.3.0/logs/hadoop-hadoop-datanode-data01.out
data02: starting datanode, logging to /home/hadoop/src/hadoop-2.3.0/logs/hadoop-hadoop-datanode-data02.out
Starting secondary namenodes [name01]
name01: starting secondarynamenode, logging to /home/hadoop/src/hadoop-2.3.0/logs/hadoop-hadoop-secondarynamenode-name01.out

在name01节点下,查看后台的jps进程如下:
[hadoop@name01 sbin]$ jps
4081 Jps
3800 NameNode
3977 SecondaryNameNode
[hadoop@name01 sbin]$
[root@hadoop03 src]# jps
13894 Jps
13859 DataNode

去data01节点看下,后台的jps进程如下:
[hadoop@data01 ~]$ jps
3194 Jps
2863 DataNode
[hadoop@data01 ~]$ 

3.4.3.2 再启动yarn
执行命令start-yarn.sh,可以单独启动资源管理器的服务器端和客户端进程,关闭的命令就是stop-yarn.sh 

[hadoop@name01 sbin]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/src/hadoop-2.3.0/logs/yarn-hadoop-resourcemanager-name01.out
data01: starting nodemanager, logging to /home/hadoop/src/hadoop-2.3.0/logs/yarn-hadoop-nodemanager-data01.out
data02: starting nodemanager, logging to /home/hadoop/src/hadoop-2.3.0/logs/yarn-hadoop-nodemanager-data02.out

在name01节点下,查看后台的jps进程,多了一个ResourceManager进程,如下所示:
[hadoop@name01 sbin]$ jps
4601 ResourceManager
3800 NameNode
4853 Jps
3977 SecondaryNameNode
[hadoop@name01 sbin]$ 

去data01节点看下,后台的jps进程多了一个NodeManager进程,如下所示:
[hadoop@data01 ~]$ jps
3249 NodeManager
2863 DataNode
3365 Jps
[hadoop@data01 ~]$

3.4.3.3 依次关闭,先关闭yarn再关闭HDFS
[hadoop@name01 sbin]$ stop-yarn.sh
stopping yarn daemons
stopping resourcemanager
data01: stopping nodemanager
data02: stopping nodemanager
no proxyserver to stop
[hadoop@name01 sbin]$ 
[hadoop@name01 sbin]$ stop-dfs.sh
Stopping namenodes on [name01]
name01: stopping namenode
data01: stopping datanode
data02: stopping datanode
Stopping secondary namenodes [name01]
name01: stopping secondarynamenode
[hadoop@name01 sbin]$ 

PS:当然,也可以先启动MapReduce,再启动HDFS。说明HDFS 和MapReduce的进程之间是互相独立的,没有依赖关系。

3.4.4,第三种,分别启动各个进程:
[root@book0 bin]# jps
14821 Jps
[root@book0 bin]# hadoop-daemon.sh start namenode
[root@book0 bin]# hadoop-daemon.sh start datanode
[root@book0 bin]# hadoop-daemon.sh start secondarynamenode
[root@book0 bin]# hadoop-daemon.sh start jobtracker
[root@book0 bin]# hadoop-daemon.sh start tasktracker
[root@book0 bin]# jps
14855 NameNode
14946 DataNode
15043 SecondaryNameNode
15196 TaskTracker
15115 JobTracker
15303 Jps
执行的命令是“hadoop-daemon.sh start [进程名称]”,这种启动方式适合于单独增加、删除节点的情况,在安装集群环境的时候会看到

3.5,另外一种检查状态hadoop集群的状态
:用"hadoop dfsadmin -report"来查看hadoop集群的状态
[hadoop@name01 sbin]$ "hadoop dfsadmin -report"
-bash: hadoop dfsadmin -report: command not found
[hadoop@name01 sbin]$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Configured Capacity: 58695090176 (54.66 GB)
Present Capacity: 48441020416 (45.11 GB)
DFS Remaining: 48440971264 (45.11 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)

Live datanodes:
Name: 192.168.52.130:50010 (data02)
Hostname: data02
Decommission Status : Normal
Configured Capacity: 29347545088 (27.33 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 5127024640 (4.77 GB)
DFS Remaining: 24220495872 (22.56 GB)
DFS Used%: 0.00%
DFS Remaining%: 82.53%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Last contact: Fri Aug 22 00:04:58 PDT 2014

Name: 192.168.52.129:50010 (data01)
Hostname: data01
Decommission Status : Normal
Configured Capacity: 29347545088 (27.33 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 5127045120 (4.77 GB)
DFS Remaining: 24220475392 (22.56 GB)
DFS Used%: 0.00%
DFS Remaining%: 82.53%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Last contact: Fri Aug 22 00:04:58 PDT 2014


3.5,hadoop集群测试
3.5.1 运行简单的 MapReduce 计算
在$HADOOP_HOME 下有个jar 包,叫hadoop-example-2.2.0.jar,没有的话找其他版本的测试jar包;
执行如下命令,命令用法如下:hadoop jar hadoop-example-1.1.2.jar,
[root@name01 ~]# find / -name hadoop-example-1.1.2.jar
[root@name01 ~]# 
jar包不存在,需要找出用来测试的jar包,用模糊*搜索find / -name hadoop-*examp*.jar,如下所示:
[root@name01 ~]# find / -name hadoop-*examp*.jar
/home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.3.0-sources.jar
/home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.3.0-test-sources.jar
/home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar
[root@name01 ~]#
hadoop jar /home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar
[root@name01 ~]# su - hadoop
[hadoop@name01 ~]$ hadoop jar /home/hadoop/src/hadoop-2.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the Words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the Words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
dbcount: An example job that count the pagevIEw counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts Words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the Words in the input files.
wordmean: A map/reduce program that counts the average length of the Words in the input files.
wordmedian: A map/reduce program that counts the median length of the Words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the Words in the input files.
[hadoop@name01 ~]$

验证是否可以登录:
hadoop fs -ls hdfs://192.168.52.128:9000/
hadoop fs -mkdir hdfs://192.168.1.201:9000/testfolder 


测试计算文本字符数目:
hadoop jar hadoop-examples-0.20.2-cdh3u5.jar wordcount /soft/BUILDING.txt /Wordcountoutput
[hadoop@hadoop01 hadoop-2.3.0]$ hadoop jar hadoop-examples-0.20.2-cdh3u5.jar wordcount /soft/hadoop-2.3.0/release-2.3.0/BUILDING.txt /Wordcountoutput

查看执行结果:
[hadoop@hadoop01 hadoop-2.2.0]$ hadoop fs -ls /Wordcountoutput
Found 2 items
-rw-r--r-- 3 hadoop supergroup 0 2014-04-02 11:30 /Wordcountoutput/_SUCCESS
-rw-r--r-- 3 hadoop supergroup 5733 2014-04-02 11:30 /Wordcountoutput/part-r-00000

[hadoop@hadoop01 hadoop-2.2.0]$ hadoop fs -text /Wordcountoutput/part-r-00000
"PLATFORM" 1
"Platform", 1
"platform". 1
$ 10
& 2
'-nsu' 1
'deploy' 1
'install', 1