sandbox上运行mapreduce程序

1、点击docker图标，启动docker引擎

2、在cmd进入到hadoop-sandbox文件下，运行

docker-compose up -d #启动实验环境

ssh -p 2222 sandbox@localhost

4、上传数据文件到sandbox

        另开一个主机命令行，查找容器ID

docker ps

        找到hadoop - sandbox - clientnode（客户端节点，用于向 Hadoop 集群提交作业、访问 HDFS 等操作。是用户与 Hadoop 集群交互的入口）的ID，这里是a7dd83327505

        将文件复制到客户端容器的/home/sandbox/目录下

docker cp F:\测试数据\grades.txt a7dd83327505:/home/sandbox/

        clientnode中，上传文件到HDFS

hdfs dfs -mkdir -p /user/sandbox/input
hdfs dfs -put /home/sandbox/grades.txt /user/sandbox/input/

        验证HDFS文件

hdfs dfs -ls /user/sandbox/input

5、上传程序文件到sandbox

        主机上执行

docker cp F:\FenBuShi\src\main\java\StudentGradeAnalysis.java a7dd83327505:/home/sandbox/mymapreduce/
Successfully copied 6.66kB to a7dd83327505:/home/sandbox/mymapreduce/

6、在clientnode运行mapreduce程序

        clientnode中进入到mymapreduce

cd ~/mymapreduce

        指定UTF-8编码编译

javac -encoding UTF-8 -classpath $(hadoop classpath)  StudentGradeAnalysis.java

        把.class文件打包为jar包

jar cf top5.jar TopStudents*.class

        运行jar包

hadoop jar StudentGradeAnalysis.jar StudentGradeAnalysis top5   /user/sandbox/input/grades.txt   /user/sandbox/output/top5_students

hadoop jar <JAR包路径> <主类名> [参数1] [输入路径] [输出路径]

        查看输出文件

hdfs dfs -cat /user/sandbox/output/top5_students/part-r-00000