Setup Maven + Hadoop in
General tips
● If you get an error, restart your IDE or terminal and try again. This often fixes the
error.
Windows
1. Install Maven & Java
Go to https://maven.apache.org/download.cgi and install Maven.
1) Unpack the archive where you would like to store the binaries, e.g.:
unzip apache-maven-3.x.y.zip
2) A directory called "apache-maven-3.x.y" will be created.
3) Add the bin directory to your PATH, e.g.:
set PATH="c:\program files\apache-maven-3.x.y\bin";%PATH%
4) Make sure JAVA_HOME is set to the location of your JDK
5) Run "mvn --version" to verify that it is correctly installed.
For installing Java 8 go to
https://www.oracle.com/nl/java/technologies/javase/javase-jdk8-downloads.html (requires
the creation of an account).
Verify JAVA_HOME
1) Open a Command Prompt window (Win⊞ + R, type cmd, hit Enter).
2) Enter the command echo %JAVA_HOME% . This should output the path to your
Java installation folder. If it doesn't, your JAVA_HOME variable was not set correctly.
2. Change local folders
● Create a folder named C:\hadoop\bin. T wo files must be added which both can be
found on https://github.com/cdarlint/winutils (choose Hadoop version 2.6.5). Add the
following files to C:\hadoop\bin:
○ hadoop.dll
○ winutils.exe
● Add hadoop.dll to C:\Windows\System32
3. Set System variables
Go to your Environment Variables.
!! Only configure System variables (bottom panel), NOT User variables (upper panel) !!
In your System variables, do the following:
● Click on ‘New…’:
1. name: HADOOP_HOME
value: C:\hadoop\
2. name: MAVEN_HOME
value: C:\Program Files\apache-maven-3.6.3
3. name: M2_HOME
value: C:\Program Files\apache-maven-3.6.3
4. name: JAVA_HOME
value: path to your Java JDK (example: C:\Program Files\Java\jdk1.8.0_281)
● Click on the Path variable and choose ‘Edit…’. Then choose ‘New’ and write
C:\Program Files\apache-maven-3.6.3
4. Start IDE and build project
Close all windows and run your IDE or terminal as administrator. Open the project (see
https://github.com/schelterlabs/uva-bigdata-course-2021-students.git) and set your JDK
under the tab ‘File’ → ‘Project Structure…’ (make sure to set it to the Java 8 JDK you
installed). Now you can build it using the Maven command:
mvn -DskipTests install
To check the code of Exercise 1:
mvn -Dtest=nl.uva.bigdata.hadoop.exercise1.WordCountLocalTest test
mvn -Dtest=nl.uva.bigdata.hadoop.exercise1.WordCountClusterTest test
MAC OS
Go to https://www.journaldev.com/2348/install-maven-mac-os and follow the steps.
After installing, follow step 4 Start IDE and build project from the Windows setup (i.e. build
project and run unit tests). In the case of a Mac OS, you don’t need to run as administrator.
LINUX
Go to https://linuxize.com/post/how-to-install-apache-maven-on-ubuntu-18-04/ and follow the
steps.
In the case that you get errors due to a wrong JDK version, you have to switch between
versions. For more information see
https://blogs.igalia.com/dpino/2011/10/13/configuring-different-jdks-with-alternatives/.
After installing, follow step 4 Start IDE and build project from the Windows setup (i.e. build
project and run unit tests). In the case of a Linux OS, you don’t need to run as administrator.
