Setting up a HDF cluster with AmbariTo have a fully functional cluster running HortonWorks Data Flow
Attention, read this first before starting the deployment of an HDF cluster(Valid end of June 2017)
The last version of Ambari (2.5.1) is well supported on Ubuntu 16 LTS and Ubuntu 14 LTS. This is also the case for the full Hortonworks Data Platform stack (HDP, version 2.6.1). Besides being supported on Oracle Linux, Suse, CentOS, RedHat and Debian.
But the HDF stack is only supported on Ubuntu 14 LTS, not on Ubuntu 16 LTS. The current version of HDF, 3.0.0, supports only Ubuntu 14 LTS.
Of course, other Linux distributions are also supported:
- RHEL/Oracle Linux/CentOS 6
- RHEL/Oracle Linux/CentOS 7
- SLES 11
- SLES 12
- Ubuntu 14
- Debian 7
This is important to know if you plan to install HDF on top of HDP. This kind of setup is only possible since HDF 3.0.0, you can install it on top a new or already existing HDP cluster.
Ambari installationFirst, you do the Ambari server installation as done in the HDP tutorial Creating a HDP cluster. You stop this tutoiral just before the cluster creation if you want to have a HDF-only cluster.
If you want to create a mixed cluster, with both HDP and HDF components installed together, you will first have to create an initial HDP cluster as described in the above link too.
Remember, to be able to mix HDP and HDF together, you’ll need:
- Ambari version 2.5.1 minimum
- HDP 2.6.1 minimum
- HDF 3.0.0 minimum
Installing a management packNow, that you have your Ambari server installed, you’ll first have to download and install the HDF Management Pack (mpack), which will tell Ambari which packages are available to be installed via its interface. You can do this:
- After having just configured a brand new HDP 2.6.1 (or superior) cluster
- After having upgraded an existing HDP cluster to Ambari 2.5.1 (or superior) and HDP 2.6.1 (or superior)
- After having installed Ambari, before launching the cluster wizard installation
admin@hdf1:~ sudo ambari-server stop
admin@hdf1:~ wget http://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/220.127.116.11/tars/hdf_ambari_mp/hdf-ambari-mpack-18.104.22.168-453.tar.gz
admin@hdf1:~ sudo ambari-server install-mpack –-mpack=hdf-ambari-mpack-22.214.171.124-453.tar.gz
admin@hdf1:~ sudo ambari-server start
The URL given here to download the MPack is the one corresponding to the HDF 3.0.0 for Ubuntu 14. If you need the URL for another version or distribution, you will find the right URL in the Release Notes of the HDF version you want to install (see http://docs.hortonworks.com for this)
Once the Mpack is installed and the Ambari server restarted, there is 2 possible scenarios:
- You’ve done this after installing Ambari, so you don’t have any cluster defined yet. You can create a new cluster that will be a HDF-only cluster
- You’ve done this after having created a first HDP cluster, so you will be able to mix components from HDP with those provided by HDF
In the second case, you will have to activate the version it is providing into the Web GUI (see the chapter below).
Activating HDF on a new or existing HDP clusterBe sure that Ambari is at least of version 2.5.1 and HDP of 2.6.1.
After logging in into the GUI, you will go to Admin -> Stack and versions menu.
You select the tab Versions, then click on “Manage versions”. Click “OK” to close the information dialog that’s popping up, because you are switching to the part related to the Ambari management and leaving the cluster management.
Click “Register version”, and you will see a list of versions known at this moment by your Ambari instance:
You should have at least 3 HDF versions listed (3.0, 2.1 and 2.0). Click on “HDF-3.0”.
Scroll down to the bottom of the page, there you will have a “Save” button. Click on it and you will be brought back the list of registered versions, where you will have one more line:
Hoover you mouse to the button “Install on…” at the end of the line showing the HDF version you just register. When hoovering on the button, the name of the your cluster. Click on it.
You are brought back to the cluster management page where you see that 2 versions are available, HDP and HDF.
From there, if you can go to “Services” and click “Add Service” in the action dropdown menu, below the list on installed services. You will find NiFi and other HDF services listed at the end of the possible service choices: