Creating a HDF cluster

Setting up a HDF cluster with Ambari

To have a fully functional cluster running HortonWorks Data Flow

 

Attention, read this first before starting the deployment of an HDF cluster

(Valid end of June 2017)
The last version of Ambari (2.5.1) is well supported on Ubuntu 16 LTS and Ubuntu 14 LTS. This is also the case for the full Hortonworks Data Platform stack (HDP, version 2.6.1). Besides being supported on Oracle Linux, Suse, CentOS, RedHat and Debian.
But the HDF stack is only supported on Ubuntu 14 LTS, not on Ubuntu 16 LTS. The current version of HDF, 3.0.0, supports only Ubuntu 14 LTS.
Of course, other Linux distributions are also supported:
  • RHEL/Oracle Linux/CentOS 6
  • RHEL/Oracle Linux/CentOS 7
  • SLES 11
  • SLES 12
  • Ubuntu 14
  • Debian 7
These are the supported distribution in common between HDF and HDP.
This is important to know if you plan to install HDF on top of HDP. This kind of setup is only possible since HDF 3.0.0, you can install it on top a new or already existing HDP cluster.

Ambari installation

First, you do the Ambari server installation as done in the HDP tutorial Creating a HDP cluster. You stop this tutoiral just before the cluster creation if you want to have a HDF-only cluster.
If you want to create a mixed cluster, with both HDP and HDF components installed together, you will first have to create an initial HDP cluster as described in the above link too.
Remember, to be able to mix HDP and HDF together, you’ll need:
  • Ambari version 2.5.1 minimum
  • HDP 2.6.1 minimum
  • HDF 3.0.0 minimum
So you will proceed to the installation of Ambari as explained in Creating a HDP cluster.

Installing a management pack

Now, that you have your Ambari server installed, you’ll first have to download and install the HDF Management Pack (mpack), which will tell Ambari which packages are available to be installed via its interface. You can do this:
  • After having just configured a brand new HDP 2.6.1 (or superior) cluster
  • After having upgraded an existing HDP cluster to Ambari 2.5.1 (or superior) and HDP 2.6.1 (or superior)
  • After having installed Ambari, before launching the cluster wizard installation
Installing the management pack is done by login at command prompt of your Ambari server and doing:
admin@hdf1:~ sudo ambari-server stop
admin@hdf1:~ wget http://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.0.0-453.tar.gz
admin@hdf1:~ sudo ambari-server install-mpack –-mpack=hdf-ambari-mpack-3.0.0.0-453.tar.gz
admin@hdf1:~ sudo ambari-server start


The URL given here to download the MPack is the one corresponding to the HDF 3.0.0 for Ubuntu 14. If you need the URL for another version or distribution, you will find the right URL in the Release Notes of the HDF version you want to install (see http://docs.hortonworks.com for this)
Once the Mpack is installed and the Ambari server restarted, there is 2 possible scenarios:
  1. You’ve done this after installing Ambari, so you don’t have any cluster defined yet. You can create a new cluster that will be a HDF-only cluster
  2. You’ve done this after having created a first HDP cluster, so you will be able to mix components from HDP with those provided by HDF
In the first case, just go the Ambari GUI, and follow the procedure described on the HDP installation to create a cluster.  The only difference will be in the proposed services and versions.

In the second case, you will have to activate the version it is providing into the Web GUI (see the chapter below).
 

Activating HDF on a new or existing HDP cluster

Be sure that Ambari is at least of version 2.5.1 and HDP of 2.6.1.
After logging in into the GUI, you will go to Admin -> Stack and versions menu.
You select the tab Versions, then click on “Manage versions”. Click “OK” to close the information dialog that’s popping up, because you are switching to the part related to the Ambari management and leaving the cluster management.
1.Managing versions.PNG

Click “Register version”, and you will see a list of versions known at this moment by your Ambari instance:
2.register version.PNG

You should have at least 3 HDF versions listed (3.0, 2.1 and 2.0). Click on “HDF-3.0”.

Scroll down to the bottom of the page, there you will have a “Save” button. Click on it and you will be brought back the list of registered versions, where you will have one more line:
3.HDF registered.PNG

Hoover you mouse to the button “Install on…” at the end of the line showing the HDF version you just register. When hoovering on the button, the name of the your cluster. Click on it.
You are brought back to the cluster management page where you see that 2 versions are available, HDP and HDF.
4.available versions.PNG

From there, if you can go to “Services” and click “Add Service” in the action dropdown menu, below the list on installed services. You will find NiFi and other HDF services listed at the end of the possible service choices:
4.Add Service.PNG