Confirming the IP Address and the Port Number in HDFS URL

An HDFS URL setting item appears when you configure a Transwarp Inceptor data connection or a Hadoop Hive data connection, as shown in the following figure.

The HDFS URL is described as follows.

• Enter the address of an active node in the Hadoop Distributed File System (HDFS).

• The format is hdfs://IP address: Port number. For example, hdfs://192.168.101.119:8020.

This document describes how to confirm the IP address and port in an HDFS URL.

Determining the Port Number

Execute the following SQL statement in the database: desc formatted Database name.Table name. Then check the Location row in the query result for the port number.

Case One

In the query result, the port number is 9000, and the hostname is hive1, as shown in the following figure.

Case Two

Sometimes, you cannot confirm the port number in the Location row.

This occurs in a high availability (HA) HDFS cluster. In an HA HDFS cluster, there are two NameNodes with equal status. One is the Active NameNode, and the other is the Standby NameNode. Both NameNodes start simultaneously, but only one NameNode enters the working state. At any time, one NameNode is in the Active state, and the other is in the Standby state. The Active NameNode is responsible for all client operations, while the Standby NameNode maintains enough state to provide a fast failover if necessary.

You can confirm the HDFS node address through the hdfs-site.xml file. A typical high availability configuration is as follows. You can obtain the Remote Procedure Call (RPC) port number from it.

hdfs-site.xml
<!-- The cluster uses NameNodes in a high availability architecture. Configure mycluster as the HDFS nameservice ID. -->
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<!-- NameNode IDs. -->
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<!-- The RPC address of NameNode 1 -->
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>node00:8020</value>
</property>
<!-- The RPC address of NameNode 2 -->
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>node01:8020</value>
</property>
<!-- The HTTP address of NameNode 1 -->
<property>
<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>node00:50070</value>
</property>
<!-- The HTTP address of NameNode 2 -->
<property>
<name>dfs.namenode.http-address.mycluster.nn2</name>
<value>node01:50070</value>
</property>
<!-- Configure shared JournalNode storage. -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node00:8485;node01:8485;node02:8485/mycluster</value>
</property>
<!-- Configure the failover proxy provider class. -->
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
</value>
</property>
<!-- Configure the fencing method used to terminate the original active NameNode during failover. -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<!-- The SSH private key used by sshfence -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<!-- The directory for storing JournalNode metadata. -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/software/hadoop/hdfs/journalnode/data</value>
</property>
<!-- Enable automatic failover when the active NameNode fails.-->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>

Obtaining the IP Address

On the server where the database is deployed, ping the hostname to resolve its IP address.

In case one, the hostname is hive1. Ping hive1 on the server where the Hive database is located, as shown in the following figure.

From the above figure, the HDFS IP address is 192.168.101.243.

In case two, an HA HDFS cluster has one Active NameNode and one Standby NameNode, and only the Active NameNode can be connected. You need to use the HA connection method to ensure that your connection can be routed to the Active NameNode, even if a failover occurs. The following describes the configuration steps.

Note: 

HA configuration is not supported in FineDataLink of versions before 4.1.13.2. You need to confirm the address of the Active NameNode.

Identify Active and Standby Namenodes using the command-line tool:

Common services, such as HDFS, usually provide built-in command-line tools, such as the HDFS command-line interface (CLI). You can use a specific command and node ID to identify the Active and Standby NameNodes.

Use the following command to determine the Active and Standby states of HDFS NameNodes.

hdfs haadmin

You can check HDFS parameters on the CDH platform. The default path of the HDFS configuration file is /etc/hadoop/conf.cloudera.hdfs/hdfs-site.xml. In the hdfs-site.xml file, you can find the following HA configurations.

<property>
  <name>dfs.ha.namenodes.sdg</name>
  <value>namenode25,namenode30</value>
</property>

The configuration shows two NameNodes. One is Active, and one is Standby.

To identify the Active NameNode and the Standby NameNode, use the hdfs haadmin -getServiceState command. For example, run the following commands to check the state of each NameNode.

$ hdfs haadmin -getServiceState namenode30
active
$ hdfs haadmin -getServiceState namenode25
standby

Then you can confirm the IP address of the active node.

Note: 

FineDataLink of V4.1.13.2 and later versions supports HA configurations.