ZKFC ActiveBreadCrumb Value

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

ZKFC ActiveBreadCrumb Value

Francisco de Freitas
When querying different HDFS clusters I get different separators (don't really know if they're actually separators).

From zkCli.sh on different clusters, running the following I get:

cmd: get /hadoop-ha/clusterX/ActiveBreadCrumb

Cluster1 (comma):
cluster1active-nn1,active-nn1.example.com �>(�>

Cluster2 (single double quote):
cluster2active-nn2"active-nn2.example.com �>(�>

Cluster3 (dollar sign):
cluster3active-nn3$active-nn3.example.com �>(�>

How can I effectively write a generic code deployed on different HDFS clusters to effectively find out which is the active NN from querying ZK?

Or am I doing something wrong? Is the behavior above expected?
Reply | Threaded
Open this post in threaded view
|

Re: ZKFC ActiveBreadCrumb Value

Harsh J-3
The value you are looking at directly in ZooKeeper is in a
serialized/encoded form. Those are not separator characters but more
likely an encoded integer binary value that your terminal is
interpreting as a printable character.

The standard way to find the active NameNode is to use the 'hdfs
haadmin -getAllServiceState' command:

[hdfs@host ~]# hdfs haadmin -getAllServiceState
host1.com:8022                  standby
host2.com:8022                  active

You can then extract out just the active NameNode hostname:

[hdfs@host ~]# hdfs haadmin -getAllServiceState | grep active | awk
-F: '{ print $1; }'
host1.com
On Thu, Sep 13, 2018 at 9:39 PM Francisco de Freitas
<[hidden email]> wrote:

>
> When querying different HDFS clusters I get different separators (don't really know if they're actually separators).
>
> From zkCli.sh on different clusters, running the following I get:
>
> cmd: get /hadoop-ha/clusterX/ActiveBreadCrumb
>
> Cluster1 (comma):
> cluster1active-nn1,active-nn1.example.com �>(�>
>
> Cluster2 (single double quote):
> cluster2active-nn2"active-nn2.example.com �>(�>
>
> Cluster3 (dollar sign):
> cluster3active-nn3$active-nn3.example.com �>(�>
>
> How can I effectively write a generic code deployed on different HDFS clusters to effectively find out which is the active NN from querying ZK?
>
> Or am I doing something wrong? Is the behavior above expected?



--
Harsh J

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ZKFC ActiveBreadCrumb Value

Francisco de Freitas
Hi Harsh, thanks for your message. 

The thing is that I need to find that out via API programatically so that's why I gave the zkCli command as an example. I'm using this Go Lib (github.com/samuel/go-zookeeper/zk) and I get the same result.

On Fri, 14 Sep 2018 at 02:22 Harsh J <[hidden email]> wrote:
The value you are looking at directly in ZooKeeper is in a
serialized/encoded form. Those are not separator characters but more
likely an encoded integer binary value that your terminal is
interpreting as a printable character.

The standard way to find the active NameNode is to use the 'hdfs
haadmin -getAllServiceState' command:

[hdfs@host ~]# hdfs haadmin -getAllServiceState
host1.com:8022                  standby
host2.com:8022                  active

You can then extract out just the active NameNode hostname:

[hdfs@host ~]# hdfs haadmin -getAllServiceState | grep active | awk
-F: '{ print $1; }'
host1.com
On Thu, Sep 13, 2018 at 9:39 PM Francisco de Freitas
<[hidden email]> wrote:
>
> When querying different HDFS clusters I get different separators (don't really know if they're actually separators).
>
> From zkCli.sh on different clusters, running the following I get:
>
> cmd: get /hadoop-ha/clusterX/ActiveBreadCrumb
>
> Cluster1 (comma):
> cluster1active-nn1,active-nn1.example.com �>(�>
>
> Cluster2 (single double quote):
> cluster2active-nn2"active-nn2.example.com �>(�>
>
> Cluster3 (dollar sign):
> cluster3active-nn3$active-nn3.example.com �>(�>
>
> How can I effectively write a generic code deployed on different HDFS clusters to effectively find out which is the active NN from querying ZK?
>
> Or am I doing something wrong? Is the behavior above expected?



--
Harsh J
Reply | Threaded
Open this post in threaded view
|

Re: ZKFC ActiveBreadCrumb Value

Wellington Chevreuil
You could still use Harsh's solution programatically, or maybe an
easier way is to use HAUtil.getAddressOfActive() [1] for that? Ideally
we should not need to query ZK directly.

[1] https://github.com/c9n/hadoop/blob/master/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java#L341
Em sex, 14 de set de 2018 às 09:00, Francisco de Freitas
<[hidden email]> escreveu:

>
> Hi Harsh, thanks for your message.
>
> The thing is that I need to find that out via API programatically so that's why I gave the zkCli command as an example. I'm using this Go Lib (github.com/samuel/go-zookeeper/zk) and I get the same result.
>
> On Fri, 14 Sep 2018 at 02:22 Harsh J <[hidden email]> wrote:
>>
>> The value you are looking at directly in ZooKeeper is in a
>> serialized/encoded form. Those are not separator characters but more
>> likely an encoded integer binary value that your terminal is
>> interpreting as a printable character.
>>
>> The standard way to find the active NameNode is to use the 'hdfs
>> haadmin -getAllServiceState' command:
>>
>> [hdfs@host ~]# hdfs haadmin -getAllServiceState
>> host1.com:8022                  standby
>> host2.com:8022                  active
>>
>> You can then extract out just the active NameNode hostname:
>>
>> [hdfs@host ~]# hdfs haadmin -getAllServiceState | grep active | awk
>> -F: '{ print $1; }'
>> host1.com
>> On Thu, Sep 13, 2018 at 9:39 PM Francisco de Freitas
>> <[hidden email]> wrote:
>> >
>> > When querying different HDFS clusters I get different separators (don't really know if they're actually separators).
>> >
>> > From zkCli.sh on different clusters, running the following I get:
>> >
>> > cmd: get /hadoop-ha/clusterX/ActiveBreadCrumb
>> >
>> > Cluster1 (comma):
>> > cluster1active-nn1,active-nn1.example.com �>(�>
>> >
>> > Cluster2 (single double quote):
>> > cluster2active-nn2"active-nn2.example.com �>(�>
>> >
>> > Cluster3 (dollar sign):
>> > cluster3active-nn3$active-nn3.example.com �>(�>
>> >
>> > How can I effectively write a generic code deployed on different HDFS clusters to effectively find out which is the active NN from querying ZK?
>> >
>> > Or am I doing something wrong? Is the behavior above expected?
>>
>>
>>
>> --
>> Harsh J

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ZKFC ActiveBreadCrumb Value

Harsh J-3
In reply to this post by Francisco de Freitas
As mentioned by Wellington, the path you're going down will not
guarantee any compatibility in future.

The format/encoding/content/location/etc. of the messages stored in ZK
by the Failover Controller is not for public access and can change
without formal deprecation/etc.

A cleaner way without the use of commands could be to simply query the
/jmx JSON and parse out the state, something like the below done with
'jq':

~> curl <a href="http://some-nn-host:webport/jmx">http://some-nn-host:webport/jmx | jq '.beans[] |
select(.name=="Hadoop:service=NameNode,name=FSNamesystem") |
.["tag.HAState"]'

That said, the protocol-buffers structure used to encode the bytes in
ZK is defined in the following file:
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HAZKInfo.proto

You can use protoc with --go_out to compile that message into a struct
definition, and then use it in a ZK program like this:
https://gist.github.com/QwertyManiac/d3ff72a29e6defd59c353b8b6ca70418

On Fri, Sep 14, 2018 at 1:30 PM Francisco de Freitas
<[hidden email]> wrote:

>
> Hi Harsh, thanks for your message.
>
> The thing is that I need to find that out via API programatically so that's why I gave the zkCli command as an example. I'm using this Go Lib (github.com/samuel/go-zookeeper/zk) and I get the same result.
>
> On Fri, 14 Sep 2018 at 02:22 Harsh J <[hidden email]> wrote:
>>
>> The value you are looking at directly in ZooKeeper is in a
>> serialized/encoded form. Those are not separator characters but more
>> likely an encoded integer binary value that your terminal is
>> interpreting as a printable character.
>>
>> The standard way to find the active NameNode is to use the 'hdfs
>> haadmin -getAllServiceState' command:
>>
>> [hdfs@host ~]# hdfs haadmin -getAllServiceState
>> host1.com:8022                  standby
>> host2.com:8022                  active
>>
>> You can then extract out just the active NameNode hostname:
>>
>> [hdfs@host ~]# hdfs haadmin -getAllServiceState | grep active | awk
>> -F: '{ print $1; }'
>> host1.com
>> On Thu, Sep 13, 2018 at 9:39 PM Francisco de Freitas
>> <[hidden email]> wrote:
>> >
>> > When querying different HDFS clusters I get different separators (don't really know if they're actually separators).
>> >
>> > From zkCli.sh on different clusters, running the following I get:
>> >
>> > cmd: get /hadoop-ha/clusterX/ActiveBreadCrumb
>> >
>> > Cluster1 (comma):
>> > cluster1active-nn1,active-nn1.example.com �>(�>
>> >
>> > Cluster2 (single double quote):
>> > cluster2active-nn2"active-nn2.example.com �>(�>
>> >
>> > Cluster3 (dollar sign):
>> > cluster3active-nn3$active-nn3.example.com �>(�>
>> >
>> > How can I effectively write a generic code deployed on different HDFS clusters to effectively find out which is the active NN from querying ZK?
>> >
>> > Or am I doing something wrong? Is the behavior above expected?
>>
>>
>>
>> --
>> Harsh J



--
Harsh J

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ZKFC ActiveBreadCrumb Value

Francisco de Freitas
Thanks a lot Harsh. This is exactly what I did. I appreciate it for having the time to point the correct path as well.

Cheers,
Francisco

On Sat, 15 Sep 2018 at 19:21, Harsh J <[hidden email]> wrote:
As mentioned by Wellington, the path you're going down will not
guarantee any compatibility in future.

The format/encoding/content/location/etc. of the messages stored in ZK
by the Failover Controller is not for public access and can change
without formal deprecation/etc.

A cleaner way without the use of commands could be to simply query the
/jmx JSON and parse out the state, something like the below done with
'jq':

~> curl http://some-nn-host:webport/jmx | jq '.beans[] |
select(.name=="Hadoop:service=NameNode,name=FSNamesystem") |
.["tag.HAState"]'

That said, the protocol-buffers structure used to encode the bytes in
ZK is defined in the following file:
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HAZKInfo.proto

You can use protoc with --go_out to compile that message into a struct
definition, and then use it in a ZK program like this:
https://gist.github.com/QwertyManiac/d3ff72a29e6defd59c353b8b6ca70418

On Fri, Sep 14, 2018 at 1:30 PM Francisco de Freitas
<[hidden email]> wrote:
>
> Hi Harsh, thanks for your message.
>
> The thing is that I need to find that out via API programatically so that's why I gave the zkCli command as an example. I'm using this Go Lib (github.com/samuel/go-zookeeper/zk) and I get the same result.
>
> On Fri, 14 Sep 2018 at 02:22 Harsh J <[hidden email]> wrote:
>>
>> The value you are looking at directly in ZooKeeper is in a
>> serialized/encoded form. Those are not separator characters but more
>> likely an encoded integer binary value that your terminal is
>> interpreting as a printable character.
>>
>> The standard way to find the active NameNode is to use the 'hdfs
>> haadmin -getAllServiceState' command:
>>
>> [hdfs@host ~]# hdfs haadmin -getAllServiceState
>> host1.com:8022                  standby
>> host2.com:8022                  active
>>
>> You can then extract out just the active NameNode hostname:
>>
>> [hdfs@host ~]# hdfs haadmin -getAllServiceState | grep active | awk
>> -F: '{ print $1; }'
>> host1.com
>> On Thu, Sep 13, 2018 at 9:39 PM Francisco de Freitas
>> <[hidden email]> wrote:
>> >
>> > When querying different HDFS clusters I get different separators (don't really know if they're actually separators).
>> >
>> > From zkCli.sh on different clusters, running the following I get:
>> >
>> > cmd: get /hadoop-ha/clusterX/ActiveBreadCrumb
>> >
>> > Cluster1 (comma):
>> > cluster1active-nn1,active-nn1.example.com �>(�>
>> >
>> > Cluster2 (single double quote):
>> > cluster2active-nn2"active-nn2.example.com �>(�>
>> >
>> > Cluster3 (dollar sign):
>> > cluster3active-nn3$active-nn3.example.com �>(�>
>> >
>> > How can I effectively write a generic code deployed on different HDFS clusters to effectively find out which is the active NN from querying ZK?
>> >
>> > Or am I doing something wrong? Is the behavior above expected?
>>
>>
>>
>> --
>> Harsh J



--
Harsh J
Reply | Threaded
Open this post in threaded view
|

RE: ZKFC ActiveBreadCrumb Value

Pankaj Kumar

Remove me.

 

---

Thanks & Regards
Pankaj Kumar

IDC Technologies Inc.

Work: 408-457-9381- Ext-4102

Mailto: [hidden email]

---------------------------------------------------------

Empowering Technologies Services

Remote Services | IT Services | BPO |

IT Consulting | Staffing Solutions |

---------------------------------------------------------

 

From: Francisco de Freitas [mailto:[hidden email]]
Sent: Monday, September 17, 2018 9:05 AM
To: Harsh J
Cc: <[hidden email]>
Subject: Re: ZKFC ActiveBreadCrumb Value

 

Thanks a lot Harsh. This is exactly what I did. I appreciate it for having the time to point the correct path as well.

 

Cheers,

Francisco

 

On Sat, 15 Sep 2018 at 19:21, Harsh J <[hidden email]> wrote:

As mentioned by Wellington, the path you're going down will not
guarantee any compatibility in future.

The format/encoding/content/location/etc. of the messages stored in ZK
by the Failover Controller is not for public access and can change
without formal deprecation/etc.

A cleaner way without the use of commands could be to simply query the
/jmx JSON and parse out the state, something like the below done with
'jq':

~> curl <a href="http://some-nn-host:webport/jmx">http://some-nn-host:webport/jmx | jq '.beans[] |
select(.name=="Hadoop:service=NameNode,name=FSNamesystem") |
.["tag.HAState"]'

That said, the protocol-buffers structure used to encode the bytes in
ZK is defined in the following file:
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/HAZKInfo.proto

You can use protoc with --go_out to compile that message into a struct
definition, and then use it in a ZK program like this:
https://gist.github.com/QwertyManiac/d3ff72a29e6defd59c353b8b6ca70418

On Fri, Sep 14, 2018 at 1:30 PM Francisco de Freitas
<[hidden email]> wrote:


>
> Hi Harsh, thanks for your message.
>
> The thing is that I need to find that out via API programatically so that's why I gave the zkCli command as an example. I'm using this Go Lib (github.com/samuel/go-zookeeper/zk) and I get the same result.
>
> On Fri, 14 Sep 2018 at 02:22 Harsh J <[hidden email]> wrote:
>>
>> The value you are looking at directly in ZooKeeper is in a
>> serialized/encoded form. Those are not separator characters but more
>> likely an encoded integer binary value that your terminal is
>> interpreting as a printable character.
>>
>> The standard way to find the active NameNode is to use the 'hdfs
>> haadmin -getAllServiceState' command:
>>
>> [hdfs@host ~]# hdfs haadmin -getAllServiceState
>> host1.com:8022                  standby
>> host2.com:8022                  active
>>
>> You can then extract out just the active NameNode hostname:
>>
>> [hdfs@host ~]# hdfs haadmin -getAllServiceState | grep active | awk
>> -F: '{ print $1; }'
>> host1.com
>> On Thu, Sep 13, 2018 at 9:39 PM Francisco de Freitas
>> <[hidden email]> wrote:
>> >
>> > When querying different HDFS clusters I get different separators (don't really know if they're actually separators).
>> >
>> > From zkCli.sh on different clusters, running the following I get:
>> >
>> > cmd: get /hadoop-ha/clusterX/ActiveBreadCrumb
>> >
>> > Cluster1 (comma):
>> > cluster1active-nn1,active-nn1.example.com >(>
>> >
>> > Cluster2 (single double quote):
>> > cluster2active-nn2"active-nn2.example.com >(>
>> >
>> > Cluster3 (dollar sign):
>> > cluster3active-nn3$active-nn3.example.com >(>
>> >
>> > How can I effectively write a generic code deployed on different HDFS clusters to effectively find out which is the active NN from querying ZK?
>> >
>> > Or am I doing something wrong? Is the behavior above expected?
>>
>>
>>
>> --
>> Harsh J



--
Harsh J