HDFS Federation with HA

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

HDFS Federation with HA

rahul patwari
Hi,

Is it possible in the current architecture of HDFS to have multiple nameservices(federation), with Active and Backup Namenode(HA) for each of the nameservices?

Thanks,
Rahul 
Reply | Threaded
Open this post in threaded view
|

Re: HDFS Federation with HA

Takanobu Asanuma
Hi Rahul,

Yes, it is possible. Here are the sample configurations.

===========================================
dfs.nameservices=ns1,ns2
dfs.ha.namenodes.ns1=nn1,nn2
dfs.ha.namenodes.ns2=nn1,nn2

dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
dfs.namenode.http-address.ns1.nn2=ns1nn2:9870

dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
dfs.namenode.http-address.ns2.nn2=ns2nn2:9870

dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
===========================================

Regards,
- Takanobu

on 2018/10/10 3:07, "rahul patwari"<[hidden email]> wrote:

    Hi,
   
   
    Is it possible in the current architecture of HDFS to have multiple nameservices(federation), with Active and Backup Namenode(HA) for each of the nameservices?
   
   
    Thanks,
    Rahul
   
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: HDFS Federation with HA

rahul patwari
Hi Takanobu,

Thanks for the reply. So, does that mean that 3 dedicated journal nodes are required for each nameservice? or can we have the same journal nodes for all the nameservices?

Thanks,
Rahul 

On Wed, Oct 10, 2018 at 1:00 PM Takanobu Asanuma <[hidden email]> wrote:
Hi Rahul,

Yes, it is possible. Here are the sample configurations.

===========================================
dfs.nameservices=ns1,ns2
dfs.ha.namenodes.ns1=nn1,nn2
dfs.ha.namenodes.ns2=nn1,nn2

dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
dfs.namenode.http-address.ns1.nn2=ns1nn2:9870

dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
dfs.namenode.http-address.ns2.nn2=ns2nn2:9870

dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
===========================================

Regards,
- Takanobu

on 2018/10/10 3:07, "rahul patwari"<[hidden email]> wrote:

    Hi,


    Is it possible in the current architecture of HDFS to have multiple nameservices(federation), with Active and Backup Namenode(HA) for each of the nameservices?


    Thanks,
    Rahul



Reply | Threaded
Open this post in threaded view
|

Re: HDFS Federation with HA

Takanobu Asanuma
Hi Rahul,

Actually, you can use the same journal nodes for other nameservices if you specify the different directories.

===========================================
dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns2
===========================================

Regards,
- Takanobu

on 2018/10/10 16:56, "rahul patwari"<[hidden email]> wrote:

    Hi Takanobu,
   
   
    Thanks for the reply. So, does that mean that 3 dedicated journal nodes are required for each nameservice? or can we have the same journal nodes for all the nameservices?
   
   
    Thanks,
    Rahul
   
   
    On Wed, Oct 10, 2018 at 1:00 PM Takanobu Asanuma <[hidden email]> wrote:
   
   
    Hi Rahul,
   
    Yes, it is possible. Here are the sample configurations.
   
    ===========================================
    dfs.nameservices=ns1,ns2
    dfs.ha.namenodes.ns1=nn1,nn2
    dfs.ha.namenodes.ns2=nn1,nn2
   
    dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
    dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
    dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
    dfs.namenode.http-address.ns1.nn2=ns1nn2:9870
   
    dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
    dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
    dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
    dfs.namenode.http-address.ns2.nn2=ns2nn2:9870
   
    dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
    dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
   
    dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
    dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
    ===========================================
   
    Regards,
    - Takanobu
   
    on 2018/10/10 3:07, "rahul patwari"<[hidden email]> wrote:
   
        Hi,
   
   
        Is it possible in the current architecture of HDFS to have multiple nameservices(federation), with Active and Backup Namenode(HA) for each of the nameservices?
   
   
        Thanks,
        Rahul
   
   
   
   
   
   
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: HDFS Federation with HA

Pankaj Kumar
In reply to this post by rahul patwari

Remove Me.

 

---

Thanks & Regards
Pankaj Kumar

IDC Technologies Inc.

Work: 408-457-9381- Ext-4102

Mailto: [hidden email]

---------------------------------------------------------

Empowering Technologies Services

Remote Services | IT Services | BPO |

IT Consulting | Staffing Solutions |

---------------------------------------------------------

 

From: rahul patwari [mailto:[hidden email]]
Sent: Wednesday, October 10, 2018 3:56 AM
To: [hidden email]
Cc: [hidden email]
Subject: Re: HDFS Federation with HA

 

Hi Takanobu,

 

Thanks for the reply. So, does that mean that 3 dedicated journal nodes are required for each nameservice? or can we have the same journal nodes for all the nameservices?

 

Thanks,

Rahul 

 

On Wed, Oct 10, 2018 at 1:00 PM Takanobu Asanuma <[hidden email]> wrote:

Hi Rahul,

Yes, it is possible. Here are the sample configurations.

===========================================
dfs.nameservices=ns1,ns2
dfs.ha.namenodes.ns1=nn1,nn2
dfs.ha.namenodes.ns2=nn1,nn2

dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
dfs.namenode.http-address.ns1.nn2=ns1nn2:9870

dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
dfs.namenode.http-address.ns2.nn2=ns2nn2:9870

dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
===========================================

Regards,
- Takanobu

on 2018/10/10 3:07, "rahul patwari"<[hidden email]> wrote:

    Hi,


    Is it possible in the current architecture of HDFS to have multiple nameservices(federation), with Active and Backup Namenode(HA) for each of the nameservices?


    Thanks,
    Rahul


Reply | Threaded
Open this post in threaded view
|

Re: HDFS Federation with HA

Rams Venkatesh
Please email to [hidden email] to get yourself removed from the distribution list


On 10 Oct 2018, at 3:14 pm, Pankaj Kumar <[hidden email]> wrote:

Remove Me.

 

---

Thanks & Regards
Pankaj Kumar

IDC Technologies Inc.

Work: 408-457-9381- Ext-4102

Mailto: [hidden email]

---------------------------------------------------------

Empowering Technologies Services

Remote Services | IT Services | BPO |

IT Consulting | Staffing Solutions |

---------------------------------------------------------

 

From: rahul patwari [[hidden email]]
Sent: Wednesday, October 10, 2018 3:56 AM
To: [hidden email]
Cc: [hidden email]
Subject: Re: HDFS Federation with HA

 

Hi Takanobu,

 

Thanks for the reply. So, does that mean that 3 dedicated journal nodes are required for each nameservice? or can we have the same journal nodes for all the nameservices?

 

Thanks,

Rahul 

 

On Wed, Oct 10, 2018 at 1:00 PM Takanobu Asanuma <[hidden email]> wrote:

Hi Rahul,

Yes, it is possible. Here are the sample configurations.

===========================================
dfs.nameservices=ns1,ns2
dfs.ha.namenodes.ns1=nn1,nn2
dfs.ha.namenodes.ns2=nn1,nn2

dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
dfs.namenode.http-address.ns1.nn2=ns1nn2:9870

dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
dfs.namenode.http-address.ns2.nn2=ns2nn2:9870

dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
===========================================

Regards,
- Takanobu

on 2018/10/10 3:07, "rahul patwari"<[hidden email]> wrote:

    Hi,


    Is it possible in the current architecture of HDFS to have multiple nameservices(federation), with Active and Backup Namenode(HA) for each of the nameservices?


    Thanks,
    Rahul