[DISCUSS] Merge HDFS-13891(RBF) to trunk

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Merge HDFS-13891(RBF) to trunk

Brahma Reddy Battula-3
Dear Hadoop Developers

I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
been working on this feature from last several months.
This feature work received the contributions from different companies. All
of the feature development happened smoothly and collaboratively in JIRAs.

Kindly do take a look at the branch and raise issues/concerns that need to
be addressed before the merge.

*Highlights of HDFS-13891 Branch:*
=============================

Adding Security to RBF(1)
Adding Missing Client API's(2)
Improvements/Bug Fixing
  Critical - HDFS-13637, HDFS-13834

*Commits:*
========

No of JIRAs Resolved: 72

All this commits are in RBF Module. No changes in hdfs/common.

*Tested Cluster:*
=============

Most of these changes verified at Uber,Microsoft,Huawei and some other
compaines.

*Uber*: Most changes are running in production @Uber including the critical
security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
Zookeeper as a state store to hold delegation tokens were also stress
tested to hold more than 2 Million tokens. --CR Hota

*MicroSoft*: Most of these changes are currently running in production at
Microsoft.The security has also been tested in a 500 server cluster with 4
subclsuters. --Inigo Goiri

*Huawei* : Deployed all this changes in 20 node cluster with 3
routers.Planning deploy 10K production cluster.

*Contributors:*
===========

Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions and
contributing to this.

*Future Tasks:*
============

 will cleanup the jira's under this umbrella and contiue to work.

Reference:
1) https://issues.apache.org/jira/browse/HDFS-13532
2) https://issues.apache.org/jira/browse/HDFS-13655




--Brahma Reddy Battula
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk

Akira Ajisaka-4
Thanks Brahma for starting the discussion.
I'm +1 for merging this.

FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
with 2 routers (not in production) and running several tests.

Regards,
Akira

On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <[hidden email]> wrote:

>
> Dear Hadoop Developers
>
> I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
> been working on this feature from last several months.
> This feature work received the contributions from different companies. All
> of the feature development happened smoothly and collaboratively in JIRAs.
>
> Kindly do take a look at the branch and raise issues/concerns that need to
> be addressed before the merge.
>
> *Highlights of HDFS-13891 Branch:*
> =============================
>
> Adding Security to RBF(1)
> Adding Missing Client API's(2)
> Improvements/Bug Fixing
>   Critical - HDFS-13637, HDFS-13834
>
> *Commits:*
> ========
>
> No of JIRAs Resolved: 72
>
> All this commits are in RBF Module. No changes in hdfs/common.
>
> *Tested Cluster:*
> =============
>
> Most of these changes verified at Uber,Microsoft,Huawei and some other
> compaines.
>
> *Uber*: Most changes are running in production @Uber including the critical
> security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
> Zookeeper as a state store to hold delegation tokens were also stress
> tested to hold more than 2 Million tokens. --CR Hota
>
> *MicroSoft*: Most of these changes are currently running in production at
> Microsoft.The security has also been tested in a 500 server cluster with 4
> subclsuters. --Inigo Goiri
>
> *Huawei* : Deployed all this changes in 20 node cluster with 3
> routers.Planning deploy 10K production cluster.
>
> *Contributors:*
> ===========
>
> Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
> Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
> Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions and
> contributing to this.
>
> *Future Tasks:*
> ============
>
>  will cleanup the jira's under this umbrella and contiue to work.
>
> Reference:
> 1) https://issues.apache.org/jira/browse/HDFS-13532
> 2) https://issues.apache.org/jira/browse/HDFS-13655
>
>
>
>
> --Brahma Reddy Battula

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk

Iñigo Goiri
Thank you Brahma for pushing this.

As you mentioned, we have already taken most of the changes into production.
I want to highlight that the main contribution is the addition of security.
We have been able to test this at a smaller scale (~500 servers and 4
subclusters) and the performance is great with our current ZooKeeper
deployment.
I would also like to highlight that all the changes are constrained to
hadoop-hdfs-rbf and there is no differences in commons or HDFS.

+1 on merging

Inigo

On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka <[hidden email]> wrote:

> Thanks Brahma for starting the discussion.
> I'm +1 for merging this.
>
> FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
> with 2 routers (not in production) and running several tests.
>
> Regards,
> Akira
>
> On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <[hidden email]>
> wrote:
> >
> > Dear Hadoop Developers
> >
> > I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
> > been working on this feature from last several months.
> > This feature work received the contributions from different companies.
> All
> > of the feature development happened smoothly and collaboratively in
> JIRAs.
> >
> > Kindly do take a look at the branch and raise issues/concerns that need
> to
> > be addressed before the merge.
> >
> > *Highlights of HDFS-13891 Branch:*
> > =============================
> >
> > Adding Security to RBF(1)
> > Adding Missing Client API's(2)
> > Improvements/Bug Fixing
> >   Critical - HDFS-13637, HDFS-13834
> >
> > *Commits:*
> > ========
> >
> > No of JIRAs Resolved: 72
> >
> > All this commits are in RBF Module. No changes in hdfs/common.
> >
> > *Tested Cluster:*
> > =============
> >
> > Most of these changes verified at Uber,Microsoft,Huawei and some other
> > compaines.
> >
> > *Uber*: Most changes are running in production @Uber including the
> critical
> > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
> > Zookeeper as a state store to hold delegation tokens were also stress
> > tested to hold more than 2 Million tokens. --CR Hota
> >
> > *MicroSoft*: Most of these changes are currently running in production at
> > Microsoft.The security has also been tested in a 500 server cluster with
> 4
> > subclsuters. --Inigo Goiri
> >
> > *Huawei* : Deployed all this changes in 20 node cluster with 3
> > routers.Planning deploy 10K production cluster.
> >
> > *Contributors:*
> > ===========
> >
> > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
> > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
> > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions
> and
> > contributing to this.
> >
> > *Future Tasks:*
> > ============
> >
> >  will cleanup the jira's under this umbrella and contiue to work.
> >
> > Reference:
> > 1) https://issues.apache.org/jira/browse/HDFS-13532
> > 2) https://issues.apache.org/jira/browse/HDFS-13655
> >
> >
> >
> >
> > --Brahma Reddy Battula
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk

Xiaoqiao He
Thanks Brahma for starting the thread.
+1 for merging.

He Xiaoqiao

On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota <[hidden email]>
wrote:

> Thanks Brahma initiating this.
> +1(non-binding) for merge.
>
> @Uber we have almost all changes specially rbf security in production for a
> while now without issues.
>
> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
> [hidden email]> wrote:
>
> > +1 on merging.
> >
> > Thanks Brahma for starting the thread.
> >
> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri <[hidden email]> wrote:
> >
> > > Thank you Brahma for pushing this.
> > >
> > > As you mentioned, we have already taken most of the changes into
> > > production.
> > > I want to highlight that the main contribution is the addition of
> > security.
> > > We have been able to test this at a smaller scale (~500 servers and 4
> > > subclusters) and the performance is great with our current ZooKeeper
> > > deployment.
> > > I would also like to highlight that all the changes are constrained to
> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS.
> > >
> > > +1 on merging
> > >
> > > Inigo
> > >
> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka <[hidden email]>
> > wrote:
> > >
> > > > Thanks Brahma for starting the discussion.
> > > > I'm +1 for merging this.
> > > >
> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
> > > > with 2 routers (not in production) and running several tests.
> > > >
> > > > Regards,
> > > > Akira
> > > >
> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
> > [hidden email]>
> > > > wrote:
> > > > >
> > > > > Dear Hadoop Developers
> > > > >
> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk.
> We
> > > have
> > > > > been working on this feature from last several months.
> > > > > This feature work received the contributions from different
> > companies.
> > > > All
> > > > > of the feature development happened smoothly and collaboratively in
> > > > JIRAs.
> > > > >
> > > > > Kindly do take a look at the branch and raise issues/concerns that
> > need
> > > > to
> > > > > be addressed before the merge.
> > > > >
> > > > > *Highlights of HDFS-13891 Branch:*
> > > > > =============================
> > > > >
> > > > > Adding Security to RBF(1)
> > > > > Adding Missing Client API's(2)
> > > > > Improvements/Bug Fixing
> > > > >   Critical - HDFS-13637, HDFS-13834
> > > > >
> > > > > *Commits:*
> > > > > ========
> > > > >
> > > > > No of JIRAs Resolved: 72
> > > > >
> > > > > All this commits are in RBF Module. No changes in hdfs/common.
> > > > >
> > > > > *Tested Cluster:*
> > > > > =============
> > > > >
> > > > > Most of these changes verified at Uber,Microsoft,Huawei and some
> > other
> > > > > compaines.
> > > > >
> > > > > *Uber*: Most changes are running in production @Uber including the
> > > > critical
> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS
> Routers.
> > > > > Zookeeper as a state store to hold delegation tokens were also
> stress
> > > > > tested to hold more than 2 Million tokens. --CR Hota
> > > > >
> > > > > *MicroSoft*: Most of these changes are currently running in
> > production
> > > at
> > > > > Microsoft.The security has also been tested in a 500 server cluster
> > > with
> > > > 4
> > > > > subclsuters. --Inigo Goiri
> > > > >
> > > > > *Huawei* : Deployed all this changes in 20 node cluster with 3
> > > > > routers.Planning deploy 10K production cluster.
> > > > >
> > > > > *Contributors:*
> > > > > ===========
> > > > >
> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu
> Asanuma,Shubham
> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> > > > > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He
> > > Xiaoqiao,Sherwood
> > > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving
> > discussions
> > > > and
> > > > > contributing to this.
> > > > >
> > > > > *Future Tasks:*
> > > > > ============
> > > > >
> > > > >  will cleanup the jira's under this umbrella and contiue to work.
> > > > >
> > > > > Reference:
> > > > > 1) https://issues.apache.org/jira/browse/HDFS-13532
> > > > > 2) https://issues.apache.org/jira/browse/HDFS-13655
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --Brahma Reddy Battula
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: [hidden email]
> > > > For additional commands, e-mail: [hidden email]
> > > >
> > > >
> > >
> >
>
>
> --
> Regards,
> Chitta
>
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Merge HDFS-13891(RBF) to trunk

Brahma Reddy Battula-3
Hi All,

    Given the positive response to the discussion thread [1], here is the
formal vote thread to merge HDFS-13891 in to trunk.

    Summary of code changes:
    1. Code changes for this branch are done in the hadoop-hdfs-rbf
subproject, there is no impact to hadoop-hdfs and hadoop-common.
    2. Added Security support for RBF
    3. Added Missing Client Protocol API's
    4. Bug Fixes/ Improvments


     The vote will run for 7 days, ending Sat June 15th. I will start this
vote with my +1.

Regards,
Brahma Reddy Battula

1).
https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He <[hidden email]> wrote:

> Thanks Brahma for starting the thread.
> +1 for merging.
>
> He Xiaoqiao
>
> On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota <[hidden email]>
> wrote:
>
>> Thanks Brahma initiating this.
>> +1(non-binding) for merge.
>>
>> @Uber we have almost all changes specially rbf security in production for
>> a
>> while now without issues.
>>
>> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
>> [hidden email]> wrote:
>>
>> > +1 on merging.
>> >
>> > Thanks Brahma for starting the thread.
>> >
>> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri <[hidden email]> wrote:
>> >
>> > > Thank you Brahma for pushing this.
>> > >
>> > > As you mentioned, we have already taken most of the changes into
>> > > production.
>> > > I want to highlight that the main contribution is the addition of
>> > security.
>> > > We have been able to test this at a smaller scale (~500 servers and 4
>> > > subclusters) and the performance is great with our current ZooKeeper
>> > > deployment.
>> > > I would also like to highlight that all the changes are constrained to
>> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS.
>> > >
>> > > +1 on merging
>> > >
>> > > Inigo
>> > >
>> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka <[hidden email]>
>> > wrote:
>> > >
>> > > > Thanks Brahma for starting the discussion.
>> > > > I'm +1 for merging this.
>> > > >
>> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
>> > > > with 2 routers (not in production) and running several tests.
>> > > >
>> > > > Regards,
>> > > > Akira
>> > > >
>> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
>> > [hidden email]>
>> > > > wrote:
>> > > > >
>> > > > > Dear Hadoop Developers
>> > > > >
>> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk.
>> We
>> > > have
>> > > > > been working on this feature from last several months.
>> > > > > This feature work received the contributions from different
>> > companies.
>> > > > All
>> > > > > of the feature development happened smoothly and collaboratively
>> in
>> > > > JIRAs.
>> > > > >
>> > > > > Kindly do take a look at the branch and raise issues/concerns that
>> > need
>> > > > to
>> > > > > be addressed before the merge.
>> > > > >
>> > > > > *Highlights of HDFS-13891 Branch:*
>> > > > > =============================
>> > > > >
>> > > > > Adding Security to RBF(1)
>> > > > > Adding Missing Client API's(2)
>> > > > > Improvements/Bug Fixing
>> > > > >   Critical - HDFS-13637, HDFS-13834
>> > > > >
>> > > > > *Commits:*
>> > > > > ========
>> > > > >
>> > > > > No of JIRAs Resolved: 72
>> > > > >
>> > > > > All this commits are in RBF Module. No changes in hdfs/common.
>> > > > >
>> > > > > *Tested Cluster:*
>> > > > > =============
>> > > > >
>> > > > > Most of these changes verified at Uber,Microsoft,Huawei and some
>> > other
>> > > > > compaines.
>> > > > >
>> > > > > *Uber*: Most changes are running in production @Uber including the
>> > > > critical
>> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS
>> Routers.
>> > > > > Zookeeper as a state store to hold delegation tokens were also
>> stress
>> > > > > tested to hold more than 2 Million tokens. --CR Hota
>> > > > >
>> > > > > *MicroSoft*: Most of these changes are currently running in
>> > production
>> > > at
>> > > > > Microsoft.The security has also been tested in a 500 server
>> cluster
>> > > with
>> > > > 4
>> > > > > subclsuters. --Inigo Goiri
>> > > > >
>> > > > > *Huawei* : Deployed all this changes in 20 node cluster with 3
>> > > > > routers.Planning deploy 10K production cluster.
>> > > > >
>> > > > > *Contributors:*
>> > > > > ===========
>> > > > >
>> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu
>> Asanuma,Shubham
>> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
>> > > > > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He
>> > > Xiaoqiao,Sherwood
>> > > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving
>> > discussions
>> > > > and
>> > > > > contributing to this.
>> > > > >
>> > > > > *Future Tasks:*
>> > > > > ============
>> > > > >
>> > > > >  will cleanup the jira's under this umbrella and contiue to work.
>> > > > >
>> > > > > Reference:
>> > > > > 1) https://issues.apache.org/jira/browse/HDFS-13532
>> > > > > 2) https://issues.apache.org/jira/browse/HDFS-13655
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --Brahma Reddy Battula
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: [hidden email]
>> > > > For additional commands, e-mail: [hidden email]
>> > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> Regards,
>> Chitta
>>
>

--



--Brahma Reddy Battula
--



--Brahma Reddy Battula
Reply | Threaded
Open this post in threaded view
|

[VOTE] Merge HDFS-13891(RBF) to trunk

Brahma Reddy Battula-3
Updated mail...

---------- Forwarded message ---------
From: Brahma Reddy Battula <[hidden email]>
Date: Sun, Jun 9, 2019 at 7:26 PM
Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk
To: Xiaoqiao He <[hidden email]>
Cc: Akira Ajisaka <[hidden email]>, Chittaranjan Hota <
[hidden email]>, Giovanni Matteo Fumarola <
[hidden email]>, Hadoop Common <[hidden email]>,
Hdfs-dev <[hidden email]>, Iñigo Goiri <[hidden email]>


Hi All,

    Given the positive response to the discussion thread [1], here is the
formal vote thread to merge HDFS-13891 in to trunk.

    Summary of code changes:
    1. Code changes for this branch are done in the hadoop-hdfs-rbf
subproject, there is no impact to hadoop-hdfs and hadoop-common.
    2. Added Security support for RBF
    3. Added Missing Client Protocol API's
    4. Bug Fixes/ Improvments


     The vote will run for 7 days, ending Sat June 15th. I will start this
vote with my +1.

Regards,
Brahma Reddy Battula

1).
https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He <[hidden email]> wrote:

> Thanks Brahma for starting the thread.
> +1 for merging.
>
> He Xiaoqiao
>
> On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota <[hidden email]>
> wrote:
>
>> Thanks Brahma initiating this.
>> +1(non-binding) for merge.
>>
>> @Uber we have almost all changes specially rbf security in production for
>> a
>> while now without issues.
>>
>> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
>> [hidden email]> wrote:
>>
>> > +1 on merging.
>> >
>> > Thanks Brahma for starting the thread.
>> >
>> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri <[hidden email]> wrote:
>> >
>> > > Thank you Brahma for pushing this.
>> > >
>> > > As you mentioned, we have already taken most of the changes into
>> > > production.
>> > > I want to highlight that the main contribution is the addition of
>> > security.
>> > > We have been able to test this at a smaller scale (~500 servers and 4
>> > > subclusters) and the performance is great with our current ZooKeeper
>> > > deployment.
>> > > I would also like to highlight that all the changes are constrained to
>> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS.
>> > >
>> > > +1 on merging
>> > >
>> > > Inigo
>> > >
>> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka <[hidden email]>
>> > wrote:
>> > >
>> > > > Thanks Brahma for starting the discussion.
>> > > > I'm +1 for merging this.
>> > > >
>> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
>> > > > with 2 routers (not in production) and running several tests.
>> > > >
>> > > > Regards,
>> > > > Akira
>> > > >
>> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
>> > [hidden email]>
>> > > > wrote:
>> > > > >
>> > > > > Dear Hadoop Developers
>> > > > >
>> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk.
>> We
>> > > have
>> > > > > been working on this feature from last several months.
>> > > > > This feature work received the contributions from different
>> > companies.
>> > > > All
>> > > > > of the feature development happened smoothly and collaboratively
>> in
>> > > > JIRAs.
>> > > > >
>> > > > > Kindly do take a look at the branch and raise issues/concerns that
>> > need
>> > > > to
>> > > > > be addressed before the merge.
>> > > > >
>> > > > > *Highlights of HDFS-13891 Branch:*
>> > > > > =============================
>> > > > >
>> > > > > Adding Security to RBF(1)
>> > > > > Adding Missing Client API's(2)
>> > > > > Improvements/Bug Fixing
>> > > > >   Critical - HDFS-13637, HDFS-13834
>> > > > >
>> > > > > *Commits:*
>> > > > > ========
>> > > > >
>> > > > > No of JIRAs Resolved: 72
>> > > > >
>> > > > > All this commits are in RBF Module. No changes in hdfs/common.
>> > > > >
>> > > > > *Tested Cluster:*
>> > > > > =============
>> > > > >
>> > > > > Most of these changes verified at Uber,Microsoft,Huawei and some
>> > other
>> > > > > compaines.
>> > > > >
>> > > > > *Uber*: Most changes are running in production @Uber including the
>> > > > critical
>> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS
>> Routers.
>> > > > > Zookeeper as a state store to hold delegation tokens were also
>> stress
>> > > > > tested to hold more than 2 Million tokens. --CR Hota
>> > > > >
>> > > > > *MicroSoft*: Most of these changes are currently running in
>> > production
>> > > at
>> > > > > Microsoft.The security has also been tested in a 500 server
>> cluster
>> > > with
>> > > > 4
>> > > > > subclsuters. --Inigo Goiri
>> > > > >
>> > > > > *Huawei* : Deployed all this changes in 20 node cluster with 3
>> > > > > routers.Planning deploy 10K production cluster.
>> > > > >
>> > > > > *Contributors:*
>> > > > > ===========
>> > > > >
>> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu
>> Asanuma,Shubham
>> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
>> > > > > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He
>> > > Xiaoqiao,Sherwood
>> > > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving
>> > discussions
>> > > > and
>> > > > > contributing to this.
>> > > > >
>> > > > > *Future Tasks:*
>> > > > > ============
>> > > > >
>> > > > >  will cleanup the jira's under this umbrella and contiue to work.
>> > > > >
>> > > > > Reference:
>> > > > > 1) https://issues.apache.org/jira/browse/HDFS-13532
>> > > > > 2) https://issues.apache.org/jira/browse/HDFS-13655
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --Brahma Reddy Battula
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: [hidden email]
>> > > > For additional commands, e-mail: [hidden email]
>> > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> Regards,
>> Chitta
>
>
>>

--



--Brahma Reddy Battula
--



--Brahma Reddy Battula
--



--Brahma Reddy Battula
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Merge HDFS-13891(RBF) to trunk

Akira Ajisaka-2
In reply to this post by Brahma Reddy Battula-3
+1

-Akira

On Sun, Jun 9, 2019 at 10:56 PM Brahma Reddy Battula <[hidden email]> wrote:

>
> Hi All,
>
>     Given the positive response to the discussion thread [1], here is the
> formal vote thread to merge HDFS-13891 in to trunk.
>
>     Summary of code changes:
>     1. Code changes for this branch are done in the hadoop-hdfs-rbf
> subproject, there is no impact to hadoop-hdfs and hadoop-common.
>     2. Added Security support for RBF
>     3. Added Missing Client Protocol API's
>     4. Bug Fixes/ Improvments
>
>
>      The vote will run for 7 days, ending Sat June 15th. I will start this
> vote with my +1.
>
> Regards,
> Brahma Reddy Battula
>
> 1).
> https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E
>
>
> On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He <[hidden email]> wrote:
>
> > Thanks Brahma for starting the thread.
> > +1 for merging.
> >
> > He Xiaoqiao
> >
> > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota <[hidden email]>
> > wrote:
> >
> >> Thanks Brahma initiating this.
> >> +1(non-binding) for merge.
> >>
> >> @Uber we have almost all changes specially rbf security in production for
> >> a
> >> while now without issues.
> >>
> >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
> >> [hidden email]> wrote:
> >>
> >> > +1 on merging.
> >> >
> >> > Thanks Brahma for starting the thread.
> >> >
> >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri <[hidden email]> wrote:
> >> >
> >> > > Thank you Brahma for pushing this.
> >> > >
> >> > > As you mentioned, we have already taken most of the changes into
> >> > > production.
> >> > > I want to highlight that the main contribution is the addition of
> >> > security.
> >> > > We have been able to test this at a smaller scale (~500 servers and 4
> >> > > subclusters) and the performance is great with our current ZooKeeper
> >> > > deployment.
> >> > > I would also like to highlight that all the changes are constrained to
> >> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS.
> >> > >
> >> > > +1 on merging
> >> > >
> >> > > Inigo
> >> > >
> >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka <[hidden email]>
> >> > wrote:
> >> > >
> >> > > > Thanks Brahma for starting the discussion.
> >> > > > I'm +1 for merging this.
> >> > > >
> >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
> >> > > > with 2 routers (not in production) and running several tests.
> >> > > >
> >> > > > Regards,
> >> > > > Akira
> >> > > >
> >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
> >> > [hidden email]>
> >> > > > wrote:
> >> > > > >
> >> > > > > Dear Hadoop Developers
> >> > > > >
> >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk.
> >> We
> >> > > have
> >> > > > > been working on this feature from last several months.
> >> > > > > This feature work received the contributions from different
> >> > companies.
> >> > > > All
> >> > > > > of the feature development happened smoothly and collaboratively
> >> in
> >> > > > JIRAs.
> >> > > > >
> >> > > > > Kindly do take a look at the branch and raise issues/concerns that
> >> > need
> >> > > > to
> >> > > > > be addressed before the merge.
> >> > > > >
> >> > > > > *Highlights of HDFS-13891 Branch:*
> >> > > > > =============================
> >> > > > >
> >> > > > > Adding Security to RBF(1)
> >> > > > > Adding Missing Client API's(2)
> >> > > > > Improvements/Bug Fixing
> >> > > > >   Critical - HDFS-13637, HDFS-13834
> >> > > > >
> >> > > > > *Commits:*
> >> > > > > ========
> >> > > > >
> >> > > > > No of JIRAs Resolved: 72
> >> > > > >
> >> > > > > All this commits are in RBF Module. No changes in hdfs/common.
> >> > > > >
> >> > > > > *Tested Cluster:*
> >> > > > > =============
> >> > > > >
> >> > > > > Most of these changes verified at Uber,Microsoft,Huawei and some
> >> > other
> >> > > > > compaines.
> >> > > > >
> >> > > > > *Uber*: Most changes are running in production @Uber including the
> >> > > > critical
> >> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS
> >> Routers.
> >> > > > > Zookeeper as a state store to hold delegation tokens were also
> >> stress
> >> > > > > tested to hold more than 2 Million tokens. --CR Hota
> >> > > > >
> >> > > > > *MicroSoft*: Most of these changes are currently running in
> >> > production
> >> > > at
> >> > > > > Microsoft.The security has also been tested in a 500 server
> >> cluster
> >> > > with
> >> > > > 4
> >> > > > > subclsuters. --Inigo Goiri
> >> > > > >
> >> > > > > *Huawei* : Deployed all this changes in 20 node cluster with 3
> >> > > > > routers.Planning deploy 10K production cluster.
> >> > > > >
> >> > > > > *Contributors:*
> >> > > > > ===========
> >> > > > >
> >> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu
> >> Asanuma,Shubham
> >> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> >> > > > > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He
> >> > > Xiaoqiao,Sherwood
> >> > > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving
> >> > discussions
> >> > > > and
> >> > > > > contributing to this.
> >> > > > >
> >> > > > > *Future Tasks:*
> >> > > > > ============
> >> > > > >
> >> > > > >  will cleanup the jira's under this umbrella and contiue to work.
> >> > > > >
> >> > > > > Reference:
> >> > > > > 1) https://issues.apache.org/jira/browse/HDFS-13532
> >> > > > > 2) https://issues.apache.org/jira/browse/HDFS-13655
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --Brahma Reddy Battula
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: [hidden email]
> >> > > > For additional commands, e-mail: [hidden email]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> Regards,
> >> Chitta
> >>
> >
>
> --
>
>
>
> --Brahma Reddy Battula
> --
>
>
>
> --Brahma Reddy Battula

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Merge HDFS-13891(RBF) to trunk

Xiaoqiao He
In reply to this post by Brahma Reddy Battula-3
+1 (non-binding)

- Try to merge branch HDFS-13891(RBF) to trunk at local and no conflict or
failure.
- Built from merged sources.
- Ran 85 RBF test class at local and result shows: (Tests run: 639,
Failures: 2, Errors: 2, Skipped: 2) - failed tests include
#TestRouterWithSecureStartup and #TestRouterHttpDelegationToken. I don't
think it is blocked issue.

Thanks Brahma for organizing the merge.


On Mon, Jun 10, 2019 at 12:27 PM Ranith Sardar <[hidden email]>
wrote:

> +1 (Non-binding)
>
> -----Original Message-----
> From: Brahma Reddy Battula [mailto:[hidden email]]
> Sent: 09 June 2019 19:31
> To: Hadoop Common <[hidden email]>; Hdfs-dev <
> [hidden email]>
> Subject: [VOTE] Merge HDFS-13891(RBF) to trunk
>
> Updated mail...
>
> ---------- Forwarded message ---------
> From: Brahma Reddy Battula <[hidden email]>
> Date: Sun, Jun 9, 2019 at 7:26 PM
> Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk
> To: Xiaoqiao He <[hidden email]>
> Cc: Akira Ajisaka <[hidden email]>, Chittaranjan Hota <
> [hidden email]>, Giovanni Matteo Fumarola <
> [hidden email]>, Hadoop Common <[hidden email]>,
> Hdfs-dev <[hidden email]>, Iñigo Goiri <[hidden email]>
>
>
> Hi All,
>
>     Given the positive response to the discussion thread [1], here is the
> formal vote thread to merge HDFS-13891 in to trunk.
>
>     Summary of code changes:
>     1. Code changes for this branch are done in the hadoop-hdfs-rbf
> subproject, there is no impact to hadoop-hdfs and hadoop-common.
>     2. Added Security support for RBF
>     3. Added Missing Client Protocol API's
>     4. Bug Fixes/ Improvments
>
>
>      The vote will run for 7 days, ending Sat June 15th. I will start this
> vote with my +1.
>
> Regards,
> Brahma Reddy Battula
>
> 1).
>
> https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E
>
>
> On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He <[hidden email]> wrote:
>
> > Thanks Brahma for starting the thread.
> > +1 for merging.
> >
> > He Xiaoqiao
> >
> > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota
> > <[hidden email]>
> > wrote:
> >
> >> Thanks Brahma initiating this.
> >> +1(non-binding) for merge.
> >>
> >> @Uber we have almost all changes specially rbf security in production
> >> for a while now without issues.
> >>
> >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
> >> [hidden email]> wrote:
> >>
> >> > +1 on merging.
> >> >
> >> > Thanks Brahma for starting the thread.
> >> >
> >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri <[hidden email]>
> wrote:
> >> >
> >> > > Thank you Brahma for pushing this.
> >> > >
> >> > > As you mentioned, we have already taken most of the changes into
> >> > > production.
> >> > > I want to highlight that the main contribution is the addition of
> >> > security.
> >> > > We have been able to test this at a smaller scale (~500 servers
> >> > > and 4
> >> > > subclusters) and the performance is great with our current
> >> > > ZooKeeper deployment.
> >> > > I would also like to highlight that all the changes are
> >> > > constrained to hadoop-hdfs-rbf and there is no differences in
> commons or HDFS.
> >> > >
> >> > > +1 on merging
> >> > >
> >> > > Inigo
> >> > >
> >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka
> >> > > <[hidden email]>
> >> > wrote:
> >> > >
> >> > > > Thanks Brahma for starting the discussion.
> >> > > > I'm +1 for merging this.
> >> > > >
> >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes
> >> > > > cluster with 2 routers (not in production) and running several
> tests.
> >> > > >
> >> > > > Regards,
> >> > > > Akira
> >> > > >
> >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
> >> > [hidden email]>
> >> > > > wrote:
> >> > > > >
> >> > > > > Dear Hadoop Developers
> >> > > > >
> >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into
> trunk.
> >> We
> >> > > have
> >> > > > > been working on this feature from last several months.
> >> > > > > This feature work received the contributions from different
> >> > companies.
> >> > > > All
> >> > > > > of the feature development happened smoothly and
> >> > > > > collaboratively
> >> in
> >> > > > JIRAs.
> >> > > > >
> >> > > > > Kindly do take a look at the branch and raise issues/concerns
> >> > > > > that
> >> > need
> >> > > > to
> >> > > > > be addressed before the merge.
> >> > > > >
> >> > > > > *Highlights of HDFS-13891 Branch:*
> >> > > > > =============================
> >> > > > >
> >> > > > > Adding Security to RBF(1)
> >> > > > > Adding Missing Client API's(2) Improvements/Bug Fixing
> >> > > > >   Critical - HDFS-13637, HDFS-13834
> >> > > > >
> >> > > > > *Commits:*
> >> > > > > ========
> >> > > > >
> >> > > > > No of JIRAs Resolved: 72
> >> > > > >
> >> > > > > All this commits are in RBF Module. No changes in hdfs/common.
> >> > > > >
> >> > > > > *Tested Cluster:*
> >> > > > > =============
> >> > > > >
> >> > > > > Most of these changes verified at Uber,Microsoft,Huawei and
> >> > > > > some
> >> > other
> >> > > > > compaines.
> >> > > > >
> >> > > > > *Uber*: Most changes are running in production @Uber
> >> > > > > including the
> >> > > > critical
> >> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS
> >> Routers.
> >> > > > > Zookeeper as a state store to hold delegation tokens were
> >> > > > > also
> >> stress
> >> > > > > tested to hold more than 2 Million tokens. --CR Hota
> >> > > > >
> >> > > > > *MicroSoft*: Most of these changes are currently running in
> >> > production
> >> > > at
> >> > > > > Microsoft.The security has also been tested in a 500 server
> >> cluster
> >> > > with
> >> > > > 4
> >> > > > > subclsuters. --Inigo Goiri
> >> > > > >
> >> > > > > *Huawei* : Deployed all this changes in 20 node cluster with
> >> > > > > 3 routers.Planning deploy 10K production cluster.
> >> > > > >
> >> > > > > *Contributors:*
> >> > > > > ===========
> >> > > > >
> >> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu
> >> Asanuma,Shubham
> >> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna
> >> > > > > Li,Gang Li,Surendra Singh Lihore,Ranith Sardar,Ayush
> >> > > > > Saxena,He
> >> > > Xiaoqiao,Sherwood
> >> > > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving
> >> > discussions
> >> > > > and
> >> > > > > contributing to this.
> >> > > > >
> >> > > > > *Future Tasks:*
> >> > > > > ============
> >> > > > >
> >> > > > >  will cleanup the jira's under this umbrella and contiue to
> work.
> >> > > > >
> >> > > > > Reference:
> >> > > > > 1) https://issues.apache.org/jira/browse/HDFS-13532
> >> > > > > 2) https://issues.apache.org/jira/browse/HDFS-13655
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --Brahma Reddy Battula
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: [hidden email]
> >> > > > For additional commands, e-mail:
> >> > > > [hidden email]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> Regards,
> >> Chitta
> >
> >
> >>
>
> --
>
>
>
> --Brahma Reddy Battula
> --
>
>
>
> --Brahma Reddy Battula
> --
>
>
>
> --Brahma Reddy Battula
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk

Arpit Agarwal-2
In reply to this post by Brahma Reddy Battula-3
I scanned the merge payload for changes to non-RBF code. The changes are minimal, which is good.

The only commit that I didn’t understand was:
https://issues.apache.org/jira/browse/HDFS-14268 <https://issues.apache.org/jira/browse/HDFS-14268>

The jira description doesn’t make it clear why ECBlockGroupStats is modified.

+0 apart from that.


> On Jun 1, 2019, at 8:40 PM, Brahma Reddy Battula <[hidden email]> wrote:
>
> Dear Hadoop Developers
>
> I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
> been working on this feature from last several months.
> This feature work received the contributions from different companies. All
> of the feature development happened smoothly and collaboratively in JIRAs.
>
> Kindly do take a look at the branch and raise issues/concerns that need to
> be addressed before the merge.
>
> *Highlights of HDFS-13891 Branch:*
> =============================
>
> Adding Security to RBF(1)
> Adding Missing Client API's(2)
> Improvements/Bug Fixing
>  Critical - HDFS-13637, HDFS-13834
>
> *Commits:*
> ========
>
> No of JIRAs Resolved: 72
>
> All this commits are in RBF Module. No changes in hdfs/common.
>
> *Tested Cluster:*
> =============
>
> Most of these changes verified at Uber,Microsoft,Huawei and some other
> compaines.
>
> *Uber*: Most changes are running in production @Uber including the critical
> security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
> Zookeeper as a state store to hold delegation tokens were also stress
> tested to hold more than 2 Million tokens. --CR Hota
>
> *MicroSoft*: Most of these changes are currently running in production at
> Microsoft.The security has also been tested in a 500 server cluster with 4
> subclsuters. --Inigo Goiri
>
> *Huawei* : Deployed all this changes in 20 node cluster with 3
> routers.Planning deploy 10K production cluster.
>
> *Contributors:*
> ===========
>
> Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
> Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
> Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions and
> contributing to this.
>
> *Future Tasks:*
> ============
>
> will cleanup the jira's under this umbrella and contiue to work.
>
> Reference:
> 1) https://issues.apache.org/jira/browse/HDFS-13532
> 2) https://issues.apache.org/jira/browse/HDFS-13655
>
>
>
>
> --Brahma Reddy Battula

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk

Iñigo Goiri
+1 from my side.


I also added a comment to HDFS-14268 explaining the reasons to modify
ECBlockGroupStats.
We can follow up into possible changes there if needed.

On Mon, Jun 10, 2019 at 8:10 AM Arpit Agarwal <[hidden email]>
wrote:

> I scanned the merge payload for changes to non-RBF code. The changes are
> minimal, which is good.
>
> The only commit that I didn’t understand was:
> https://issues.apache.org/jira/browse/HDFS-14268 <
> https://issues.apache.org/jira/browse/HDFS-14268>
>
> The jira description doesn’t make it clear why ECBlockGroupStats is
> modified.
>
> +0 apart from that.
>
>
> > On Jun 1, 2019, at 8:40 PM, Brahma Reddy Battula <[hidden email]>
> wrote:
> >
> > Dear Hadoop Developers
> >
> > I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
> > been working on this feature from last several months.
> > This feature work received the contributions from different companies.
> All
> > of the feature development happened smoothly and collaboratively in
> JIRAs.
> >
> > Kindly do take a look at the branch and raise issues/concerns that need
> to
> > be addressed before the merge.
> >
> > *Highlights of HDFS-13891 Branch:*
> > =============================
> >
> > Adding Security to RBF(1)
> > Adding Missing Client API's(2)
> > Improvements/Bug Fixing
> >  Critical - HDFS-13637, HDFS-13834
> >
> > *Commits:*
> > ========
> >
> > No of JIRAs Resolved: 72
> >
> > All this commits are in RBF Module. No changes in hdfs/common.
> >
> > *Tested Cluster:*
> > =============
> >
> > Most of these changes verified at Uber,Microsoft,Huawei and some other
> > compaines.
> >
> > *Uber*: Most changes are running in production @Uber including the
> critical
> > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
> > Zookeeper as a state store to hold delegation tokens were also stress
> > tested to hold more than 2 Million tokens. --CR Hota
> >
> > *MicroSoft*: Most of these changes are currently running in production at
> > Microsoft.The security has also been tested in a 500 server cluster with
> 4
> > subclsuters. --Inigo Goiri
> >
> > *Huawei* : Deployed all this changes in 20 node cluster with 3
> > routers.Planning deploy 10K production cluster.
> >
> > *Contributors:*
> > ===========
> >
> > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
> > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
> > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions
> and
> > contributing to this.
> >
> > *Future Tasks:*
> > ============
> >
> > will cleanup the jira's under this umbrella and contiue to work.
> >
> > Reference:
> > 1) https://issues.apache.org/jira/browse/HDFS-13532
> > 2) https://issues.apache.org/jira/browse/HDFS-13655
> >
> >
> >
> >
> > --Brahma Reddy Battula
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk

Brahma Reddy Battula-3
In reply to this post by Arpit Agarwal-2
Dear Arpit,

Thanks for taking look into it.

ECBlockGroupStats.merge() is Utility method which is moved to
hadoop-hdfs-client module. Ideally it could have been seperate jira. But
this changes will not induce any issues, will take necessary action for
this.

On Mon, Jun 10, 2019 at 8:40 PM Arpit Agarwal <[hidden email]> wrote:

> I scanned the merge payload for changes to non-RBF code. The changes are
> minimal, which is good.
>
> The only commit that I didn’t understand was:
> https://issues.apache.org/jira/browse/HDFS-14268
>
> The jira description doesn’t make it clear why ECBlockGroupStats is
> modified.
>
> +0 apart from that.
>
>
> On Jun 1, 2019, at 8:40 PM, Brahma Reddy Battula <[hidden email]>
> wrote:
>
> Dear Hadoop Developers
>
> I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
> been working on this feature from last several months.
> This feature work received the contributions from different companies. All
> of the feature development happened smoothly and collaboratively in JIRAs.
>
> Kindly do take a look at the branch and raise issues/concerns that need to
> be addressed before the merge.
>
> *Highlights of HDFS-13891 Branch:*
> =============================
>
> Adding Security to RBF(1)
> Adding Missing Client API's(2)
> Improvements/Bug Fixing
>  Critical - HDFS-13637, HDFS-13834
>
> *Commits:*
> ========
>
> No of JIRAs Resolved: 72
>
> All this commits are in RBF Module. No changes in hdfs/common.
>
> *Tested Cluster:*
> =============
>
> Most of these changes verified at Uber,Microsoft,Huawei and some other
> compaines.
>
> *Uber*: Most changes are running in production @Uber including the critical
> security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
> Zookeeper as a state store to hold delegation tokens were also stress
> tested to hold more than 2 Million tokens. --CR Hota
>
> *MicroSoft*: Most of these changes are currently running in production at
> Microsoft.The security has also been tested in a 500 server cluster with 4
> subclsuters. --Inigo Goiri
>
> *Huawei* : Deployed all this changes in 20 node cluster with 3
> routers.Planning deploy 10K production cluster.
>
> *Contributors:*
> ===========
>
> Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
> Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
> Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions and
> contributing to this.
>
> *Future Tasks:*
> ============
>
> will cleanup the jira's under this umbrella and contiue to work.
>
> Reference:
> 1) https://issues.apache.org/jira/browse/HDFS-13532
> 2) https://issues.apache.org/jira/browse/HDFS-13655
>
>
>
>
> --Brahma Reddy Battula
>
>
>

--



--Brahma Reddy Battula
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk

Arpit Agarwal-2
Thanks for the explanation Brahma and Iñigo!

+0 from me (haven’t looked at it closely enough to give a +1).

Regards,
Arpit


> On Jun 10, 2019, at 10:12 AM, Brahma Reddy Battula <[hidden email]> wrote:
>
> Dear Arpit,
>
> Thanks for taking look into it.
>
> ECBlockGroupStats.merge() is Utility method which is moved to hadoop-hdfs-client module. Ideally it could have been seperate jira. But this changes will not induce any issues, will take necessary action for this.
>
> On Mon, Jun 10, 2019 at 8:40 PM Arpit Agarwal <[hidden email] <mailto:[hidden email]>> wrote:
> I scanned the merge payload for changes to non-RBF code. The changes are minimal, which is good.
>
> The only commit that I didn’t understand was:
> https://issues.apache.org/jira/browse/HDFS-14268 <https://issues.apache.org/jira/browse/HDFS-14268>
>
> The jira description doesn’t make it clear why ECBlockGroupStats is modified.
>
> +0 apart from that.
>
>
>> On Jun 1, 2019, at 8:40 PM, Brahma Reddy Battula <[hidden email] <mailto:[hidden email]>> wrote:
>>
>> Dear Hadoop Developers
>>
>> I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
>> been working on this feature from last several months.
>> This feature work received the contributions from different companies. All
>> of the feature development happened smoothly and collaboratively in JIRAs.
>>
>> Kindly do take a look at the branch and raise issues/concerns that need to
>> be addressed before the merge.
>>
>> *Highlights of HDFS-13891 Branch:*
>> =============================
>>
>> Adding Security to RBF(1)
>> Adding Missing Client API's(2)
>> Improvements/Bug Fixing
>>  Critical - HDFS-13637, HDFS-13834
>>
>> *Commits:*
>> ========
>>
>> No of JIRAs Resolved: 72
>>
>> All this commits are in RBF Module. No changes in hdfs/common.
>>
>> *Tested Cluster:*
>> =============
>>
>> Most of these changes verified at Uber,Microsoft,Huawei and some other
>> compaines.
>>
>> *Uber*: Most changes are running in production @Uber including the critical
>> security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
>> Zookeeper as a state store to hold delegation tokens were also stress
>> tested to hold more than 2 Million tokens. --CR Hota
>>
>> *MicroSoft*: Most of these changes are currently running in production at
>> Microsoft.The security has also been tested in a 500 server cluster with 4
>> subclsuters. --Inigo Goiri
>>
>> *Huawei* : Deployed all this changes in 20 node cluster with 3
>> routers.Planning deploy 10K production cluster.
>>
>> *Contributors:*
>> ===========
>>
>> Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
>> Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
>> Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
>> Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions and
>> contributing to this.
>>
>> *Future Tasks:*
>> ============
>>
>> will cleanup the jira's under this umbrella and contiue to work.
>>
>> Reference:
>> 1) https://issues.apache.org/jira/browse/HDFS-13532 <https://issues.apache.org/jira/browse/HDFS-13532>
>> 2) https://issues.apache.org/jira/browse/HDFS-13655 <https://issues.apache.org/jira/browse/HDFS-13655>
>>
>>
>>
>>
>> --Brahma Reddy Battula
>
>
>
> --
>
>
>
> --Brahma Reddy Battula

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk

Giovanni Matteo Fumarola
+1 from my side.


On Mon, Jun 10, 2019 at 10:56 AM Arpit Agarwal
<[hidden email]> wrote:

> Thanks for the explanation Brahma and Iñigo!
>
> +0 from me (haven’t looked at it closely enough to give a +1).
>
> Regards,
> Arpit
>
>
> > On Jun 10, 2019, at 10:12 AM, Brahma Reddy Battula <[hidden email]>
> wrote:
> >
> > Dear Arpit,
> >
> > Thanks for taking look into it.
> >
> > ECBlockGroupStats.merge() is Utility method which is moved to
> hadoop-hdfs-client module. Ideally it could have been seperate jira. But
> this changes will not induce any issues, will take necessary action for
> this.
> >
> > On Mon, Jun 10, 2019 at 8:40 PM Arpit Agarwal <[hidden email]
> <mailto:[hidden email]>> wrote:
> > I scanned the merge payload for changes to non-RBF code. The changes are
> minimal, which is good.
> >
> > The only commit that I didn’t understand was:
> > https://issues.apache.org/jira/browse/HDFS-14268 <
> https://issues.apache.org/jira/browse/HDFS-14268>
> >
> > The jira description doesn’t make it clear why ECBlockGroupStats is
> modified.
> >
> > +0 apart from that.
> >
> >
> >> On Jun 1, 2019, at 8:40 PM, Brahma Reddy Battula <[hidden email]
> <mailto:[hidden email]>> wrote:
> >>
> >> Dear Hadoop Developers
> >>
> >> I would like to propose RBF Branch (HDFS-13891) merge into trunk. We
> have
> >> been working on this feature from last several months.
> >> This feature work received the contributions from different companies.
> All
> >> of the feature development happened smoothly and collaboratively in
> JIRAs.
> >>
> >> Kindly do take a look at the branch and raise issues/concerns that need
> to
> >> be addressed before the merge.
> >>
> >> *Highlights of HDFS-13891 Branch:*
> >> =============================
> >>
> >> Adding Security to RBF(1)
> >> Adding Missing Client API's(2)
> >> Improvements/Bug Fixing
> >>  Critical - HDFS-13637, HDFS-13834
> >>
> >> *Commits:*
> >> ========
> >>
> >> No of JIRAs Resolved: 72
> >>
> >> All this commits are in RBF Module. No changes in hdfs/common.
> >>
> >> *Tested Cluster:*
> >> =============
> >>
> >> Most of these changes verified at Uber,Microsoft,Huawei and some other
> >> compaines.
> >>
> >> *Uber*: Most changes are running in production @Uber including the
> critical
> >> security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
> >> Zookeeper as a state store to hold delegation tokens were also stress
> >> tested to hold more than 2 Million tokens. --CR Hota
> >>
> >> *MicroSoft*: Most of these changes are currently running in production
> at
> >> Microsoft.The security has also been tested in a 500 server cluster
> with 4
> >> subclsuters. --Inigo Goiri
> >>
> >> *Huawei* : Deployed all this changes in 20 node cluster with 3
> >> routers.Planning deploy 10K production cluster.
> >>
> >> *Contributors:*
> >> ===========
> >>
> >> Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
> >> Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> >> Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
> >> Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions
> and
> >> contributing to this.
> >>
> >> *Future Tasks:*
> >> ============
> >>
> >> will cleanup the jira's under this umbrella and contiue to work.
> >>
> >> Reference:
> >> 1) https://issues.apache.org/jira/browse/HDFS-13532 <
> https://issues.apache.org/jira/browse/HDFS-13532>
> >> 2) https://issues.apache.org/jira/browse/HDFS-13655 <
> https://issues.apache.org/jira/browse/HDFS-13655>
> >>
> >>
> >>
> >>
> >> --Brahma Reddy Battula
> >
> >
> >
> > --
> >
> >
> >
> > --Brahma Reddy Battula
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Merge HDFS-13891(RBF) to trunk

Takanobu Asanuma
In reply to this post by Xiaoqiao He
+1(non-binding).

Regards,
- Takanobu

________________________________________
From: Xiaoqiao He <[hidden email]>
Sent: Monday, June 10, 2019 15:47
To: Ranith Sardar
Cc: Brahma Reddy Battula; Hadoop Common; Hdfs-dev
Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk

+1 (non-binding)

- Try to merge branch HDFS-13891(RBF) to trunk at local and no conflict or
failure.
- Built from merged sources.
- Ran 85 RBF test class at local and result shows: (Tests run: 639,
Failures: 2, Errors: 2, Skipped: 2) - failed tests include
#TestRouterWithSecureStartup and #TestRouterHttpDelegationToken. I don't
think it is blocked issue.

Thanks Brahma for organizing the merge.


On Mon, Jun 10, 2019 at 12:27 PM Ranith Sardar <[hidden email]>
wrote:

> +1 (Non-binding)
>
> -----Original Message-----
> From: Brahma Reddy Battula [mailto:[hidden email]]
> Sent: 09 June 2019 19:31
> To: Hadoop Common <[hidden email]>; Hdfs-dev <
> [hidden email]>
> Subject: [VOTE] Merge HDFS-13891(RBF) to trunk
>
> Updated mail...
>
> ---------- Forwarded message ---------
> From: Brahma Reddy Battula <[hidden email]>
> Date: Sun, Jun 9, 2019 at 7:26 PM
> Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk
> To: Xiaoqiao He <[hidden email]>
> Cc: Akira Ajisaka <[hidden email]>, Chittaranjan Hota <
> [hidden email]>, Giovanni Matteo Fumarola <
> [hidden email]>, Hadoop Common <[hidden email]>,
> Hdfs-dev <[hidden email]>, Iñigo Goiri <[hidden email]>
>
>
> Hi All,
>
>     Given the positive response to the discussion thread [1], here is the
> formal vote thread to merge HDFS-13891 in to trunk.
>
>     Summary of code changes:
>     1. Code changes for this branch are done in the hadoop-hdfs-rbf
> subproject, there is no impact to hadoop-hdfs and hadoop-common.
>     2. Added Security support for RBF
>     3. Added Missing Client Protocol API's
>     4. Bug Fixes/ Improvments
>
>
>      The vote will run for 7 days, ending Sat June 15th. I will start this
> vote with my +1.
>
> Regards,
> Brahma Reddy Battula
>
> 1).
>
> https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E
>
>
> On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He <[hidden email]> wrote:
>
> > Thanks Brahma for starting the thread.
> > +1 for merging.
> >
> > He Xiaoqiao
> >
> > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota
> > <[hidden email]>
> > wrote:
> >
> >> Thanks Brahma initiating this.
> >> +1(non-binding) for merge.
> >>
> >> @Uber we have almost all changes specially rbf security in production
> >> for a while now without issues.
> >>
> >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
> >> [hidden email]> wrote:
> >>
> >> > +1 on merging.
> >> >
> >> > Thanks Brahma for starting the thread.
> >> >
> >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri <[hidden email]>
> wrote:
> >> >
> >> > > Thank you Brahma for pushing this.
> >> > >
> >> > > As you mentioned, we have already taken most of the changes into
> >> > > production.
> >> > > I want to highlight that the main contribution is the addition of
> >> > security.
> >> > > We have been able to test this at a smaller scale (~500 servers
> >> > > and 4
> >> > > subclusters) and the performance is great with our current
> >> > > ZooKeeper deployment.
> >> > > I would also like to highlight that all the changes are
> >> > > constrained to hadoop-hdfs-rbf and there is no differences in
> commons or HDFS.
> >> > >
> >> > > +1 on merging
> >> > >
> >> > > Inigo
> >> > >
> >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka
> >> > > <[hidden email]>
> >> > wrote:
> >> > >
> >> > > > Thanks Brahma for starting the discussion.
> >> > > > I'm +1 for merging this.
> >> > > >
> >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes
> >> > > > cluster with 2 routers (not in production) and running several
> tests.
> >> > > >
> >> > > > Regards,
> >> > > > Akira
> >> > > >
> >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
> >> > [hidden email]>
> >> > > > wrote:
> >> > > > >
> >> > > > > Dear Hadoop Developers
> >> > > > >
> >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into
> trunk.
> >> We
> >> > > have
> >> > > > > been working on this feature from last several months.
> >> > > > > This feature work received the contributions from different
> >> > companies.
> >> > > > All
> >> > > > > of the feature development happened smoothly and
> >> > > > > collaboratively
> >> in
> >> > > > JIRAs.
> >> > > > >
> >> > > > > Kindly do take a look at the branch and raise issues/concerns
> >> > > > > that
> >> > need
> >> > > > to
> >> > > > > be addressed before the merge.
> >> > > > >
> >> > > > > *Highlights of HDFS-13891 Branch:*
> >> > > > > =============================
> >> > > > >
> >> > > > > Adding Security to RBF(1)
> >> > > > > Adding Missing Client API's(2) Improvements/Bug Fixing
> >> > > > >   Critical - HDFS-13637, HDFS-13834
> >> > > > >
> >> > > > > *Commits:*
> >> > > > > ========
> >> > > > >
> >> > > > > No of JIRAs Resolved: 72
> >> > > > >
> >> > > > > All this commits are in RBF Module. No changes in hdfs/common.
> >> > > > >
> >> > > > > *Tested Cluster:*
> >> > > > > =============
> >> > > > >
> >> > > > > Most of these changes verified at Uber,Microsoft,Huawei and
> >> > > > > some
> >> > other
> >> > > > > compaines.
> >> > > > >
> >> > > > > *Uber*: Most changes are running in production @Uber
> >> > > > > including the
> >> > > > critical
> >> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS
> >> Routers.
> >> > > > > Zookeeper as a state store to hold delegation tokens were
> >> > > > > also
> >> stress
> >> > > > > tested to hold more than 2 Million tokens. --CR Hota
> >> > > > >
> >> > > > > *MicroSoft*: Most of these changes are currently running in
> >> > production
> >> > > at
> >> > > > > Microsoft.The security has also been tested in a 500 server
> >> cluster
> >> > > with
> >> > > > 4
> >> > > > > subclsuters. --Inigo Goiri
> >> > > > >
> >> > > > > *Huawei* : Deployed all this changes in 20 node cluster with
> >> > > > > 3 routers.Planning deploy 10K production cluster.
> >> > > > >
> >> > > > > *Contributors:*
> >> > > > > ===========
> >> > > > >
> >> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu
> >> Asanuma,Shubham
> >> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna
> >> > > > > Li,Gang Li,Surendra Singh Lihore,Ranith Sardar,Ayush
> >> > > > > Saxena,He
> >> > > Xiaoqiao,Sherwood
> >> > > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving
> >> > discussions
> >> > > > and
> >> > > > > contributing to this.
> >> > > > >
> >> > > > > *Future Tasks:*
> >> > > > > ============
> >> > > > >
> >> > > > >  will cleanup the jira's under this umbrella and contiue to
> work.
> >> > > > >
> >> > > > > Reference:
> >> > > > > 1) https://issues.apache.org/jira/browse/HDFS-13532
> >> > > > > 2) https://issues.apache.org/jira/browse/HDFS-13655
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --Brahma Reddy Battula
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: [hidden email]
> >> > > > For additional commands, e-mail:
> >> > > > [hidden email]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> Regards,
> >> Chitta
> >
> >
> >>
>
> --
>
>
>
> --Brahma Reddy Battula
> --
>
>
>
> --Brahma Reddy Battula
> --
>
>
>
> --Brahma Reddy Battula
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[RESULT][VOTE] Merge HDFS-13891(RBF) to trunk

Brahma Reddy Battula-3
Vote is passed with 3 binding +1's  and no -1 ( 5 nonbinding votes) .

thanks to all for voting.

Will try to merge this week..


On Tue, Jun 11, 2019 at 1:13 PM Takanobu Asanuma <[hidden email]>
wrote:

> +1(non-binding).
>
> Regards,
> - Takanobu
>
> ________________________________________
> From: Xiaoqiao He <[hidden email]>
> Sent: Monday, June 10, 2019 15:47
> To: Ranith Sardar
> Cc: Brahma Reddy Battula; Hadoop Common; Hdfs-dev
> Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk
>
> +1 (non-binding)
>
> - Try to merge branch HDFS-13891(RBF) to trunk at local and no conflict or
> failure.
> - Built from merged sources.
> - Ran 85 RBF test class at local and result shows: (Tests run: 639,
> Failures: 2, Errors: 2, Skipped: 2) - failed tests include
> #TestRouterWithSecureStartup and #TestRouterHttpDelegationToken. I don't
> think it is blocked issue.
>
> Thanks Brahma for organizing the merge.
>
>
> On Mon, Jun 10, 2019 at 12:27 PM Ranith Sardar <[hidden email]>
> wrote:
>
> > +1 (Non-binding)
> >
> > -----Original Message-----
> > From: Brahma Reddy Battula [mailto:[hidden email]]
> > Sent: 09 June 2019 19:31
> > To: Hadoop Common <[hidden email]>; Hdfs-dev <
> > [hidden email]>
> > Subject: [VOTE] Merge HDFS-13891(RBF) to trunk
> >
> > Updated mail...
> >
> > ---------- Forwarded message ---------
> > From: Brahma Reddy Battula <[hidden email]>
> > Date: Sun, Jun 9, 2019 at 7:26 PM
> > Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk
> > To: Xiaoqiao He <[hidden email]>
> > Cc: Akira Ajisaka <[hidden email]>, Chittaranjan Hota <
> > [hidden email]>, Giovanni Matteo Fumarola <
> > [hidden email]>, Hadoop Common <
> [hidden email]>,
> > Hdfs-dev <[hidden email]>, Iñigo Goiri <[hidden email]>
> >
> >
> > Hi All,
> >
> >     Given the positive response to the discussion thread [1], here is the
> > formal vote thread to merge HDFS-13891 in to trunk.
> >
> >     Summary of code changes:
> >     1. Code changes for this branch are done in the hadoop-hdfs-rbf
> > subproject, there is no impact to hadoop-hdfs and hadoop-common.
> >     2. Added Security support for RBF
> >     3. Added Missing Client Protocol API's
> >     4. Bug Fixes/ Improvments
> >
> >
> >      The vote will run for 7 days, ending Sat June 15th. I will start
> this
> > vote with my +1.
> >
> > Regards,
> > Brahma Reddy Battula
> >
> > 1).
> >
> >
> https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E
> >
> >
> > On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He <[hidden email]> wrote:
> >
> > > Thanks Brahma for starting the thread.
> > > +1 for merging.
> > >
> > > He Xiaoqiao
> > >
> > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota
> > > <[hidden email]>
> > > wrote:
> > >
> > >> Thanks Brahma initiating this.
> > >> +1(non-binding) for merge.
> > >>
> > >> @Uber we have almost all changes specially rbf security in production
> > >> for a while now without issues.
> > >>
> > >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
> > >> [hidden email]> wrote:
> > >>
> > >> > +1 on merging.
> > >> >
> > >> > Thanks Brahma for starting the thread.
> > >> >
> > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri <[hidden email]>
> > wrote:
> > >> >
> > >> > > Thank you Brahma for pushing this.
> > >> > >
> > >> > > As you mentioned, we have already taken most of the changes into
> > >> > > production.
> > >> > > I want to highlight that the main contribution is the addition of
> > >> > security.
> > >> > > We have been able to test this at a smaller scale (~500 servers
> > >> > > and 4
> > >> > > subclusters) and the performance is great with our current
> > >> > > ZooKeeper deployment.
> > >> > > I would also like to highlight that all the changes are
> > >> > > constrained to hadoop-hdfs-rbf and there is no differences in
> > commons or HDFS.
> > >> > >
> > >> > > +1 on merging
> > >> > >
> > >> > > Inigo
> > >> > >
> > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka
> > >> > > <[hidden email]>
> > >> > wrote:
> > >> > >
> > >> > > > Thanks Brahma for starting the discussion.
> > >> > > > I'm +1 for merging this.
> > >> > > >
> > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes
> > >> > > > cluster with 2 routers (not in production) and running several
> > tests.
> > >> > > >
> > >> > > > Regards,
> > >> > > > Akira
> > >> > > >
> > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
> > >> > [hidden email]>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > Dear Hadoop Developers
> > >> > > > >
> > >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into
> > trunk.
> > >> We
> > >> > > have
> > >> > > > > been working on this feature from last several months.
> > >> > > > > This feature work received the contributions from different
> > >> > companies.
> > >> > > > All
> > >> > > > > of the feature development happened smoothly and
> > >> > > > > collaboratively
> > >> in
> > >> > > > JIRAs.
> > >> > > > >
> > >> > > > > Kindly do take a look at the branch and raise issues/concerns
> > >> > > > > that
> > >> > need
> > >> > > > to
> > >> > > > > be addressed before the merge.
> > >> > > > >
> > >> > > > > *Highlights of HDFS-13891 Branch:*
> > >> > > > > =============================
> > >> > > > >
> > >> > > > > Adding Security to RBF(1)
> > >> > > > > Adding Missing Client API's(2) Improvements/Bug Fixing
> > >> > > > >   Critical - HDFS-13637, HDFS-13834
> > >> > > > >
> > >> > > > > *Commits:*
> > >> > > > > ========
> > >> > > > >
> > >> > > > > No of JIRAs Resolved: 72
> > >> > > > >
> > >> > > > > All this commits are in RBF Module. No changes in hdfs/common.
> > >> > > > >
> > >> > > > > *Tested Cluster:*
> > >> > > > > =============
> > >> > > > >
> > >> > > > > Most of these changes verified at Uber,Microsoft,Huawei and
> > >> > > > > some
> > >> > other
> > >> > > > > compaines.
> > >> > > > >
> > >> > > > > *Uber*: Most changes are running in production @Uber
> > >> > > > > including the
> > >> > > > critical
> > >> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS
> > >> Routers.
> > >> > > > > Zookeeper as a state store to hold delegation tokens were
> > >> > > > > also
> > >> stress
> > >> > > > > tested to hold more than 2 Million tokens. --CR Hota
> > >> > > > >
> > >> > > > > *MicroSoft*: Most of these changes are currently running in
> > >> > production
> > >> > > at
> > >> > > > > Microsoft.The security has also been tested in a 500 server
> > >> cluster
> > >> > > with
> > >> > > > 4
> > >> > > > > subclsuters. --Inigo Goiri
> > >> > > > >
> > >> > > > > *Huawei* : Deployed all this changes in 20 node cluster with
> > >> > > > > 3 routers.Planning deploy 10K production cluster.
> > >> > > > >
> > >> > > > > *Contributors:*
> > >> > > > > ===========
> > >> > > > >
> > >> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu
> > >> Asanuma,Shubham
> > >> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna
> > >> > > > > Li,Gang Li,Surendra Singh Lihore,Ranith Sardar,Ayush
> > >> > > > > Saxena,He
> > >> > > Xiaoqiao,Sherwood
> > >> > > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving
> > >> > discussions
> > >> > > > and
> > >> > > > > contributing to this.
> > >> > > > >
> > >> > > > > *Future Tasks:*
> > >> > > > > ============
> > >> > > > >
> > >> > > > >  will cleanup the jira's under this umbrella and contiue to
> > work.
> > >> > > > >
> > >> > > > > Reference:
> > >> > > > > 1) https://issues.apache.org/jira/browse/HDFS-13532
> > >> > > > > 2) https://issues.apache.org/jira/browse/HDFS-13655
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --Brahma Reddy Battula
> > >> > > >
> > >> > > >
> > >> ---------------------------------------------------------------------
> > >> > > > To unsubscribe, e-mail: [hidden email]
> > >> > > > For additional commands, e-mail:
> > >> > > > [hidden email]
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> Regards,
> > >> Chitta
> > >
> > >
> > >>
> >
> > --
> >
> >
> >
> > --Brahma Reddy Battula
> > --
> >
> >
> >
> > --Brahma Reddy Battula
> > --
> >
> >
> >
> > --Brahma Reddy Battula
> >
>


--



--Brahma Reddy Battula
--



--Brahma Reddy Battula
Reply | Threaded
Open this post in threaded view
|

Re: [Notice] Merge HDFS-13891(RBF) to trunk

Brahma Reddy Battula-2
In reply to this post by Brahma Reddy Battula-3

Merged. thanks to Arpit for quoting the merge process.
________________________________
From: Brahma Reddy Battula <[hidden email]>
Sent: Tuesday, June 25, 2019 12:39 AM
To: [hidden email]; [hidden email]
Cc: Akira Ajisaka; Chittaranjan Hota; Giovanni Matteo Fumarola; Hadoop Common; Hdfs-dev; Iñigo Goiri; Hadoop Common
Subject: [Notice] Merge HDFS-13891(RBF) to trunk

Rebased HDFS-13891 against trunk and ran the jenkins (1).Only two four testcases are failing which flaky and there is already jira to track this. (HDFS-14461)


Going to merge the branch soon which contains 77 commits,so please bear with us.



1.https://builds.apache.org/job/PreCommit-HDFS-Build/27044/console


________________________________
From: Brahma Reddy Battula <[hidden email]>
Sent: Sunday, June 9, 2019 9:56 PM
To: Xiaoqiao He
Cc: Akira Ajisaka; Chittaranjan Hota; Giovanni Matteo Fumarola; Hadoop Common; Hdfs-dev; Iñigo Goiri
Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk

Hi All,

    Given the positive response to the discussion thread [1], here is the
formal vote thread to merge HDFS-13891 in to trunk.

    Summary of code changes:
    1. Code changes for this branch are done in the hadoop-hdfs-rbf
subproject, there is no impact to hadoop-hdfs and hadoop-common.
    2. Added Security support for RBF
    3. Added Missing Client Protocol API's
    4. Bug Fixes/ Improvments


     The vote will run for 7 days, ending Sat June 15th. I will start this
vote with my +1.

Regards,
Brahma Reddy Battula

1).
https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He <[hidden email]> wrote:

> Thanks Brahma for starting the thread.
> +1 for merging.
>
> He Xiaoqiao
>
> On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota <[hidden email]>
> wrote:
>
>> Thanks Brahma initiating this.
>> +1(non-binding) for merge.
>>
>> @Uber we have almost all changes specially rbf security in production for
>> a
>> while now without issues.
>>
>> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
>> [hidden email]> wrote:
>>
>> > +1 on merging.
>> >
>> > Thanks Brahma for starting the thread.
>> >
>> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri <[hidden email]> wrote:
>> >
>> > > Thank you Brahma for pushing this.
>> > >
>> > > As you mentioned, we have already taken most of the changes into
>> > > production.
>> > > I want to highlight that the main contribution is the addition of
>> > security.
>> > > We have been able to test this at a smaller scale (~500 servers and 4
>> > > subclusters) and the performance is great with our current ZooKeeper
>> > > deployment.
>> > > I would also like to highlight that all the changes are constrained to
>> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS.
>> > >
>> > > +1 on merging
>> > >
>> > > Inigo
>> > >
>> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka <[hidden email]>
>> > wrote:
>> > >
>> > > > Thanks Brahma for starting the discussion.
>> > > > I'm +1 for merging this.
>> > > >
>> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
>> > > > with 2 routers (not in production) and running several tests.
>> > > >
>> > > > Regards,
>> > > > Akira
>> > > >
>> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
>> > [hidden email]>
>> > > > wrote:
>> > > > >
>> > > > > Dear Hadoop Developers
>> > > > >
>> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk.
>> We
>> > > have
>> > > > > been working on this feature from last several months.
>> > > > > This feature work received the contributions from different
>> > companies.
>> > > > All
>> > > > > of the feature development happened smoothly and collaboratively
>> in
>> > > > JIRAs.
>> > > > >
>> > > > > Kindly do take a look at the branch and raise issues/concerns that
>> > need
>> > > > to
>> > > > > be addressed before the merge.
>> > > > >
>> > > > > *Highlights of HDFS-13891 Branch:*
>> > > > > =============================
>> > > > >
>> > > > > Adding Security to RBF(1)
>> > > > > Adding Missing Client API's(2)
>> > > > > Improvements/Bug Fixing
>> > > > >   Critical - HDFS-13637, HDFS-13834
>> > > > >
>> > > > > *Commits:*
>> > > > > ========
>> > > > >
>> > > > > No of JIRAs Resolved: 72
>> > > > >
>> > > > > All this commits are in RBF Module. No changes in hdfs/common.
>> > > > >
>> > > > > *Tested Cluster:*
>> > > > > =============
>> > > > >
>> > > > > Most of these changes verified at Uber,Microsoft,Huawei and some
>> > other
>> > > > > compaines.
>> > > > >
>> > > > > *Uber*: Most changes are running in production @Uber including the
>> > > > critical
>> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS
>> Routers.
>> > > > > Zookeeper as a state store to hold delegation tokens were also
>> stress
>> > > > > tested to hold more than 2 Million tokens. --CR Hota
>> > > > >
>> > > > > *MicroSoft*: Most of these changes are currently running in
>> > production
>> > > at
>> > > > > Microsoft.The security has also been tested in a 500 server
>> cluster
>> > > with
>> > > > 4
>> > > > > subclsuters. --Inigo Goiri
>> > > > >
>> > > > > *Huawei* : Deployed all this changes in 20 node cluster with 3
>> > > > > routers.Planning deploy 10K production cluster.
>> > > > >
>> > > > > *Contributors:*
>> > > > > ===========
>> > > > >
>> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu
>> Asanuma,Shubham
>> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
>> > > > > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He
>> > > Xiaoqiao,Sherwood
>> > > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving
>> > discussions
>> > > > and
>> > > > > contributing to this.
>> > > > >
>> > > > > *Future Tasks:*
>> > > > > ============
>> > > > >
>> > > > >  will cleanup the jira's under this umbrella and contiue to work.
>> > > > >
>> > > > > Reference:
>> > > > > 1) https://issues.apache.org/jira/browse/HDFS-13532
>> > > > > 2) https://issues.apache.org/jira/browse/HDFS-13655
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --Brahma Reddy Battula
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: [hidden email]
>> > > > For additional commands, e-mail: [hidden email]
>> > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> Regards,
>> Chitta
>>
>

--



--Brahma Reddy Battula
--



--Brahma Reddy Battula