Re: HDFS Scalability Limit?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: HDFS Scalability Limit?

Kihwal Lee-4
Hi Wei-Chiu,

We have experience with 5,000 - 6,000 node clusters.  Although it ran/runs fine, any heavy hitter activities such as decommissioning needed to be carefully planned.   In terms of files and blocks, we have multiple clusters running stable with over 500M files and blocks.  Some at over 800M with the max heap at 256GB. It can probably go higher, but we haven't done performance testing & optimizations beyond 256GB yet.  All our clusters are un-federated. Funny how the feature was developed in Yahoo! and ended up not being used here. :)  We have a cluster with about 180PB of provisioned space. Many clusters are using over 100PB in their steady state.  We don't run datanodes too dense, so can't tell what the per-datanode limit is.

Thanks and 73
Kihwal

On Thu, Jun 13, 2019 at 1:57 PM Wei-Chiu Chuang <[hidden email]> wrote:
Hi community,

I am currently drafting a HDFS scalability guideline doc, and I'd like to
understand any data points regarding HDFS scalability limit. I'd like to
share it publicly eventually.

As an example, through my workplace, and through community chatters, I am
aware that HDFS is capable of operating at the following scale:

Number of DataNodes:
Unfederated: I can reasonably believe a single HDFS NameNode can manage up
to 4000 DataNodes. Is there any one who would like to share an even larger
cluster?

Federated: I am aware of one federated HDFS cluster composed of 20,000
DataNodes. JD.com
<https://conferences.oreilly.com/strata/strata-eu-2018/public/schedule/detail/64692>
has a 15K DN cluster and 210PB total capacity. I suspect it's a federated
HDFS cluster.

Number of blocks & files:
500 million files&blocks seems to be the upper limit at this point. At this
scale NameNode consumes around 200GB heap, and my experience told me any
number beyond 200GB is unstable. But at some point I recalled some one
mentioned a 400GB NN heap.

Amount of Data:
I am aware a few clusters more than 100PB in size (federated or not) --
Uber, Yahoo Japan, JD.com.

Number of Volumes in a DataNode:
DataNodes with 24 volumes is known to work reasonably well. If DataNode is
used for archival use cases, a DN can have up to 48 volumes. This is
certainly hardware dependent, but if I know where the current limit is, I
can start optimizing the software.

Total disk space:
CDH
<https://www.cloudera.com/documentation/enterprise/release-notes/topics/hardware_requirements_guide.html#concept_fzz_dq4_gbb>
recommends no more than 100TB per DataNode. Are there successful
deployments that install more than this number? Of course, you can easily
exceed this number if it is used purely for data archival.


What are other scalability limits that people are interested?

Best,
Wei-Chiu
Reply | Threaded
Open this post in threaded view
|

Re: HDFS Scalability Limit?

Konstantin Shvachko-2
Hi Wei-Chiu,

We run similar Hadoop installations as Kihwal describes. Thanks for sharing Kihwal.
Our clusters are also not Federated.
So far the growth rate is the same as we reported in our talk last year (slide #5) 2x per year:

We track three main metrics: Total space used, Number of Objects (Files+ blocks), and Number of Tasks per day.
I found that the number of nodes is mostly irrelevant in measuring cluster size, since the nodes are very diverse in configuration and are constantly upgrading, so you may have the same #nodes, but much more drives, cores, RAM on each of them - a bigger cluster.

I do not see 200 GB heap size as a limit. We ran Dynamometer experiments with a bigger heap fitting 1 billion files and blocks. It should be doable, but we may hit other scalability limits when we get to so many objects. See Erik's talk discussing the experiments and solutions:

Well Hadoop scalability has always been a moving target for me. I don't think you can set it in stone once and for all.

Thanks,
--Konstantin

On Sat, Jun 15, 2019 at 5:20 PM Wei-Chiu Chuang <[hidden email]> wrote:
Thank you, Kihwal for the insightful comments!

As I understand it, Yahoo's ops team has a good control of application
behavior. I tend to be conservative in terms of number of files&blocks and
heap size. We don't have such luxury, and our customers have a wide
spectrum of workloads and features (e.g., snapshots, data at-rest
encryption, Impala).

Yes -- decomm/recomm is a pain, and I am working with my colleague, @Stephen
O'Donnell <[hidden email]> , to address this problem. Have you
tried maintenance mode? It's in Hadoop 2.9 but a number of decomm/recomm
needs are alleviated by maintenance mode.

I know Twitter is a big user of maintenance mode, and I'm wondering if
Twitter folks can offer some experience with it at large scale. CDH
supports maintenance mode, but our users don't seem to be quite familiar
with it. Are there issues that were dealt with, but not reported in the
JIRA? Basically, I'd like to know the operational complexity of this
feature at large scale.

On Thu, Jun 13, 2019 at 4:00 PM Kihwal Lee <[hidden email].invalid>
wrote:

> Hi Wei-Chiu,
>
> We have experience with 5,000 - 6,000 node clusters.  Although it ran/runs
> fine, any heavy hitter activities such as decommissioning needed to be
> carefully planned.   In terms of files and blocks, we have multiple
> clusters running stable with over 500M files and blocks.  Some at over 800M
> with the max heap at 256GB. It can probably go higher, but we haven't done
> performance testing & optimizations beyond 256GB yet.  All our clusters are
> un-federated. Funny how the feature was developed in Yahoo! and ended up
> not being used here. :)  We have a cluster with about 180PB of provisioned
> space. Many clusters are using over 100PB in their steady state.  We don't
> run datanodes too dense, so can't tell what the per-datanode limit is.
>
> Thanks and 73
> Kihwal
>
> On Thu, Jun 13, 2019 at 1:57 PM Wei-Chiu Chuang <[hidden email]>
> wrote:
>
>> Hi community,
>>
>> I am currently drafting a HDFS scalability guideline doc, and I'd like to
>> understand any data points regarding HDFS scalability limit. I'd like to
>> share it publicly eventually.
>>
>> As an example, through my workplace, and through community chatters, I am
>> aware that HDFS is capable of operating at the following scale:
>>
>> Number of DataNodes:
>> Unfederated: I can reasonably believe a single HDFS NameNode can manage up
>> to 4000 DataNodes. Is there any one who would like to share an even larger
>> cluster?
>>
>> Federated: I am aware of one federated HDFS cluster composed of 20,000
>> DataNodes. JD.com
>> <
>> https://conferences.oreilly.com/strata/strata-eu-2018/public/schedule/detail/64692
>> >
>> has a 15K DN cluster and 210PB total capacity. I suspect it's a federated
>> HDFS cluster.
>>
>> Number of blocks & files:
>> 500 million files&blocks seems to be the upper limit at this point. At
>> this
>> scale NameNode consumes around 200GB heap, and my experience told me any
>> number beyond 200GB is unstable. But at some point I recalled some one
>> mentioned a 400GB NN heap.
>>
>> Amount of Data:
>> I am aware a few clusters more than 100PB in size (federated or not) --
>> Uber, Yahoo Japan, JD.com.
>>
>> Number of Volumes in a DataNode:
>> DataNodes with 24 volumes is known to work reasonably well. If DataNode is
>> used for archival use cases, a DN can have up to 48 volumes. This is
>> certainly hardware dependent, but if I know where the current limit is, I
>> can start optimizing the software.
>>
>> Total disk space:
>> CDH
>> <
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/hardware_requirements_guide.html#concept_fzz_dq4_gbb
>> >
>> recommends no more than 100TB per DataNode. Are there successful
>> deployments that install more than this number? Of course, you can easily
>> exceed this number if it is used purely for data archival.
>>
>>
>> What are other scalability limits that people are interested?
>>
>> Best,
>> Wei-Chiu
>>
>