Re: How can I deploy 100 blocks onto 10 datanodes with each node have 10 blocks?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: How can I deploy 100 blocks onto 10 datanodes with each node have 10 blocks?

Jason Venner
If you set your replication count to one and on each datanode, create 10
files, you will achieve the pattern you are trying for.

By default when a file is created on a machine hosting a datanode, that
datanode will receive 1 replica of the file, and will be responsible for
sending the file data to the next replica if any.


On Thu, Oct 15, 2009 at 1:39 PM, Huang Qian <[hidden email]> wrote:

> Hi everyone. I am working on a project with hadoop and now I come across
> some problem. How can I deploy 100 files, with each file have one block by
> setting the blocksize and controling the file size, on to 10 datanode, and
> make sure each datanode has 10 blocks. I know the file system can deploy
> the
> blocks automaticly, but I want to make sure for the assigns files, the
> files
> will be deployed well-proportioned. How can I make it by the hadoop tool or
> api?
>
> Huang Qian(黄骞)
> Institute of Remote Sensing and GIS,Peking University
> Phone: (86-10) 5276-3109
> Mobile: (86) 1590-126-8883
> Address:Rm.554,Building 1,ChangChunXinYuan,Peking
> Univ.,Beijing(100871),CHINA
>



--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Reply | Threaded
Open this post in threaded view
|

Re: How can I deploy 100 blocks onto 10 datanodes with each node have 10 blocks?

Huang Qian
How can I creat 10 files on each datanode? I think I can only create files
on filesystem,
Is there any method to assign a file to a datanode?

2009/10/19 Jason Venner <[hidden email]>

> If you set your replication count to one and on each datanode, create 10
> files, you will achieve the pattern you are trying for.
>
> By default when a file is created on a machine hosting a datanode, that
> datanode will receive 1 replica of the file, and will be responsible for
> sending the file data to the next replica if any.
>
>
> On Thu, Oct 15, 2009 at 1:39 PM, Huang Qian <[hidden email]> wrote:
>
> > Hi everyone. I am working on a project with hadoop and now I come across
> > some problem. How can I deploy 100 files, with each file have one block
> by
> > setting the blocksize and controling the file size, on to 10 datanode,
> and
> > make sure each datanode has 10 blocks. I know the file system can deploy
> > the
> > blocks automaticly, but I want to make sure for the assigns files, the
> > files
> > will be deployed well-proportioned. How can I make it by the hadoop tool
> or
> > api?
> >
> > Huang Qian(黄骞)
> > Institute of Remote Sensing and GIS,Peking University
> > Phone: (86-10) 5276-3109
> > Mobile: (86) 1590-126-8883
> > Address:Rm.554,Building 1,ChangChunXinYuan,Peking
> > Univ.,Beijing(100871),CHINA
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>
Reply | Threaded
Open this post in threaded view
|

Re: How can I deploy 100 blocks onto 10 datanodes with each node have 10 blocks?

Brian Bockelman

On Oct 19, 2009, at 11:13 PM, Huang Qian wrote:

> How can I creat 10 files on each datanode? I think I can only create  
> files
> on filesystem,
> Is there any method to assign a file to a datanode?
>

No, and you probably don't want to.  At such a small scale, I can't  
think of a benefit to having the cluster *exactly* balanced.  Hadoop  
is designed to scale large, and it's simply too inefficient to put  
such a mechanism in place.

If you want *approximately* the same number of blocks on each node,  
you can use the rebalancer.

Brian

PS - if you really think this is mission-critical (and if you do, I'd  
advise to re-check your assumptions), you may want to look at the  
custom block placement plugins in the upcoming 0.21.0 release.

> 2009/10/19 Jason Venner <[hidden email]>
>
>> If you set your replication count to one and on each datanode,  
>> create 10
>> files, you will achieve the pattern you are trying for.
>>
>> By default when a file is created on a machine hosting a datanode,  
>> that
>> datanode will receive 1 replica of the file, and will be  
>> responsible for
>> sending the file data to the next replica if any.
>>
>>
>> On Thu, Oct 15, 2009 at 1:39 PM, Huang Qian <[hidden email]>  
>> wrote:
>>
>>> Hi everyone. I am working on a project with hadoop and now I come  
>>> across
>>> some problem. How can I deploy 100 files, with each file have one  
>>> block
>> by
>>> setting the blocksize and controling the file size, on to 10  
>>> datanode,
>> and
>>> make sure each datanode has 10 blocks. I know the file system can  
>>> deploy
>>> the
>>> blocks automaticly, but I want to make sure for the assigns files,  
>>> the
>>> files
>>> will be deployed well-proportioned. How can I make it by the  
>>> hadoop tool
>> or
>>> api?
>>>
>>> Huang Qian(黄骞)
>>> Institute of Remote Sensing and GIS,Peking University
>>> Phone: (86-10) 5276-3109
>>> Mobile: (86) 1590-126-8883
>>> Address:Rm.554,Building 1,ChangChunXinYuan,Peking
>>> Univ.,Beijing(100871),CHINA
>>>
>>
>>
>>
>> --
>> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
>> http://www.amazon.com/dp/1430219424?tag=jewlerymall
>> www.prohadoopbook.com a community for Hadoop Professionals
>>


smime.p7s (3K) Download Attachment