No Space left on device

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

No Space left on device

Nuthalapati, Ramesh
Strangely isee the tmp folder has enough space. What else could be the problem ? How much should my tmp space be ?


Error: java.io.IOException: No space left on device
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:260)
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:190)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:84)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:218)
        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2454)

java.io.IOException: Task: attempt_201204240741_0003_r_000000_1 - The reduce copier failed
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for file:/opt/hadoop/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201204240741_0003/attempt_201204240741_0003_r_000000_1/output/map_122.out
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2434)


Reply | Threaded
Open this post in threaded view
|

Re: No Space left on device

alo alt
looks like the hadoop partition is full.

sent via my mobile device

On Apr 25, 2012, at 9:13 PM, "Nuthalapati, Ramesh" <[hidden email]> wrote:

> Strangely isee the tmp folder has enough space. What else could be the problem ? How much should my tmp space be ?
>
>
> Error: java.io.IOException: No space left on device
>        at java.io.FileOutputStream.writeBytes(Native Method)
>        at java.io.FileOutputStream.write(FileOutputStream.java:260)
>        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:190)
>        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
>        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>        at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:84)
>        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>        at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:218)
>        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2454)
>
> java.io.IOException: Task: attempt_201204240741_0003_r_000000_1 - The reduce copier failed
>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for file:/opt/hadoop/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201204240741_0003/attempt_201204240741_0003_r_000000_1/output/map_122.out
>        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
>        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2434)
>
>
Reply | Threaded
Open this post in threaded view
|

Re: No Space left on device

Harsh J-2
In reply to this post by Nuthalapati, Ramesh
This is from your mapred.local.dir (which by default may reuse hadoop.tmp.dir).

Do you see free space available when you do the following?:
df -h /opt/hadoop

On Thu, Apr 26, 2012 at 12:43 AM, Nuthalapati, Ramesh
<[hidden email]> wrote:

> Strangely isee the tmp folder has enough space. What else could be the problem ? How much should my tmp space be ?
>
>
> Error: java.io.IOException: No space left on device
>        at java.io.FileOutputStream.writeBytes(Native Method)
>        at java.io.FileOutputStream.write(FileOutputStream.java:260)
>        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:190)
>        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
>        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>        at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:84)
>        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>        at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:218)
>        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2454)
>
> java.io.IOException: Task: attempt_201204240741_0003_r_000000_1 - The reduce copier failed
>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for file:/opt/hadoop/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201204240741_0003/attempt_201204240741_0003_r_000000_1/output/map_122.out
>        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
>        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2434)
>
>



--
Harsh J
Reply | Threaded
Open this post in threaded view
|

RE: No Space left on device

Nuthalapati, Ramesh
I have lot of space available

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/sysvg-opt
                       14G  1.2G   12G   9% /opt

My input files are around 10G, is there a requirement that the hadoop tmp dir should be at certain % of the input files or something ?

Thanks !

-----Original Message-----
From: Harsh J [mailto:[hidden email]]
Sent: Wednesday, April 25, 2012 3:19 PM
To: [hidden email]
Subject: Re: No Space left on device

This is from your mapred.local.dir (which by default may reuse hadoop.tmp.dir).

Do you see free space available when you do the following?:
df -h /opt/hadoop

On Thu, Apr 26, 2012 at 12:43 AM, Nuthalapati, Ramesh <[hidden email]> wrote:

> Strangely isee the tmp folder has enough space. What else could be the problem ? How much should my tmp space be ?
>
>
> Error: java.io.IOException: No space left on device
>        at java.io.FileOutputStream.writeBytes(Native Method)
>        at java.io.FileOutputStream.write(FileOutputStream.java:260)
>        at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(
> RawLocalFileSystem.java:190)
>        at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>        at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
>        at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutp
> utStream.java:49)
>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>        at
> org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.jav
> a:84)
>        at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutp
> utStream.java:49)
>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>        at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:218)
>        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
>        at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Red
> uceTask.java:2454)
>
> java.io.IOException: Task: attempt_201204240741_0003_r_000000_1 - The
> reduce copier failed
>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException:
> Could not find any valid local directory for
> file:/opt/hadoop/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/j
> ob_201204240741_0003/attempt_201204240741_0003_r_000000_1/output/map_1
> 22.out
>        at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPat
> hForWrite(LocalDirAllocator.java:343)
>        at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAl
> locator.java:124)
>        at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Red
> uceTask.java:2434)
>
>



--
Harsh J
Reply | Threaded
Open this post in threaded view
|

Re: No Space left on device

Harsh J-2
Ramesh,

That explains it then.

Going from Map to Reduce requires disk storage worth at least the
amount of data you're gonna be sending between them. If you're running
your 'cluster' on a single machine, the answer to your question is
yes.

On Thu, Apr 26, 2012 at 1:01 AM, Nuthalapati, Ramesh
<[hidden email]> wrote:

> I have lot of space available
>
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/sysvg-opt
>                       14G  1.2G   12G   9% /opt
>
> My input files are around 10G, is there a requirement that the hadoop tmp dir should be at certain % of the input files or something ?
>
> Thanks !
>
> -----Original Message-----
> From: Harsh J [mailto:[hidden email]]
> Sent: Wednesday, April 25, 2012 3:19 PM
> To: [hidden email]
> Subject: Re: No Space left on device
>
> This is from your mapred.local.dir (which by default may reuse hadoop.tmp.dir).
>
> Do you see free space available when you do the following?:
> df -h /opt/hadoop
>
> On Thu, Apr 26, 2012 at 12:43 AM, Nuthalapati, Ramesh <[hidden email]> wrote:
>> Strangely isee the tmp folder has enough space. What else could be the problem ? How much should my tmp space be ?
>>
>>
>> Error: java.io.IOException: No space left on device
>>        at java.io.FileOutputStream.writeBytes(Native Method)
>>        at java.io.FileOutputStream.write(FileOutputStream.java:260)
>>        at
>> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(
>> RawLocalFileSystem.java:190)
>>        at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>>        at
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
>>        at
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutp
>> utStream.java:49)
>>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>        at
>> org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.jav
>> a:84)
>>        at
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutp
>> utStream.java:49)
>>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>        at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:218)
>>        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
>>        at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Red
>> uceTask.java:2454)
>>
>> java.io.IOException: Task: attempt_201204240741_0003_r_000000_1 - The
>> reduce copier failed
>>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
>>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException:
>> Could not find any valid local directory for
>> file:/opt/hadoop/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/j
>> ob_201204240741_0003/attempt_201204240741_0003_r_000000_1/output/map_1
>> 22.out
>>        at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPat
>> hForWrite(LocalDirAllocator.java:343)
>>        at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAl
>> locator.java:124)
>>        at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Red
>> uceTask.java:2434)
>>
>>
>
>
>
> --
> Harsh J



--
Harsh J
Reply | Threaded
Open this post in threaded view
|

RE: No Space left on device

Nuthalapati, Ramesh
Harsh -

Even if it's the case, my free tmp memory is more than the dfs used - Isn't it ?

Configured Capacity: 116258406400 (108.27 GB)
Present Capacity: 110155911168 (102.59 GB)
DFS Remaining: 101976682496 (94.97 GB)
DFS Used: 8179228672 (7.62 GB)
DFS Used%: 7.43%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: 172.17.7.83:50010
Decommission Status : Normal
Configured Capacity: 116258406400 (108.27 GB)
DFS Used: 8179228672 (7.62 GB)
Non DFS Used: 6102495232 (5.68 GB)
DFS Remaining: 101976682496(94.97 GB)
DFS Used%: 7.04%
DFS Remaining%: 87.72%
Last contact: Wed Apr 25 12:52:19 PDT 2012

Thanks !

-----Original Message-----
From: Harsh J [mailto:[hidden email]]
Sent: Wednesday, April 25, 2012 3:42 PM
To: [hidden email]
Subject: Re: No Space left on device

Ramesh,

That explains it then.

Going from Map to Reduce requires disk storage worth at least the amount of data you're gonna be sending between them. If you're running your 'cluster' on a single machine, the answer to your question is yes.

On Thu, Apr 26, 2012 at 1:01 AM, Nuthalapati, Ramesh <[hidden email]> wrote:

> I have lot of space available
>
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/sysvg-opt
>                       14G  1.2G   12G   9% /opt
>
> My input files are around 10G, is there a requirement that the hadoop tmp dir should be at certain % of the input files or something ?
>
> Thanks !
>
> -----Original Message-----
> From: Harsh J [mailto:[hidden email]]
> Sent: Wednesday, April 25, 2012 3:19 PM
> To: [hidden email]
> Subject: Re: No Space left on device
>
> This is from your mapred.local.dir (which by default may reuse hadoop.tmp.dir).
>
> Do you see free space available when you do the following?:
> df -h /opt/hadoop
>
> On Thu, Apr 26, 2012 at 12:43 AM, Nuthalapati, Ramesh <[hidden email]> wrote:
>> Strangely isee the tmp folder has enough space. What else could be the problem ? How much should my tmp space be ?
>>
>>
>> Error: java.io.IOException: No space left on device
>>        at java.io.FileOutputStream.writeBytes(Native Method)
>>        at java.io.FileOutputStream.write(FileOutputStream.java:260)
>>        at
>> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write
>> (
>> RawLocalFileSystem.java:190)
>>        at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65
>> )
>>        at
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
>>        at
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
>> p
>> utStream.java:49)
>>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>        at
>> org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.ja
>> v
>> a:84)
>>        at
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
>> p
>> utStream.java:49)
>>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>        at
>> org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:218)
>>        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
>>        at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Re
>> d
>> uceTask.java:2454)
>>
>> java.io.IOException: Task: attempt_201204240741_0003_r_000000_1 - The
>> reduce copier failed
>>        at
>> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
>>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException:
>> Could not find any valid local directory for
>> file:/opt/hadoop/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/
>> j
>> ob_201204240741_0003/attempt_201204240741_0003_r_000000_1/output/map_
>> 1
>> 22.out
>>        at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPa
>> t
>> hForWrite(LocalDirAllocator.java:343)
>>        at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirA
>> l
>> locator.java:124)
>>        at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Re
>> d
>> uceTask.java:2434)
>>
>>
>
>
>
> --
> Harsh J



--
Harsh J
Reply | Threaded
Open this post in threaded view
|

Re: No Space left on device

Harsh J-2
The transient Map->Reduce files do not go to the DFS, but rather onto
the local filesystem directories specified by the "mapred.local.dir"
parameter. If you expand this configuration to be similar to
"dfs.data.dir" (as your DataNode may be carrying), then it will get
more space/disks to do its work.

See this very recent conversation for more information:
http://search-hadoop.com/m/DWbsZ1m0Ttx

On Thu, Apr 26, 2012 at 1:24 AM, Nuthalapati, Ramesh
<[hidden email]> wrote:

> Harsh -
>
> Even if it's the case, my free tmp memory is more than the dfs used - Isn't it ?
>
> Configured Capacity: 116258406400 (108.27 GB)
> Present Capacity: 110155911168 (102.59 GB)
> DFS Remaining: 101976682496 (94.97 GB)
> DFS Used: 8179228672 (7.62 GB)
> DFS Used%: 7.43%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 1 (1 total, 0 dead)
>
> Name: 172.17.7.83:50010
> Decommission Status : Normal
> Configured Capacity: 116258406400 (108.27 GB)
> DFS Used: 8179228672 (7.62 GB)
> Non DFS Used: 6102495232 (5.68 GB)
> DFS Remaining: 101976682496(94.97 GB)
> DFS Used%: 7.04%
> DFS Remaining%: 87.72%
> Last contact: Wed Apr 25 12:52:19 PDT 2012
>
> Thanks !
>
> -----Original Message-----
> From: Harsh J [mailto:[hidden email]]
> Sent: Wednesday, April 25, 2012 3:42 PM
> To: [hidden email]
> Subject: Re: No Space left on device
>
> Ramesh,
>
> That explains it then.
>
> Going from Map to Reduce requires disk storage worth at least the amount of data you're gonna be sending between them. If you're running your 'cluster' on a single machine, the answer to your question is yes.
>
> On Thu, Apr 26, 2012 at 1:01 AM, Nuthalapati, Ramesh <[hidden email]> wrote:
>> I have lot of space available
>>
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/mapper/sysvg-opt
>>                       14G  1.2G   12G   9% /opt
>>
>> My input files are around 10G, is there a requirement that the hadoop tmp dir should be at certain % of the input files or something ?
>>
>> Thanks !
>>
>> -----Original Message-----
>> From: Harsh J [mailto:[hidden email]]
>> Sent: Wednesday, April 25, 2012 3:19 PM
>> To: [hidden email]
>> Subject: Re: No Space left on device
>>
>> This is from your mapred.local.dir (which by default may reuse hadoop.tmp.dir).
>>
>> Do you see free space available when you do the following?:
>> df -h /opt/hadoop
>>
>> On Thu, Apr 26, 2012 at 12:43 AM, Nuthalapati, Ramesh <[hidden email]> wrote:
>>> Strangely isee the tmp folder has enough space. What else could be the problem ? How much should my tmp space be ?
>>>
>>>
>>> Error: java.io.IOException: No space left on device
>>>        at java.io.FileOutputStream.writeBytes(Native Method)
>>>        at java.io.FileOutputStream.write(FileOutputStream.java:260)
>>>        at
>>> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write
>>> (
>>> RawLocalFileSystem.java:190)
>>>        at
>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65
>>> )
>>>        at
>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
>>>        at
>>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
>>> p
>>> utStream.java:49)
>>>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>        at
>>> org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.ja
>>> v
>>> a:84)
>>>        at
>>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
>>> p
>>> utStream.java:49)
>>>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>        at
>>> org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:218)
>>>        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
>>>        at
>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Re
>>> d
>>> uceTask.java:2454)
>>>
>>> java.io.IOException: Task: attempt_201204240741_0003_r_000000_1 - The
>>> reduce copier failed
>>>        at
>>> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
>>>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException:
>>> Could not find any valid local directory for
>>> file:/opt/hadoop/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/
>>> j
>>> ob_201204240741_0003/attempt_201204240741_0003_r_000000_1/output/map_
>>> 1
>>> 22.out
>>>        at
>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPa
>>> t
>>> hForWrite(LocalDirAllocator.java:343)
>>>        at
>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirA
>>> l
>>> locator.java:124)
>>>        at
>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Re
>>> d
>>> uceTask.java:2434)
>>>
>>>
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Harsh J



--
Harsh J
Reply | Threaded
Open this post in threaded view
|

Re: No Space left on device

Sidney Simmons
Might be worth checking your inode usage as lack of inodes will result in the same error

HTH




On 26 Apr 2012, at 07:01, Harsh J <[hidden email]> wrote:

> The transient Map->Reduce files do not go to the DFS, but rather onto
> the local filesystem directories specified by the "mapred.local.dir"
> parameter. If you expand this configuration to be similar to
> "dfs.data.dir" (as your DataNode may be carrying), then it will get
> more space/disks to do its work.
>
> See this very recent conversation for more information:
> http://search-hadoop.com/m/DWbsZ1m0Ttx
>
> On Thu, Apr 26, 2012 at 1:24 AM, Nuthalapati, Ramesh
> <[hidden email]> wrote:
>> Harsh -
>>
>> Even if it's the case, my free tmp memory is more than the dfs used - Isn't it ?
>>
>> Configured Capacity: 116258406400 (108.27 GB)
>> Present Capacity: 110155911168 (102.59 GB)
>> DFS Remaining: 101976682496 (94.97 GB)
>> DFS Used: 8179228672 (7.62 GB)
>> DFS Used%: 7.43%
>> Under replicated blocks: 0
>> Blocks with corrupt replicas: 0
>> Missing blocks: 0
>>
>> -------------------------------------------------
>> Datanodes available: 1 (1 total, 0 dead)
>>
>> Name: 172.17.7.83:50010
>> Decommission Status : Normal
>> Configured Capacity: 116258406400 (108.27 GB)
>> DFS Used: 8179228672 (7.62 GB)
>> Non DFS Used: 6102495232 (5.68 GB)
>> DFS Remaining: 101976682496(94.97 GB)
>> DFS Used%: 7.04%
>> DFS Remaining%: 87.72%
>> Last contact: Wed Apr 25 12:52:19 PDT 2012
>>
>> Thanks !
>>
>> -----Original Message-----
>> From: Harsh J [mailto:[hidden email]]
>> Sent: Wednesday, April 25, 2012 3:42 PM
>> To: [hidden email]
>> Subject: Re: No Space left on device
>>
>> Ramesh,
>>
>> That explains it then.
>>
>> Going from Map to Reduce requires disk storage worth at least the amount of data you're gonna be sending between them. If you're running your 'cluster' on a single machine, the answer to your question is yes.
>>
>> On Thu, Apr 26, 2012 at 1:01 AM, Nuthalapati, Ramesh <[hidden email]> wrote:
>>> I have lot of space available
>>>
>>> Filesystem            Size  Used Avail Use% Mounted on
>>> /dev/mapper/sysvg-opt
>>>                       14G  1.2G   12G   9% /opt
>>>
>>> My input files are around 10G, is there a requirement that the hadoop tmp dir should be at certain % of the input files or something ?
>>>
>>> Thanks !
>>>
>>> -----Original Message-----
>>> From: Harsh J [mailto:[hidden email]]
>>> Sent: Wednesday, April 25, 2012 3:19 PM
>>> To: [hidden email]
>>> Subject: Re: No Space left on device
>>>
>>> This is from your mapred.local.dir (which by default may reuse hadoop.tmp.dir).
>>>
>>> Do you see free space available when you do the following?:
>>> df -h /opt/hadoop
>>>
>>> On Thu, Apr 26, 2012 at 12:43 AM, Nuthalapati, Ramesh <[hidden email]> wrote:
>>>> Strangely isee the tmp folder has enough space. What else could be the problem ? How much should my tmp space be ?
>>>>
>>>>
>>>> Error: java.io.IOException: No space left on device
>>>>        at java.io.FileOutputStream.writeBytes(Native Method)
>>>>        at java.io.FileOutputStream.write(FileOutputStream.java:260)
>>>>        at
>>>> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write
>>>> (
>>>> RawLocalFileSystem.java:190)
>>>>        at
>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65
>>>> )
>>>>        at
>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
>>>>        at
>>>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
>>>> p
>>>> utStream.java:49)
>>>>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>>        at
>>>> org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.ja
>>>> v
>>>> a:84)
>>>>        at
>>>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOut
>>>> p
>>>> utStream.java:49)
>>>>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>>        at
>>>> org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:218)
>>>>        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
>>>>        at
>>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Re
>>>> d
>>>> uceTask.java:2454)
>>>>
>>>> java.io.IOException: Task: attempt_201204240741_0003_r_000000_1 - The
>>>> reduce copier failed
>>>>        at
>>>> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
>>>>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException:
>>>> Could not find any valid local directory for
>>>> file:/opt/hadoop/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/
>>>> j
>>>> ob_201204240741_0003/attempt_201204240741_0003_r_000000_1/output/map_
>>>> 1
>>>> 22.out
>>>>        at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPa
>>>> t
>>>> hForWrite(LocalDirAllocator.java:343)
>>>>        at
>>>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirA
>>>> l
>>>> locator.java:124)
>>>>        at
>>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Re
>>>> d
>>>> uceTask.java:2434)
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Harsh J
Reply | Threaded
Open this post in threaded view
|

Re: No Space left on device

JunYong Li
In reply to this post by Nuthalapati, Ramesh
maybe exists file hole, are -sh and du -sch /tmp results same?
Reply | Threaded
Open this post in threaded view
|

Re: No Space left on device

Chris Curtin
Look at how many job_ directories there are on your slave nodes. We're
using Cloudera so they are under the 'userlogs' directory, not sure on
'pure' Apache where they are.

As we approach 30k we see this.(we run a monthly report does 10s of
thousands of jobs in a few days) We've tried tuning the # of jobs stored in
the history on the jobtracker but it doesn't always help. So we have an
hourly cron job that finds any files older than 4 hours in that directory
and removes them. None of our individual jobs runs for more than 30
minutes, so waiting 4 hours and blowing them away hasn't caused us any
problems.



On Thu, Apr 26, 2012 at 5:17 AM, JunYong Li <[hidden email]> wrote:

> maybe exists file hole, are -sh and du -sch /tmp results same?
>