aggregated logs on HDFS garbled

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

aggregated logs on HDFS garbled

yao930

Hello

 

I set yarn.log-aggregation-enable to true, and got logs on HDFS, but when I use

“bin/hdfs dfs -cat” or download files from the HDFS website, some garble appeared, and it looks like this:

 

\D1\D3h\91\B5׶9\DFA@\92\BA\E1P \00VERSION\00\00\00\00APPLICATION_ACL$\00VIEW_APP\00root \00

MODIFY_APP\00root \00APPLICATION_OWNER\00root(\00&container_1525851594379_0001_01_000003\00stderr\000\00stdout\000\00\00\00\00

\00VERSION*(\00&container_1525851594379_0001_01_000003none̡̡data:BCFile.indexnone\CC\ED data:TFile.indexnone̷66data:TFile.metanone̱\00\00\00\00\00\00\00\F8\00\00\00\D1\D3h\91\B5׶9\DFA@\92\BA\E1P


What can I do to fix this?

 

Thanks in advance!

Reply | Threaded
Open this post in threaded view
|

Re: aggregated logs on HDFS garbled

Hariharan Iyer
Application logs are not saved in text format. They are saved in a
binary format called TFile. You can use the "yarn logs"[1] command to
view them. Alternatively you can use a tool like this tfile reader[2].

[1] - https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#logs
[2] - https://github.com/alexandruanghel/hadoop-snippets/tree/master/read-tfile

Thanks,
Hari


On Thu, May 10, 2018 at 6:06 AM,  <[hidden email]> wrote:

> Hello,
>
>
>
> I set “yarn.log-aggregation-enable” to true, and got logs on HDFS, but when
> I use
>
> “bin/hdfs dfs -cat” or download files from the HDFS website, some garble
> appeared, and it looks like this:
>
>
>
> \D1 \D3h\91\B5׶9\DFA@\92\BA\E1P
> \00 VERSION \00\00\00  \00 APPLICATION_ACL$\00 VIEW_APP\00 root \00
>
> MODIFY_APP\00 root
>  \00 APPLICATION_OWNER \00 root(\00&container_1525851594379_0001_01_000003 \00 stderr\00 0\00 stdout\00 0\00 \00\00 \00
>
> \00 VERSION*(\00&container_1525851594379_0001_01_000003  none  ̡̡  data:BCFile.index none\CC\ED   data:TFile.index none̷66 data:TFile.meta none̱  \00\00\00\00\00\00\00\F8\00 \00\00\D1 \D3h\91\B5׶9\DFA@\92\BA\E1P
>
>
> What can I do to fix this?
>
>
>
> Thanks in advance!

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Re: aggregated logs on HDFS garbled

yao930
Thanks for your advice!

I tried the tflie reader you provided in [2],but there's still some garble appear at the beginnig and the end, looks like this:
stdout0

and I found another tool that works fine:
https://github.com/shanyu/hadooplogparser 

在2018-05-10 11:13:38,Hariharan Iyer<[hidden email]>写道:

> Application logs are not saved in text format. They are saved in a
> binary format called TFile. You can use the "yarn logs"[1] command to
> view them. Alternatively you can use a tool like this tfile reader[2].
>
> [1] - https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#logs
> [2] - https://github.com/alexandruanghel/hadoop-snippets/tree/master/read-tfile
>
> Thanks,
> Hari
>
>
> On Thu, May 10, 2018 at 6:06 AM,  <[hidden email]> wrote:
> > Hello,
> >
> >
> >
> > I set “yarn.log-aggregation-enable” to true, and got logs on HDFS, but when
> > I use
> >
> > “bin/hdfs dfs -cat” or download files from the HDFS website, some garble
> > appeared, and it looks like this:
> >
> >
> >
> > \D1 \D3h\91\B5׶9\DFA@\92\BA\E1P
> > \00 VERSION \00\00\00  \00 APPLICATION_ACL$\00 VIEW_APP\00 root \00
> >
> > MODIFY_APP\00 root
> >  \00 APPLICATION_OWNER \00 root(\00&container_1525851594379_0001_01_000003 \00 stderr\00 0\00 stdout\00 0\00 \00\00 \00
> >
> > \00 VERSION*(\00&container_1525851594379_0001_01_000003  none  ̡̡  data:BCFile.index none\CC\ED   data:TFile.index none̷66 data:TFile.meta none̱  \00\00\00\00\00\00\00\F8\00 \00\00\D1 \D3h\91\B5׶9\DFA@\92\BA\E1P
> >
> >
> > What can I do to fix this?
> >
> >
> >
> > Thanks in advance!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Re: aggregated logs on HDFS garbled

Robert Kanter
As a shameless self-promotion :), you can try this program that I wrote which will convert the log files back to plain text files:


- Robert 

On Thu, May 10, 2018 at 12:44 AM, <[hidden email]> wrote:
Thanks for your advice!

I tried the tflie reader you provided in [2],but there's still some garble appear at the beginnig and the end, looks like this:
stdout0

and I found another tool that works fine:
https://github.com/shanyu/hadooplogparser 

在2018-05-10 11:13:38,Hariharan Iyer<[hidden email]>写道:
> Application logs are not saved in text format. They are saved in a
> binary format called TFile. You can use the "yarn logs"[1] command to
> view them. Alternatively you can use a tool like this tfile reader[2].
>
> [1] - https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#logs
> [2] - https://github.com/alexandruanghel/hadoop-snippets/tree/master/read-tfile
>
> Thanks,
> Hari
>
>
> On Thu, May 10, 2018 at 6:06 AM,  <[hidden email]> wrote:
> > Hello,
> >
> >
> >
> > I set “yarn.log-aggregation-enable” to true, and got logs on HDFS, but when
> > I use
> >
> > “bin/hdfs dfs -cat” or download files from the HDFS website, some garble
> > appeared, and it looks like this:
> >
> >
> >
> > \D1 \D3h\91\B5׶9\DFA@\92\BA\E1P
> > \00 VERSION \00\00\00  \00 APPLICATION_ACL$\00 VIEW_APP\00 root \00
> >
> > MODIFY_APP\00 root
> >  \00 APPLICATION_OWNER \00 root(\00&container_1525851594379_0001_01_000003 \00 stderr\00 0\00 stdout\00 0\00 \00\00 \00
> >
> > \00 VERSION*(\00&container_1525851594379_0001_01_000003  none  ̡̡  data:BCFile.index none\CC\ED   data:TFile.index none̷66 data:TFile.meta none̱  \00\00\00\00\00\00\00\F8\00 \00\00\D1 \D3h\91\B5׶9\DFA@\92\BA\E1P
> >
> >
> > What can I do to fix this?
> >
> >
> >
> > Thanks in advance!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>