Using Ozone as the default file system

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Using Ozone as the default file system

Pontus Amberg
I'm playing around with a small Hadoop 3.1.1 cluster and I have applied Ozone 0.3.0-alpha to all the nodes. The hadoop-ozone-datanode-plugin-0.3.0-alpha.jar and hadoop-ozone-filesystem-0.3.0-alpha.jar have been added to the classpath and I can start and init SCM, OM and create volumes, buckets and add new keys with values. But when I try to use Ozone as the default file system ( as described here  https://hadoop.apache.org/ozone/docs/0.3.0-alpha/ozonefs.html ) starting the name nodes fails with the following message:

2019-02-12 17:32:48,467 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): o3fs://internal.address:9864/volume/bucket is not of scheme 'hdfs'.

I have also tried Hadoop 3.2.0 and I encountered the same error message.

Anyone have a clue what the cause might be?

/Pontus
Reply | Threaded
Open this post in threaded view
|

Re: Using Ozone as the default file system

Bharat Viswanadham
Hi,
Could you give a try by setting it as below.

It should be set as o3fs://bucket.volume

<property>
  <name>fs.o3fs.impl</name>
  <value>org.apache.hadoop.fs.ozone.OzoneFileSystem</value>
</property>
<property>
  <name>fs.defaultFS</name>
  <value>o3fs://bucket.volume</value>
</property>

This bug is fixed as part of https://issues.apache.org/jira/browse/HDDS-913 which will be released as part of 0.4.0



Thanks,
Bharat

 
From: Pontus Amberg <[hidden email]>
Date: Tuesday, February 12, 2019 at 9:58 AM
To: "[hidden email]" <[hidden email]>
Subject: Using Ozone as the default file system
 
I'm playing around with a small Hadoop 3.1.1 cluster and I have applied Ozone 0.3.0-alpha to all the nodes. The hadoop-ozone-datanode-plugin-0.3.0-alpha.jar and hadoop-ozone-filesystem-0.3.0-alpha.jar have been added to the classpath and I can start and init SCM, OM and create volumes, buckets and add new keys with values. But when I try to use Ozone as the default file system ( as described here  https://hadoop.apache.org/ozone/docs/0.3.0-alpha/ozonefs.html ) starting the name nodes fails with the following message:
 
2019-02-12 17:32:48,467 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): o3fs://internal.address:9864/volume/bucket is not of scheme 'hdfs'.
 
I have also tried Hadoop 3.2.0 and I encountered the same error message.
 
Anyone have a clue what the cause might be?
 
/Pontus


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Using Ozone as the default file system

Pontus Amberg
I'm not sure but it looks as if when "dfs.namenode.rpc-address" is missing from hdfs-site.xml Hadoop will try deduce the address of the name node from "fs.defaultFS". That logic seems to take for granted that the "fs.defaultFS" must start with "hdfs".

Specifying "dfs.namenode.rpc-address" in hdfs-site.xml seems to have fixed my first problem. Using "o3fs://bucket.volume"
seems to work. Thanks for the help!

/Pontus

On Tue, Feb 12, 2019 at 9:04 PM Bharat Viswanadham <[hidden email]> wrote:
Hi,
Could you give a try by setting it as below.

It should be set as o3fs://bucket.volume

<property>
  <name>fs.o3fs.impl</name>
  <value>org.apache.hadoop.fs.ozone.OzoneFileSystem</value>
</property>
<property>
  <name>fs.defaultFS</name>
  <value>o3fs://bucket.volume</value>
</property>

This bug is fixed as part of https://issues.apache.org/jira/browse/HDDS-913 which will be released as part of 0.4.0



Thanks,
Bharat

 
From: Pontus Amberg <[hidden email]>
Date: Tuesday, February 12, 2019 at 9:58 AM
To: "[hidden email]" <[hidden email]>
Subject: Using Ozone as the default file system
 
I'm playing around with a small Hadoop 3.1.1 cluster and I have applied Ozone 0.3.0-alpha to all the nodes. The hadoop-ozone-datanode-plugin-0.3.0-alpha.jar and hadoop-ozone-filesystem-0.3.0-alpha.jar have been added to the classpath and I can start and init SCM, OM and create volumes, buckets and add new keys with values. But when I try to use Ozone as the default file system ( as described here  https://hadoop.apache.org/ozone/docs/0.3.0-alpha/ozonefs.html ) starting the name nodes fails with the following message:
 
2019-02-12 17:32:48,467 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): o3fs://internal.address:9864/volume/bucket is not of scheme 'hdfs'.
 
I have also tried Hadoop 3.2.0 and I encountered the same error message.
 
Anyone have a clue what the cause might be?
 
/Pontus