从远程HDFS读取文件

我想从远程HDFS读取文件。我无法查看文件的内容。请帮助我。我在这里附上我的代码。运行此代码时，我没有得到任何输出。该计划保持活跃而没有给出任何结果。从远程HDFS读取文件

package com.cts.peg.iot.accessRemoteHDFS02; 
import java.io.BufferedInputStream; 
import java.io.BufferedOutputStream; 
import java.io.File; 
import java.io.FileInputStream; 
import java.io.FileOutputStream; 
import java.io.IOException; 
import java.io.InputStream; 
import java.io.OutputStream; 

import org.apache.hadoop.conf.Configuration; 
import org.apache.hadoop.fs.BlockLocation; 
import org.apache.hadoop.fs.FSDataInputStream; 
import org.apache.hadoop.fs.FSDataOutputStream; 
import org.apache.hadoop.fs.FileStatus; 
import org.apache.hadoop.fs.FileSystem; 
import org.apache.hadoop.fs.Path; 
import org.apache.hadoop.hdfs.DistributedFileSystem; 
import org.apache.hadoop.hdfs.protocol.DatanodeInfo; 
public class ReadFromHDFS { 

    public static void main(String[] args) throws Exception { 
     Configuration conf = new Configuration(); 
     conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()); 
     conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName()); 
     String dest = args[0]; 
     conf.addResource(new Path("/etc/hadoop/conf/core-site.xml")); 
     conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml")); 
     conf.addResource(new Path("/etc/hadoop/conf/mapred-site.xml")); 
     FileSystem fileSystem = FileSystem.get(conf); 
     Path dstPath = new Path(dest); 
     FSDataInputStream in = fileSystem.open(dstPath); 
     // Check if the file already exists 
     if (!(fileSystem.exists(dstPath))) { 
     System.out.println("No such destination " + dstPath); 
     return; 
     } 
     // Get the filename out of the file path 

     try{   
     String filename = dest.substring(dest.lastIndexOf('/') + 1, dest.length()); 
       OutputStream out = new BufferedOutputStream(new FileOutputStream(
       new File(filename))); 
       byte[] b = new byte[1024]; 
       int numBytes = 0; 
       while ((numBytes = in.read(b)) > 0) { 
       out.write(b, 0, numBytes); 
       } 

     }catch(Exception e){ 
     System.err.println("Exception caught! :" + e); 
     System.exit(1); 
     }finally{ 
      in.close(); 
     fileSystem.close(); 
     } 

    } 

}

来源

2016-10-11 Janaarthan

我在这里看不到远程目的地。尝试如下更新代码：

Configuration conf = new Configuration(); 
conf.set("fs.defaultFS", "hdfs://master:8020"); 
conf.set("mapreduce.framework.name", "yarn"); 
conf.set("yarn.resourcemanager.address", "master:8032"); 
FileSystem fs = FileSystem.get(conf);

希望这有助于。

来源

2016-12-02 23:34:27 SBA

从远程HDFS读取文件

回答

相关问题