Java执行hadoop的基本操作实例代码

Java执行hadoop的基本操作实例代码

在观山湖等地区，都构建了全面的区域性战略布局，加强发展的系统性、市场前瞻性、产品创新能力，以专注、极致的服务理念，为客户提供做网站、成都网站建设网站设计制作按需策划,公司网站建设,企业网站建设,品牌网站设计,营销型网站,外贸星空体育app最新版本(2024已更新)建设,观山湖网站建设费用合理。

向HDFS上传本地文件

public static void uploadInputFile(String localFile) throws IOException{
    Configuration conf = new Configuration();
    String hdfsPath = "hdfs://localhost:9000/";
    String hdfsInput = "hdfs://localhost:9000/user/hadoop/input";
    FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
    fs.copyFromLocalFile(new Path(localFile), new Path(hdfsInput));
    fs.close();
    System.out.println("已经上传文件到input文件夹啦");
  }

将output文件下载到本地

public static void getOutput(String outputfile) throws IOException{
    String remoteFile = "hdfs://localhost:9000/user/hadoop/output/part-r-00000";
    Path path = new Path(remoteFile);
    Configuration conf = new Configuration();
    String hdfsPath = "hdfs://localhost:9000/";
    FileSystem fs = FileSystem.get(URI.create(hdfsPath),conf);
    fs.copyToLocalFile(path, new Path(outputfile));
    System.out.println("已经将输出文件保留到本地文件");
    fs.close();
  }

删除hdfs中的文件

 public static void deleteOutput() throws IOException{
    Configuration conf = new Configuration();
    String hdfsOutput = "hdfs://localhost:9000/user/hadoop/output";
    String hdfsPath = "hdfs://localhost:9000/";
    Path path = new Path(hdfsOutput);
    FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
    fs.deleteOnExit(path);
    fs.close();
    System.out.println("output文件已经删除");
  }

执行mapReduce程序

创建Mapper类和Reducer类

public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context) throws IOException, InterruptedException{
      String line = value.toString();
      line = line.replace("\\", "");
      String regex = "性别：</span><span class=\"pt_detail\">(.*?)</span>";
      Pattern pattern = Pattern.compile(regex);
      Matcher matcher = pattern.matcher(line);
      while(matcher.find()){
        String term = matcher.group(1);
        word.set(term);
        context.write(word, one);
      }
    }
  }

  public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable>{

    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException{
      int sum = 0;
      for(IntWritable val :values){
        sum+= val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

执行mapReduce程序

public static void runMapReduce(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
    if(otherArgs.length != 2){
      System.err.println("Usage: wordcount<in> <out>");
      System.exit(2);
    }
    Job job = new Job(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
    FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
    System.out.println("mapReduce 执行完毕！");
    System.exit(job.waitForCompletion(true)?0:1);

  }

感谢阅读，希望能帮助到大家，谢谢大家对本站的支持！

名称栏目：Java执行hadoop的基本操作实例代码
本文URL：/article16/jhidgg.html

成都网站建设公司_创新互联，为您提供网站设计、、企业建站、软件开发、App开发、关键词优化

声明：本网站发布的内容（图片、视频和文字）以用户投稿、用户转载内容为主，如果涉及侵权请尽快告知，我们将会在第一时间删除。文章观点不代表本网站立场，如需处理请联系客服。电话：028-86922220；邮箱：631063699@qq.com。内容未经允许不得转载，或转载时需注明来源：创新互联

猜你还喜欢下面的内容