[MapReduce] Join操作在mapreduce中的实现

Join 操作分为Map Join/Reduce Join

Reduce Join(存在数据倾斜的可能)

Map端主要工作:

为来自不同表或文件的k-v键值对，打标签以区别不同的来源，以连接字段作为key，其余部分加上标签作为value，然后输出.

Reduce端主要工作

在Reduce端以连接字段作为key的分组已经完成，只需要在每一个分组中，把来源自不同表/文件的记录通过标签分开，最后合并.

实例:
order.txt

id  pid amount
1001    1   1
1002    2   3

pd.txt

pid pname
1   华为
2   小米
3   格力

---

代码如下:

Mapper:

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;

import java.io.IOException;

public class TableMapper extends Mapper<LongWritable, Text, Text, TableBean> {
        // 文件名称
        private  String fName;
        private TableBean tableBean = new TableBean();
        private Text k = new Text();
        public TableMapper() {
                super();
        }

        @Override
        protected void setup(Context context) throws IOException, InterruptedException {
                // 判断数据来源
                FileSplit split = (FileSplit) context.getInputSplit();
                 fName = split.getPath().getName();
        }

        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
                String[] args = value.toString().split("\t");
                // 根据来源构造tableBean
                if(fName.contains("order")){
                        tableBean.setId(args[0]);
                        tableBean.setPid(args[1]);
                        tableBean.setAmount(Integer.parseInt(args[2]));
                        tableBean.setpName("");
                        tableBean.setFlag("order");
                        k.set(args[1]);
                }
                else {
                        tableBean.setId("");
                        tableBean.setPid(args[0]);
                        tableBean.setAmount(-1);
                        tableBean.setpName(args[1]);
                        tableBean.setFlag("pd");
                        k.set(args[0]);
                }
                context.write(k, tableBean);

        }
}

Reducer:

import com.sun.org.apache.bcel.internal.generic.TABLESWITCH;
import org.apache.commons.beanutils.BeanUtils;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import java.util.ArrayList;
import java.util.List;

public class TableReducer extends Reducer<Text, TableBean, TableBean, NullWritable> {
        @Override
        protected void reduce(Text key, Iterable<TableBean> values, Context context) throws IOException, InterruptedException {
                List<TableBean> orderBeans = new ArrayList<>();
                TableBean pdBean = new TableBean();
                for (TableBean value: values) {
                        if(value.getFlag().equals("order")){
                                TableBean tmp = new TableBean();

                                // 将value的属性 copy到tmp中
                                try {
                                        BeanUtils.copyProperties(tmp, value);
                                } catch (IllegalAccessException | InvocationTargetException e) {
                                        e.printStackTrace();
                                }

                                orderBeans.add(tmp);
                        }
                        else {
                                try {
                                        BeanUtils.copyProperties(pdBean, value);
                                } catch (IllegalAccessException | InvocationTargetException e) {
                                        e.printStackTrace();
                                }
                        }
                }


                for(TableBean bean : orderBeans){
                        bean.setpName(pdBean.getpName());
                        context.write(bean, NullWritable.get());
                }

        }
}

Main 方法:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;

public class TableDriver {

        public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
                String inputDir = args[0];
                String outputDir = args[1];

                Configuration conf = new Configuration();
                Job job = Job.getInstance(conf);
                job.setJarByClass(TableDriver.class);
                job.setMapperClass(TableMapper.class);
                job.setReducerClass(TableReducer.class);

                job.setMapOutputKeyClass(Text.class);
                job.setMapOutputValueClass(TableBean.class);


                job.setOutputKeyClass(TableBean.class);
                job.setOutputValueClass(NullWritable.class);

                FileInputFormat.setInputPaths(job, new Path(inputDir));
                FileOutputFormat.setOutputPath(job, new Path(outputDir));

                System.exit(job.waitForCompletion(true)?0:1);
        }
}

Map Join

适用场景: 一张表大，一张表小, 减少reduce端的压力，缓解数据倾斜

DistributedCacheDriver 缓存文件

1. Driver里设置加载缓存数据:job.addCacheFile(new Path("xxxx"));

2.Map Join 不需要Reduce阶段 设置reduce task number为0： job.setNumReduceTask(0);

思路 :
1） 在Map的setup中读取pd表,获取所有的pid->pname
2） map 时 根据order表的pid添加对应pname。