kafka api使用

1.生产者api

引入依赖

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.11.0.0</version> //版本为0.11.0.0
</dependency>

生产者的配置项都在ProducerConfig类中说明，每一项配置都有对应的doc说明。

生产者使用api 带回调函数demo，还有阻塞方式运行，返回Future对象，通过future对象get()到返回的值。

public class CustomProducer {
    public static void main(String[] args) throws ExecutionException, InterruptedException {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.152.3:9092,192.168.152.2:9092,192.168.152.1:9092");//
        props.put(ProducerConfig.ACKS_CONFIG, "all"); //leader 确认机制 0 1 all
        props.put(ProducerConfig.RETRIES_CONFIG, 1);//重试次数
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);//生产者批发送大小
        props.put(ProducerConfig.LINGER_MS_CONFIG, 1);//生产者达不到批发送大小，最短等待时间
        props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 33554432);//RecordAccumulator 缓冲区大小
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer"); //key的序列化器
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer"); //value的序列化器
        Producer<String, String> producer = new KafkaProducer<>(props); 
        //ProducerConfig对象传参到KafkaProducer构造函数，生成producer对象
        for (int i = 0; i < 10; i++) {
            //producer.send(),消息封装成ProducerRecord对象
            //带回调发送消息，如果发送失败会自动重试
            producer.send(new ProducerRecord<String, String>("minerprofit", Integer.toString(i), Integer.toString(i)), (RecordMetadata metadata,Exception exception) -> {
                if (exception == null) {
                    System.out.println("success->" +
                            metadata.offset());
                } else {
                    exception.printStackTrace();
                }
            }
        });
    }
    producer.close(); //关闭消费者
}

2.消费者api

自动提交offset方式

public class CustomConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.152.3:9092,192.168.152.2:9092,192.168.152.1:9092");//
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "miner"); //消费者组
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true"); //开启自动提交
        props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000"); //自动提交最短时间
        //key反序列化类
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringDeserializer"); 
        //value反序列化类
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringDeserializer");
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Arrays.asList("profit")); //消费者组订阅的topic
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(100); //拉取数据
            for (ConsumerRecord<String, String> record : records){
                System.out.printf("offset = %d, key = %s, value= %s%n", record.offset(), record.key(), record.value());
            }                
        }
    }
}

手动提交offset方式:

手动提交有两种提交方式一种是同步提交，一种是异步提交。

public class CustomConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.152.3:9092,192.168.152.2:9092,192.168.152.1:9092");//
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "miner"); //消费者组
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); //开启自动提交
        //props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000"); //自动提交最短时间
        //key反序列化类
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringDeserializer"); 
        //value反序列化类
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringDeserializer");
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Arrays.asList("profit")); //消费者组订阅的topic
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(100); //拉取数据
            for (ConsumerRecord<String, String> record : records){
                System.out.printf("offset = %d, key = %s, value= %s%n", record.offset(), record.key(), record.value());
            }
            consumer.commitSync(); //同步提交offset,会阻塞当前线程的运行
        }
    }
}

异步提交

consumer.commitAsync(new OffsetCommitCallback() {
    @Override
    public void onComplete(Map<TopicPartition,OffsetAndMetadata> offsets, Exception exception) {
        if (exception != null) {
            System.err.println("Commit failed：" +
                    offsets);
        }
    }
});

3.如何保证消息中间件幂等性

什么是幂等性：

生产者生产的消息能够发送到消息中间件中，消息中间件不会重复接受也不会少接收；消费者进行消费消息，不会重复消费，也不会少消费。

kafka结合具体业务如何保证幂等性：

kafka 生产者确认acks使用all级别，生产者发送到kafka的消息只可能重复不可能丢失，保证at least once;消费者使用异步提交offset,在业务中将得到的消息首先入数据库，如果库中已经存在了相同的消息，那么如果得到了新的相同的消息，那么就可以剔除重复的消息。