Kafka Series 4 - Java Project

Create Producer in Java.
Before running the Java project, make sure
1.Zookeeper is running
2.Kafka is running
3.Topic is created
4.Consumer is running so you can see the output

Problem1:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.ht... for further details.
Solution:
In the dependencies under build.gradle, I have

// https://mvnrepository.com/artifact/org.slf4j/slf4j-simple
testImplementation 'org.slf4j:slf4j-simple:1.7.36'

Should remove the 'test' and make it

// https://mvnrepository.com/artifact/org.slf4j/slf4j-simple
implementation 'org.slf4j:slf4j-simple:1.7.36'

Refresh the project, run again

Problem2:

org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-my-second-application-1, groupId=my-second-application] Node -1 disconnected.
Solution:
Typo in server.
Also could be Zookeeper and Kafka not running.

Problem3:

Created Consumer in Java, run, not retriving the historical messages.
Solution:
It's because offsets for your group id got committed.
Change the group id. (I updated from String groupId = "my-second-application"; to String groupId = "my-second-application-2";)

Partition Strategies
RangeAssignor
RoundRobin
StickyAssignor
CorperativeStickyAssignor
The default assignor is [RangeAssignor,CorperativeStickyAssignor]

Problem4:

Incompatible types. Found: 'io.conduktor.demos.kafka.wikimedia.WikimediaChangeHandler', required: 'java.beans.EventHandler'
Solution:

there are 2 types, choose com.launchdarkly.eventsource

Problem5:

[okhttp-eventsource-events-[]-0] ERROR WikimediaChangeHandler - Error in Stream Reading
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Solution5.1 (failed):
Step1: Check java properties and keystore
run:

    System.out.println("java.home="+System.getProperty("java.home"));
    System.out.println("java.runtime.version="+System.getProperty("java.runtime.version"));
    System.out.println("javax.net.ssl.trustStore="+System.getProperty("javax.net.ssl.trustStore"));
    System.out.println("javax.net.ssl.keyStore="+System.getProperty("javax.net.ssl.keyStore"));

result:

The truststore and keystore return null, indicating you do not have keystore for wikimedia.
However, this problem only appears on my Windows not Mac. I run the same on Mac, truststore and keystore also return null, but the program runs well. So this may not be the root cause?
Step2: Determine the Root CA cert
Use the openssl s_client -connect flag to display diagnostic information about the SSL connection to the server. The information will include the servers certificate chain, printed as subject and issuer. The end entity server certificate will be the only certificate printed in PEM format.

Command not working, run it in Git Bash

Step3:
download the root certificate pem file from
https://letsencrypt.org/certi...
ISRG Root X1 - Self-signed - pem
save it to a location
find cacert under C:\Program Files\Java\jdk-18.0.1.1\lib\security, copy it to a location
Step4:
run below in command
keytool -import -alias wikipediaroot -file "C:\Users\EC137TG\OneDrive - EY\Desktop\Kafka\certification\isrgrootx1.pem" -keystore "C:\Users\EC137TG\OneDrive - EY\Desktop\Kafka\certification\cacerts"

Step5:
add this
System.setProperty("javax.net.ssl.trustStore", "C:/Users/EC137TG/OneDrive - EY/Desktop/Kafka/certification/cacerts");
to the java program and run, but still not working.

Solution5.2 (Succeeded):
https://stackoverflow.com/que...

Approach5.2.1 (Failed):
Step1:
Go to the website, export the certificate https://stream.wikimedia.org/...

click Connection is secure

click the certification logo on the upper right corner

click Copy to File, will open Certificate Export Wizard, follow the instruction to export and save the cert file to a location.
Step2:
run the below query in Command

keytool -import -alias wikipediaroot -file "C:\Users\EC137TG\OneDrive - EY\Desktop\Kafka\certification\wikipedia.cer" -keystore "C:\Program Files\Java\jdk-18.0.1.1\lib\security\cacerts"

password is "changeit"
return "Certificate was added to keystore"
Step3:
restart computer, run the java program again, still not working
Step4:
notice that in our previou test, System.getProperty("java.home") gives "C:\Users\EC137TG.jdks\corretto-16.0.2" not "C:\Program Files\Java\jdk-18.0.1.1"
try this command

keytool -import -alias wikipediaroot -file "C:\Users\EC137TG\OneDrive - EY\Desktop\Kafka\certification\wikipedia.cer" -keystore "C:\Users\EC137TG.jdks\corretto-16.0.2\lib\security\cacerts"

restart computer, run the java program again, still not working
Step5:
Go back to the website certification page, notice that there are parent certifications. Try export the Zscaler Root CA instead of *.wikipedia.org and follow the same steps. Succeeded. See Approach5.2.2.

Approach5.2.2 (Succeeded):
Step1:
Go to the website, export the certificate https://stream.wikimedia.org/...

click Connection is secure

click the certification logo on the upper right corner

Select Zscaler Root CA, click View Certificate.

In the new window, click Copy to File, will open Certificate Export Wizard, follow the instruction to export and save the cert file to a location. I save it as "C:\Users\EC137TG\OneDrive - EY\Desktop\Kafka\certification\zscaler.cer"
Step2:
Run this in a Java program to get the home location

System.out.println(System.getProperty("java.home"))

My output is "C:\Users\EC137TG.jdks\corretto-16.0.2"
Step3:
Go to "C:\Users\EC137TG.jdks\corretto-16.0.2" and check the cacerts location. Usually it's under \lib\security.
Found it as "C:\Users\EC137TG.jdks\corretto-16.0.2\lib\security\cacerts".
Step4:
Run the below query in Command:

keytool -import -alias zscalerroot -file "C:\Users\EC137TG\OneDrive - EY\Desktop\Kafka\certification\zscaler.cer" -keystore "C:\Users\EC137TG.jdks\corretto-16.0.2\lib\security\cacerts"

Restart computer, run the Java program again. Succeed.

Problem6:

try(openSearchClient; consumer) {
    ...
    }
    while (true){
        ...
    }

giving error "The consumer has alreay been closed".
bracket is misplaced.
put while loop in try block, it works.

try(openSearchClient; consumer) {
    ...
    while (true){
        ...
    }
}

Problem7:

"cd .." means one level up

"cd ~" means back to the starting level /home/emmajin
use pwd to check current directory

?

Access Windows File from Ubuntu: cd ../../mnt
http://www.linfo.org/mnt.html...,%2Fetc%2C%20%2Fhome%2C%20%2Fproc%2C%20%2Froot%2C%20%2Fsbin%2C%20%2Fusr%20and%20%2Fvar.

cd "../../mnt/c/Users/EC137TG/OneDrive - EY/Desktop/Kafka"

cd ~/kafka_2.13-3.1.0/connectors/kafka-connect-wikimedia
cp ~/../../mnt/c/Users/EC137TG/'OneDrive - EY'/Desktop/Kafka/kafka-connect-wikimedia-1.0-all.jar .

the copy command format is cp original-file newly-copied-file, notice the dot and the end, meaning copy to the current directory

unzip ~/../../mnt/c/Users/EC137TG/'OneDrive - EY'/Desktop/Kafka/confluentinc-kafka-connect-elasticsearch-13.1.0.zip

java客户端使用kafka时什么情况下使用kafka client和spring kafka？
首先，spring最主要的用途是作为一个ioc框架，管理系统内的资源，spring集成kafka是为了方便管理及使用kafka的对象及资源， spring集成任何框架、组件、工具的目的都在于此。这个问题实际上就等于在问什么时候用spring