Blog recommendation｜Zero experience playing isolation strategy: multiple Pulsar clusters

Reminder: This article is "Pulsar's Handbook for Dummies". It is a zero-experience Xiaobai pit mine for lightning protection. Just follow the instructions and the success rate is 100%!

This article is the second article of the "Pulsar Isolation Strategy Series", and the first blog of the series- "In-depth analysis of how to achieve isolation in Pulsar" -focuses on the Pulsar isolation strategy and how to achieve resource isolation in the following three ways :

• Multiple Pulsar clusters
• Shared BookKeeper cluster
• Single Pulsar cluster

This article will explain in detail the first method, that is, how to play with the Pulsar isolation strategy in a "multiple Pulsar cluster" environment, and provide detailed steps for the following operations:

• Verify data isolation.
• Synchronize and migrate cluster data.
• Expand and shrink nodes.

Prepare the environment

This article uses macOS (version 11.2.3, memory 8G) as an example.

As shown in the figure below, this article deploys 2 Pulsar clusters, each of which provides the following services:

• 1 ZooKeeper node
• 1 bookie node
• 1 broker node

Software requirements

Java 8

Environmental details

Two Pulsar clusters will be deployed below. The component details are shown in the table below.

Deployment preparation

1. download Pulsar and unzip it. This article takes the installation of Pulsar 2.7.0 as an example.
2. In any local location, create a corresponding empty folder according to the following directory structure.

|-separate-clusters
    |-configuration-store
        |-zk1
    |-cluster1
        |-zk1
        |-bk1
        |-broker1
    |-cluster2
        |-zk1
        |-bk1
        |-broker1

3. Copy the contents of the Pulsar folder after decompression to each folder created in the previous step.
4. Start configuration store .

The Configuration store provides cross-cluster configuration management and task coordination for Pulsar instances. Pulsar cluster 1 and Pulsar cluster 2 share the configuration store.

cd configuration-store/zk1
bin/pulsar-daemon start configuration-store

Deploy Pulsar cluster 1

1. Start local ZooKeeper . Deploy 1 local ZooKeeper for each Pulsar cluster, which is responsible for managing configuration and coordinating tasks for the cluster.

cd cluster1/zk1
bin/pulsar-daemon start zookeeper

2. initializes metadata . After setting up the configuration store and local ZooKeeper, you need to write metadata to ZooKeeper.

cd cluster1/zk1
bin/pulsar initialize-cluster-metadata \
  --cluster cluster1 \
  --zookeeper localhost:2181 \
  --configuration-store localhost:2184 \
  --web-service-url http://localhost:8080/ \
  --web-service-url-tls https://localhost:8443/ \
  --broker-service-url pulsar://localhost:6650/ \
  --broker-service-url-tls pulsar+ssl://localhost:6651/

3. deploys BookKeeper . BookKeeper provides message persistent storage for Pulsar. Each Pulsar broker has a bookie. The BookKeeper cluster and Pulsar cluster share local ZooKeeper.

1). Configure bookie.

Change the values of the following configuration items in the cluster1/bk1/conf/bookkeeper.conf

allowLoopback=true
prometheusStatsHttpPort=8002
httpServerPort=8002

2). Start bookie.

cd cluster1/bk1
bin/pulsar-daemon start bookie

Verify that bookie is successfully started.

bin/bookkeeper shell bookiesanity

output

Bookie sanity test succeeded

4. Deploy the broker.

1) Configure the broker.

Change the values of the following configuration items in the cluster1/broker1/conf/broker.conf

zookeeperServers=127.0.0.1:2181
configurationStoreServers=127.0.0.1:2184
clusterName=cluster1
managedLedgerDefaultEnsembleSize=1
managedLedgerDefaultWriteQuorum=1
managedLedgerDefaultAckQuorum=1

2) Start the broker.

cd cluster1/broker1
bin/pulsar-daemon start broker

Deploy Pulsar cluster 2

1. Deploy local ZooKeeper.

1) Configure local ZooKeeper.

• Change the values of the following configuration items in the cluster2/zk1/conf/zookeeper.conf

clientPort=2186admin.serverPort=9992

• Add the following configuration items to the cluster2/zk1/conf/pulsar_env.sh file.

OPTS="-Dstats_server_port=8011"

2) Start local ZooKeeper.

cd cluster2/zk1
bin/pulsar-daemon start zookeeper

2. Initialize metadata.

bin/pulsar initialize-cluster-metadata \
  --cluster cluster2 \
  --zookeeper localhost:2186 \
  --configuration-store localhost:2184 \
  --web-service-url http://localhost:8081/ \
  --web-service-url-tls https://localhost:8444/ \
  --broker-service-url pulsar://localhost:6660/ \
  --broker-service-url-tls pulsar+ssl://localhost:6661/

3. Deploy BookKeeper.

1) Configure bookie.

Change the values of the following configuration items in the cluster2/bk1/conf/bookkeeper.conf

bookiePort=3182
zkServers=localhost:2186
allowLoopback=true
prometheusStatsHttpPort=8003
httpServerPort=8003

2) Start bookie.

cd cluster2/bk1
bin/pulsar-daemon start bookie

Verify that bookie is successfully started.

bin/bookkeeper shell bookiesanity

output

Bookie sanity test succeeded

5. Deploy the broker.

1) Configure the broker.

• Change the values of the following configuration items in the cluster2/broker1/conf/broker.conf

clusterName=cluster2
zookeeperServers=127.0.0.1:2186
configurationStoreServers=127.0.0.1:2184
brokerServicePort=6660
webServicePort=8081
managedLedgerDefaultEnsembleSize=1
managedLedgerDefaultWriteQuorum=1
managedLedgerDefaultAckQuorum=1

• Change the values of the following configuration items in the cluster2/broker1/conf/client.conf

webServiceUrl=http://localhost:8081/
brokerServiceUrl=pulsar://localhost:6660/

2) Start the broker.

cd cluster2/broker1
bin/pulsar-daemon start broker

Verify data isolation

This chapter verifies whether the data in the two Pulsar clusters are isolated.

1. Create namespace1 and assign namespace1 to cluster1.

Tip: The naming rule of namespace is /. For more information about namespace, see here .

cd cluster1/broker1
bin/pulsar-admin namespaces create -c cluster1 public/namespace1

Validation results

bin/pulsar-admin namespaces list public

output

"public/default"
"public/namespace1"

2. Set the message retention policy of namespace1.

Note: If you do not set a message retention policy and the topic is not subscribed, the topic data will be automatically cleaned up after a period of time.

bin/pulsar-admin namespaces set-retention -s 100M -t 3d public/namespace1

3. Create topic1 in namespace1 and use it to write 1000 pieces of data.

Tip: pulsar-client is a command line tool for sending and consuming data. For more information about the Pulsar command line tool, see here .

bin/pulsar-client produce -m 'hello c1 to c2' -n 1000 public/namespace1/topic1
09:56:34.504 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 1000 messages successfully produced

Validation results

bin/pulsar-admin --admin-url http://localhost:8080 topics stats-internal public/namespace1/topic1

output

entriesAddedCounter shows that 1000 entries have been added.

{ 
 "entriesAddedC{
 "entriesAddedCounter" : 1000,
 "numberOfEntries" : 1000,
 "totalSize" : 65616,
 "currentLedgerEntries" : 1000,
 "currentLedgerSize" : 65616,
 "lastLedgerCreatedTimestamp" : "2021-04-22T10:24:00.582+08:00",
 "waitingCursorsCount" : 0,
 "pendingAddEntriesCount" : 0,
 "lastConfirmedEntry" : "4:999",
 "state" : "LedgerOpened",
 "ledgers" : [ {
   "ledgerId" : 4,
   "entries" : 0,
   "size" : 0,
   "offloaded" : false
 } ],
 "cursors" : { },
 "compactedLedger" : {
   "ledgerId" : -1,
   "entries" : -1,
   "size" : -1,
   "offloaded" : false
 }
}ounter" : 1000, "numberOfEntries" : 1000, "totalSize" : 65616, "currentLedgerEntries" : 1000, "currentLedgerSize" : 65616, "lastLedgerCreatedTimestamp" : "2021-04-22T10:24:00.582+08:00", "waitingCursorsCount" : 0, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "4:999", "state" : "LedgerOpened", "ledgers" : [ {   "ledgerId" : 4,   "entries" : 0,   "size" : 0,   "offloaded" : false } ], "cursors" : { }, "compactedLedger" : {   "ledgerId" : -1,   "entries" : -1,   "size" : -1,   "offloaded" : false }}

4. View the data of public/namespace1/topic1 through cluster2 (localhost:8081).

bin/pulsar-admin --admin-url http://localhost:8081 topics stats-internal public/namespace1/topic1

outputs

View failed. The printed information shows that public/namespace1 is only allocated to cluster1 and not to cluster 2. It is now verified that the data is quarantined.

Namespace missing local cluster name in clusters list: local_cluster=cluster2 ns=public/namespace1 clusters=[cluster1]
Reason: Namespace missing local cluster name in clusters list: local_cluster=cluster2 ns=public/namespace1 clusters=[cluster1]

5. In cluster2, write data to public/namespace1/topic1.

cd cluster2/broker1
bin/pulsar-client produce -m 'hello c1 to c2' -n 1000 public/namespace1/topic1

outputs

The result shows that the number of written messages is 0, and the operation failed because namespace1 is only allocated to cluster1 and not to cluster 2. It is now verified that the data is quarantined.

12:09:50.005 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully produced

Migrate data

The following steps continue with the previous "Verify Data Isolation".

After confirming that the data has been isolated, this chapter describes how to synchronize (using the cross-region replication function) and migrate cluster data.

1. Assign namespace1 to cluster2, that is, add cluster2 to the cluster list of namesapce1.

This step enables cross-region replication to synchronize the data of cluster1 and cluster2.

bin/pulsar-admin namespaces set-clusters --clusters cluster1,cluster2 public/namespace1

Validation results

bin/pulsar-admin namespaces get-clusters public/namespace1

output

"cluster1"
"cluster2"

2. Check if cluster2 has topic1.

bin/pulsar-admin --admin-url http://localhost:8081 topics stats-internal public/namespace1/topic1

output

The result shows that the topic1 of cluster2 has 1000 pieces of data, indicating that the data of topic1 of cluster1 has been successfully copied to cluster2.

{
  "entriesAddedCounter" : 1000,
  "numberOfEntries" : 1000,
  "totalSize" : 75616,
  "currentLedgerEntries" : 1000,
  "currentLedgerSize" : 75616,
  "lastLedgerCreatedTimestamp" : "2021-04-23T12:02:52.929+08:00",
  "waitingCursorsCount" : 1,
  "pendingAddEntriesCount" : 0,
  "lastConfirmedEntry" : "1:999",
  "state" : "LedgerOpened",
  "ledgers" : [ {
    "ledgerId" : 1,
    "entries" : 0,
    "size" : 0,
    "offloaded" : false
  } ],
  "cursors" : {
    "pulsar.repl.cluster1" : {
      "markDeletePosition" : "1:999",
      "readPosition" : "1:1000",
      "waitingReadOp" : true,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : 1000,
      "cursorLedger" : 2,
      "cursorLedgerLastEntry" : 2,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2021-04-23T12:02:53.248+08:00",
      "state" : "Open",
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "properties" : { }
    }
  },
  "compactedLedger" : {
    "ledgerId" : -1,
    "entries" : -1,
    "size" : -1,
    "offloaded" : false
  }
}

3. Migrate the producer and consumer of cluster1 to cluster2.

PulsarClient pulsarClient1 = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
// migrate the client to cluster2 pulsar://localhost:6660
PulsarClient pulsarClient2 = PulsarClient.builder().serviceUrl("pulsar://localhost:6660").build();

4. Remove cluster1 from the cluster list of namespace1.

bin/pulsar-admin namespaces set-clusters --clusters cluster2 public/namespace1

5. Check the data of topic1 of cluster1.

cd cluster1/broker1
bin/pulsar-admin --admin-url http://localhost:8080 topics stats-internal public/namespace1/topic1

output

The result shows that the data is empty, indicating that the data has been successfully removed from topic1 of cluster1.

{
  "entriesAddedCounter" : 0,
  "numberOfEntries" : 0,
  "totalSize" : 0,
  "currentLedgerEntries" : 0,
  "currentLedgerSize" : 0,
  "lastLedgerCreatedTimestamp" : "2021-04-23T15:20:08.1+08:00",
  "waitingCursorsCount" : 1,
  "pendingAddEntriesCount" : 0,
  "lastConfirmedEntry" : "3:-1",
  "state" : "LedgerOpened",
  "ledgers" : [ {
    "ledgerId" : 3,
    "entries" : 0,
    "size" : 0,
    "offloaded" : false
  } ],
  "cursors" : {
    "pulsar.repl.cluster2" : {
      "markDeletePosition" : "3:-1",
      "readPosition" : "3:0",
      "waitingReadOp" : true,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : 0,
      "cursorLedger" : 4,
      "cursorLedgerLastEntry" : 0,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2021-04-23T15:20:08.122+08:00",
      "state" : "Open",
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "properties" : { }
    }
  },
  "compactedLedger" : {
    "ledgerId" : -1,
    "entries" : -1,
    "size" : -1,
    "offloaded" : false
  }
}

So far, we have successfully copied the topic1 data of cluster1 to cluster2, and then removed the topic1 data of cluster1.

Scaling nodes

This chapter describes how to scale up and down broker and bookie nodes.

Broker

Add broker node

In this example, two partitioned topics are created in cluster1/broker1, and two broker nodes are added. Then unload the partitioned topic data and check the distribution of the data among the 3 brokers.

1. View the broker information of cluster1.

cd/cluster1/broker1
bin/pulsar-admin brokers list cluster1

outputs

The result shows that the current cluster1 only has broker1.

"192.168.0.105:8080"

2. Create 2 partitioned topics on cluster1/broker1.

Create 6 partitions for partitioned-topic1 and 7 partitions for partitioned-topic2.

bin/pulsar-admin topics create-partitioned-topic -p 6 public/namespace1/partitioned-topic1
bin/pulsar-admin topics create-partitioned-topic -p 7 public/namespace1/partitioned-topic2

View Results.

bin/pulsar-admin topics partitioned-lookup public/namespace1/partitioned-topic1

output

The result shows that all data of partitioned-topic1 belongs to broker1.

"persistent://public/namespace1/partitioned-topic1-partition-0    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-1    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-2    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-3    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-4    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-5    pulsar://192.168.0.105:6650"

input

bin/pulsar-admin topics partitioned-lookup public/namespace1/partitioned-topic2

output

The result shows that all data of partitioned-topic2 belongs to broker1.

"persistent://public/namespace1/partitioned-topic2-partition-0    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-1    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-2    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-3    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-4    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-5    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-6    pulsar://192.168.0.105:6650"

3. Two new broker nodes are added: broker2 and broker3.

1) Deployment preparation. Create broker2 and broker3 folders in the cluster1 folder, and copy the contents of the Pulsar folder after decompression to the broker2 and broker3 folders.

|-separate-clusters
    |-configuration-store
        |-zk1
    |-cluster1
        |-zk1
        |-bk1
        |-broker1
        |-broker2
        |-broker3
    |-cluster2
        |-zk1
        |-bk1
        |-broker1

2) Deploy the broker.

a). Configure the broker.

b). Start the broker.

c). View the brokers that have been started in cluster1.

bin/pulsar-admin brokers list cluster1

output

"192.168.0.105:8080" // broker1
"192.168.0.105:8082" // broker2
"192.168.0.105:8083" // broker3

5. Uninstall the partitioned-topic1 data in namespace 1.

bin/pulsar-admin namespaces unload public/namespace1

Validation results.

1). View the data distribution of partitioned-topic1.

bin/pulsar-admin topics partitioned-lookup public/namespace1/partitioned-topic1

output

The results show that the data of partitioned-topic1 is evenly distributed among broker1, broker2, and broker3.

"persistent://public/namespace1/partitioned-topic1-partition-0    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-1    pulsar://192.168.0.105:6653"
"persistent://public/namespace1/partitioned-topic1-partition-2    pulsar://192.168.0.105:6652"
"persistent://public/namespace1/partitioned-topic1-partition-3    pulsar://192.168.0.105:6653"
"persistent://public/namespace1/partitioned-topic1-partition-4    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-5    pulsar://192.168.0.105:6653"

2). View the data distribution of partitioned-topic2.

bin/pulsar-admin topics partitioned-lookup public/namespace1/partitioned-topic2

The results show that the data of partitioned-topic2 is evenly distributed among broker1, broker2, and broker3.

output

"persistent://public/namespace1/partitioned-topic2-partition-0    pulsar://192.168.0.105:6653"
"persistent://public/namespace1/partitioned-topic2-partition-1    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-2    pulsar://192.168.0.105:6653"
"persistent://public/namespace1/partitioned-topic2-partition-3    pulsar://192.168.0.105:6652"
"persistent://public/namespace1/partitioned-topic2-partition-4    pulsar://192.168.0.105:6653"
"persistent://public/namespace1/partitioned-topic2-partition-5    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-6    pulsar://192.168.0.105:6653"

Reduce broker nodes

The following steps continue from the previous "How to add a broker node" to continue.

This example reduces one broker node in cluster1 and checks the distribution of partitioned topic data among the remaining two brokers.

1. Reduce one broker node, that is, stop broker3.

cd/cluster1/broker3
bin/pulsar-daemon stop broker

Validation results.

bin/pulsar-admin brokers list cluster1

output

The result shows that the current cluster1 only starts broker1 and broker2.

"192.168.0.105:8080" // broker1
"192.168.0.105:8082" // broker2

2. Check the distribution of partitioned-topic1 data.

bin/pulsar-admin topics partitioned-lookup public/namespace1/partitioned-topic1

output

The results show that the partitioned-topic1 data is evenly distributed to broker1 and broker2, that is, the data that originally belonged to broker3 has been evenly redistributed to broker1 and broker2.

"persistent://public/namespace1/partitioned-topic1-partition-0    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-1    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-2    pulsar://192.168.0.105:6652"
"persistent://public/namespace1/partitioned-topic1-partition-3    pulsar://192.168.0.105:6652"
"persistent://public/namespace1/partitioned-topic1-partition-4    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic1-partition-5    pulsar://192.168.0.105:6650"

In the same way, the data of partitioned-topic2 is equally distributed to broker1 and broker2.

bin/pulsar-admin topics partitioned-lookup public/namespace1/partitioned-topic2

outputs

"persistent://public/namespace1/partitioned-topic2-partition-0    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-1    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-2    pulsar://192.168.0.105:6652"
"persistent://public/namespace1/partitioned-topic2-partition-3    pulsar://192.168.0.105:6652"
"persistent://public/namespace1/partitioned-topic2-partition-4    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-5    pulsar://192.168.0.105:6650"
"persistent://public/namespace1/partitioned-topic2-partition-6    pulsar://192.168.0.105:6652"

Bookie

Add bookie node

In this example, there is already bookie1 in cluster1/bookkeeper 1. After adding 2 bookie nodes, write data to topic1 and check whether the data has multiple copies.

1. View the bookie information of cluster1.

cd cluster1/bk1
bin/bookkeeper shell listbookies -rw -h

output

The result shows that the current cluster1 only has bookie1.

12:31:34.933 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - ReadWrite Bookies :
12:31:34.946 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:192.168.0.105:3181, IP:192.168.0.105, Port:3181, Hostname:192.168.0.105

2. Allow 3 bookie node services.

Change the values of the following configuration items in the cluster1/broker1/conf/broker.conf

managedLedgerDefaultEnsembleSize=3 // 指定 bookie 节点服务的数量
managedLedgerDefaultWriteQuorum=3 // 指定数据副本写入的数量
managedLedgerDefaultAckQuorum=2  // 指定数据成功写入几个副本后，数据才算写入成功

3. Restart broker1 to make the configuration effective.

cd cluster1/broker1
bin/pulsar-daemon stop broker
bin/pulsar-daemon start broker

4. Set the public/default message retention strategy.

Note: If you do not set a message retention policy and the topic is not subscribed, the topic data will be automatically cleaned up after a period of time.

cd cluster1/broker1
bin/pulsar-admin namespaces set-retention -s 100M -t 3d public/default

5. Create topic1 in public/default and write 100 pieces of data.

bin/pulsar-client produce -m 'hello' -n 100 topic1

output

The result shows that insufficient number of bookie nodes causes data write failure.

···
12:40:38.886 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ClientCnx - [id: 0x56f92aff, L:/192.168.0.105:53069 - R:/192.168.0.105:6650] Received error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
...
12:40:38.886 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages
…
12:40:38.890 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully produced

6. Add 2 bookie nodes: bookie2 and bookie3.

1). Deployment preparation.

Add bk2 and bk3 folders to cluster1, and copy the contents of the Pulsar folder after decompression to bk2 and bk3 folders.

|-separate-clusters
    |-configuration-store
        |-zk1
    |-cluster1
        |-zk1
        |-bk1
        |-bk2
        |-bk3
        |-broker1
    |-cluster2
        |-zk1
        |-bk1
        |-broker1

2). Deploy bookie.

a). Configure bookie.

b). Start bookie.

c). Check the started bookie in cluster1.

bin/bookkeeper shell listbookies -rw -h

output

The result shows that 3 bookies have been started in cluster 1:

• bookie1：192.168.0.105:3181
• bookie2：192.168.0.105:3183
• bookie3：192.168.0.105:3184

12:12:47.574 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:192.168.0.105:3183, IP:192.168.0.105, Port:3183, Hostname:192.168.0.105 
12:12:47.575 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:192.168.0.105:3184, IP:192.168.0.105, Port:3184, Hostname:192.168.0.105
12:12:47.576 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:192.168.0.105:3181, IP:192.168.0.105, Port:3181, Hostname:192.168.0.105 

7. Set the public/default message retention strategy.

Note: If you do not set a message retention policy and the topic is not subscribed, the topic data will be automatically cleaned up after a period of time.

cd cluster1/broker1bin/pulsar-admin namespaces set-retention -s 100M -t 3d public/default

8. Create topic1 in public/default and write 100 pieces of data.

bin/pulsar-client produce -m 'hello' -n 100 topic1

output

The result shows that the data was written successfully.

...
12:17:40.222 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 100 messages successfully produced

9. View the information of topic1.

bin/pulsar-admin topics stats-internal topic1

output

The result shows that ledgerId 5 saves the data of topic1.

{
  "entriesAddedCounter" : 100,
  "numberOfEntries" : 100,
  "totalSize" : 5500,
  "currentLedgerEntries" : 100,
  "currentLedgerSize" : 5500,
  "lastLedgerCreatedTimestamp" : "2021-05-11T12:17:38.881+08:00",
  "waitingCursorsCount" : 0,
  "pendingAddEntriesCount" : 0,
  "lastConfirmedEntry" : "5:99",
  "state" : "LedgerOpened",
  "ledgers" : [ {
    "ledgerId" : 5,
    "entries" : 0,
    "size" : 0,
    "offloaded" : false
  } ],
  "cursors" : { },
  "compactedLedger" : {
    "ledgerId" : -1,
    "entries" : -1,
    "size" : -1,
    "offloaded" : false
  }
}

10. Check which bookie nodes ledgerid 5 is stored on.

bin/bookkeeper shell ledgermetadata -ledgerid 5

output

The result shows that as configured in the previous article, ledgerid 5 is stored on bookie1 (3181), bookie2 (3183) and bookie3 (3184).

...
12:23:17.705 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - ledgerID: 5
12:23:17.714 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - LedgerMetadata{formatVersion=3, ensembleSize=3, writeQuorumSize=3, ackQuorumSize=2, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.168.0.105:3184, 192.168.0.105:3181, 192.168.0.105:3183]}, customMetadata={component=base64:bWFuYWdlZC1sZWRnZXI=, pulsar/managed-ledger=base64:cHVibGljL2RlZmF1bHQvcGVyc2lzdGVudC90b3BpYzE=, application=base64:cHVsc2Fy}}
...

Reduce bookie nodes

The following steps continue from the previous "How to add a bookie node" to continue.

This example reduces 2 bookie nodes in cluster1, then writes data to topic2, and sees which nodes the data is stored in.

1. Allow 1 bookie node to serve.

Change the values of the following configuration items in the cluster1/broker1/conf/broker.conf

managedLedgerDefaultEnsembleSize=1 // 指定 bookie 节点服务的数量
managedLedgerDefaultWriteQuorum=1 // 指定数据副本写入的数量
managedLedgerDefaultAckQuorum=1  // 指定数据成功写入几个副本后，数据才算写入成功

2. Restart broker1 to make the configuration effective.

cd cluster1/broker1
bin/pulsar-daemon stop broker
bin/pulsar-daemon start broker

3. View the bookie information of cluster1.

cd cluster1/bk1
bin/bookkeeper shell listbookies -rw -h

output

The result shows that the current cluster1 has started bookie1 (3181), bookie2 (3183) and bookie3 (3184).

...
15:47:41.370 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - ReadWrite Bookies :
15:47:41.382 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:192.168.0.105:3183, IP:192.168.0.105, Port:3183, Hostname:192.168.0.105
15:47:41.383 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:192.168.0.105:3184, IP:192.168.0.105, Port:3184, Hostname:192.168.0.105
15:47:41.384 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:192.168.0.105:3181, IP:192.168.0.105, Port:3181, Hostname:192.168.0.105
…

4. Reduce 2 bookie nodes, that is, stop bookie2 and bookie3.

Tip: For more information on how to reduce bookie, see here .

cd cluster1/bk2
bin/bookkeeper shell listunderreplicated
bin/pulsar-daemon stop bookie
bin/bookkeeper shell decommissionbookie

cd cluster1/bk3
bin/bookkeeper shell listunderreplicated
bin/pulsar-daemon stop bookie
bin/bookkeeper shell decommissionbookie

5. View the bookie information of cluster1.

cd cluster1/bk1
bin/bookkeeper shell listbookies -rw -hcd cluster1/bk1bin/bookkeeper shell listbookies -rw -h

outputs

The result shows that the current cluster1 only starts bookie1 (3181).

…
16:05:28.690 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - ReadWrite Bookies :
16:05:28.700 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:192.168.0.105:3181, IP:192.168.0.105, Port:3181, Hostname:192.168.0.105
...

6. Set the public/default message retention policy.

Note: If you do not set a message retention policy and do not subscribe, the data will be automatically cleaned up after a period of time.

cd cluster1/broker1
bin/pulsar-admin namespaces set-retention -s 100M -t 3d public/default

7. Create topic2 in public/default and write 100 pieces of data.

bin/pulsar-client produce -m 'hello' -n 100 topic2

output

The result shows that the data was written successfully.

…
16:06:59.448 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 100 messages successfully produced

8. View the information of topic2.

bin/pulsar-admin topics stats-internal topic2

output

The result shows that ledgerId 7 saves the data of topic 2.

{
  "entriesAddedCounter" : 100,
  "numberOfEntries" : 100,
  "totalSize" : 5400,
  "currentLedgerEntries" : 100,
  "currentLedgerSize" : 5400,
  "lastLedgerCreatedTimestamp" : "2021-05-11T16:06:59.058+08:00",
  "waitingCursorsCount" : 0,
  "pendingAddEntriesCount" : 0,
  "lastConfirmedEntry" : "7:99",
  "state" : "LedgerOpened",
  "ledgers" : [ {
    "ledgerId" : 7,
    "entries" : 0,
    "size" : 0,
    "offloaded" : false
  } ],
  "cursors" : { },
  "compactedLedger" : {
    "ledgerId" : -1,
    "entries" : -1,
    "size" : -1,
    "offloaded" : false
  }
}

9. Check which bookie nodes ledgerid 7 is stored on.

bin/bookkeeper shell ledgermetadata -ledgerid 7

output

The result shows that ledgerid 7 is stored on bookie1 (3181).

...
16:11:28.843 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - ledgerID: 7
16:11:28.846 [main] INFO  org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - LedgerMetadata{formatVersion=3, ensembleSize=1, writeQuorumSize=1, ackQuorumSize=1, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.168.0.105:3181]}, customMetadata={component=base64:bWFuYWdlZC1sZWRnZXI=, pulsar/managed-ledger=base64:cHVibGljL2RlZmF1bHQvcGVyc2lzdGVudC90b3BpYzM=, application=base64:cHVsc2Fy}}
…

Summarize

This article is the second in the "Pulsar Isolation Strategy Series". It explains how to verify data isolation, synchronize cluster data, and scale nodes in a "multiple Pulsar cluster" environment.

The next blog in this series will analyze in detail how to play the Pulsar isolation strategy in the "Shared BookKeeper Cluster" environment. It will also be the "Pulsar Dummies Manual"! Stay tuned!

More information

If you are interested in the "Pulsar Isolation Strategy Series", welcome to read more related resources:

• Basics: Pulsar Isolation Strategy-Official Website Document
• Advanced articles: depth analysis of how to achieve isolation in Pulsar

Click the link to get the Apache Pulsar hard core dry goods information!