nsq deployment/delivery/consumption/cluster example

docker pull nsqio/nsq

Service ports and relationships

topic & channel

cluster mode

pre-preparation

Because the multi-node deployment is implemented on a single machine through the docker container, the nsqd/nsqadmin container wants to communicate with nsqlookup , and needs to access the service port exposed by nsqlookup on the host machine. Therefore, we need to communicate with the address related to nsqlookup when creating the nsqd/nsqadmin container. Fill in the host machine ip .

# 获取宿主机内网ip
ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:"

For example, mine is 10.10.31.147 , please pay attention to the replacement later.

nsqlookupd

# 4160 tcp 供 nsqd 注册用
# 4161 http 供 nsqdadmin 和 consumer 查询服务名字
docker run -d --name nsqlookupd \
-p 4160:4160 -p 4161:4161 \
nsqio/nsq /nsqlookupd

nsqd

Create two nsqd nodes

nsq has two producer service ports tcp-address and http-address

# --broadcast-address 节点主机地址 用来供外放访问
# 下文设为宿主机IP 以便admin访问统计实例状态
# --tcp-address tcp 协议的 producer 端口
# --http-address http 协议的 producer 端口
# --lookupd-tcp-address lookupd 的 tcp 地址
# --data-path 数据持久化存储路径

# nsq0 tcp://127.0.0.1:4150/ http://127.0.0.1:4151/
docker run -d -v /tmp/nsq0:/tmp/nsq \
-p 4150:4150 -p 4151:4151 \
--name nsqd0 nsqio/nsq /nsqd \
--tcp-address :4150 \
--http-address :4151 \
--broadcast-address=10.10.31.147 \
--lookupd-tcp-address=10.10.31.147:4160 \
--data-path /tmp/nsq

# nsq1 tcp://127.0.0.1:4250/ http://127.0.0.1:4251/
docker run -d -v /tmp/nsq1:/tmp/nsq \
-p 4250:4250 -p 4251:4251 \
--name nsqd1 nsqio/nsq /nsqd \
--tcp-address :4250 \
--http-address :4251 \
--broadcast-address=10.10.31.147 \
--lookupd-tcp-address=10.10.31.147:4160 \
--data-path /tmp/nsq

nsqadmin

# 4171 admin管理平台服务端口
# --lookupd-http-address lookupd 的 http 地址
docker run -d --name nsqadmin \
-p 4171:4171 nsqio/nsq /nsqadmin \
--lookupd-http-address=10.10.31.147:4161

http://127.0.0.1:4171/nodes

topic

# 创建主题
curl -X POST http://127.0.0.1:4151/topic/create?topic=test
curl -X POST http://127.0.0.1:4251/topic/create?topic=test

channel

# channel 相当于消费组 channel 与 topic 之间相当于订阅发布的关系
curl -X POST 'http://127.0.0.1:4151/channel/create?topic=test&channel=chan_4151_1'
curl -X POST 'http://127.0.0.1:4151/channel/create?topic=test&channel=chan_4151_2'

curl -X POST 'http://127.0.0.1:4251/channel/create?topic=test&channel=chan_4251_1'
curl -X POST 'http://127.0.0.1:4251/channel/create?topic=test&channel=chan_4251_2'

consumer

Here we borrow the nsq related command script in the nsqlookupd container.
nsq_to_file used as a consumer to obtain all nodes containing the specified topic by querying lookupd , and bind channel . When a producer sends a message to this topic , the subscribed channel consumes it.

# 通过查询 lookupd 获取所有的 topic
# 订阅 topic > channel

docker exec -it nsqlookupd nsq_to_file \
--topic=test --channel=chan_4151_1 \
--output-dir=/tmp/chan_4151_1 \
--lookupd-http-address=127.0.0.1:4161

docker exec -it nsqlookupd nsq_to_file  \
--topic=test --channel=chan_4251_1 \
--output-dir=/tmp/chan_4251_1 \
--lookupd-http-address=127.0.0.1:4161

producer

# topic 发布消息 每个 channel 会受到此消息
# 且负载轮训分配给channel下的其中一个消费者
curl -d 'hello world 4151' 'http://127.0.0.1:4151/pub?topic=test'
curl -d 'hello world 4251' 'http://127.0.0.1:4251/pub?topic=test'

High Availability Scenario

copies, idempotent consumption

A high-availability cluster naturally requires the concept of replicas, but the nsq cluster does not have a node data synchronization mechanism, unlike other advanced queues that have the concept of maintaining replicas of synchronized data, so the replicas of nsq need to be maintained and implemented at the code level.

Take nsqd0 nsqd1 two nodes as an example, how to achieve high availability of the cluster? Create topic_ha & channel_replic nsqd0 nsqd1 and send the message to both nodes at the same time.

When consumers subscribe for consumption through lookupd mode, they can subscribe to channel_replic topic_ha At this time, when sending the same message to nsqd0 and nsqd1 , topic_ha maintains a backup copy to do message idempotent consumption to prevent repeated processing. When one of the nsqd nodes hangs, we can still deliver and consume business messages normally. .

curl -X POST http://127.0.0.1:4151/topic/create?topic=test_ha
curl -X POST http://127.0.0.1:4251/topic/create?topic=test_ha

curl -X POST 'http://127.0.0.1:4151/channel/create?topic=test_ha&channel=chan_replic'
curl -X POST 'http://127.0.0.1:4251/channel/create?topic=test_ha&channel=chan_replic'

# 或者使用文中最后的 go 代码体验集群投递/消费的概念
docker exec -it nsqlookupd nsq_to_file  \
--topic=test_ha --channel=chan_replic \
--output-dir=/tmp/chan_replic \
--lookupd-http-address=127.0.0.1:4161

You can see that all nodes containing this topic&channel are obtained through nsqlookupd

2022/03/03 09:49:34 INF    1 [test_ha/chan_replic] querying nsqlookupd http://127.0.0.1:4161/lookup?topic=test_ha
2022/03/03 09:49:34 INF    1 [test_ha/chan_replic] (10.10.31.147:4150) connecting to nsqd
2022/03/03 09:49:34 INF    1 [test_ha/chan_replic] (10.10.31.147:4250) connecting to nsqd

basic concepts

1. The high-availability cluster of nsq does not have the function of automatically synchronizing copies. That is, if you have N nodes, you need to create a topic with the same name on the N nodes, and you need to deliver a message to these N nodes once when delivering a message.
2. For the consumption of nsq, the best method is for consumers to connect to the lookupd service and query which nodes the subscribed topics are distributed in. Consumers are mainly topic-based and will subscribe to all message data including topic nodes.
3. Channel is the consumer group, the load balancing message queue within the group, and the subscription and publishing between the groups. Like Kafka's low-level consumer group. Join the same consumer group, load balance consumption, and different consumer groups are subscribers to each other before.
4. On the same node, if you subscribe to the same topic and the same channel, you will join the consumption group, and the load will be balanced for consumption within the group.
5. On the same node, subscribing to the same topic and different channels is called subscription publishing, and at least one consumer in each consumer group can get the message.

Instance (cluster delivery/consumption)

nsqProducer
I also encapsulated here to automatically obtain all nsqd containing topic through lookupd and build tcpProducer . In this way, when delivering messages to a topic distributed on multiple nsqd nodes, there is no need to hand-deliver them one by one.

package main

import (
    "encoding/json"
    "errors"
    "flag"
    "github.com/nsqio/go-nsq"
    "io/ioutil"
    "log"
    "net/http"
    "os"
    "os/signal"
    "strconv"
    "strings"
    "syscall"
    "time"
)

var TopicProducers map[string][]*nsq.Producer

type LookupTopicRes struct {
    Channels  []string       `json:"channels"`
    Producers []ProducerInfo `json:"producers"`
}

type ProducerInfo struct {
    RemoteAddress    string `json:"remote_address"`
    Hostname         string `json:"hostname"`
    BroadcastAddress string `json:"broadcast_address"`
    TcpPort          int    `json:"tcp_port"`
    HttpPort         int    `json:"http_port"`
    Version          string `json:"version"`
}

// nsq 内部日志
type nsqServerLogger struct {
}

func (nsl *nsqServerLogger) Output(callDepth int, s string) error {
    log.Println("nsqServerLogger", callDepth, s[:3], strings.Trim(s[3:], " "))
    return nil
}

func main() {
    var topic string

    flag.StringVar(&topic, "topic", "test", "topic name default test")
    flag.Parse()

    // 为每个包含topic的nsqd节点 创建1个生产者
    NewTopicProducer(topic)

    go func() {
        timerTicker := time.Tick(2 * time.Second)
        for {
            <-timerTicker
            totalNode, failedNode, err := PublishTopicMsg(topic, []byte("hello nsq "+time.Now().Format("2006-01-02 15:04:05")))
            if err != nil {
                log.Fatalln("PublishTopicMsg err topic", topic, "err", err.Error())
            }
            log.Println("PublishTopicMsg ok topic", topic, "totalNode", totalNode, "failedNode", failedNode)
        }
    }()

    // wait for signal to exit
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
    sigMsg := <-sigChan
    log.Println("sigMsg", sigMsg)

    // Gracefully stop the producer.
    for _, producers := range TopicProducers {
        for _, producer := range producers {
            producer.Stop()
        }
    }
}

// NewTopicProducer
// 获取 topic 所有的 nsqd 节点 并建立 tcp 链接
func NewTopicProducer(topic string) {
    TopicProducers = make(map[string][]*nsq.Producer)
    config := nsq.NewConfig()
    topicNodeAddr := getTopicNodeAddrSet(topic)
    var producers []*nsq.Producer
    for _, addr := range topicNodeAddr {
        producer, err := nsq.NewProducer(addr, config)
        if err != nil {
            log.Fatalln("newProducer err topic", topic, "err", err.Error())
        }
        producer.SetLogger(&nsqServerLogger{}, nsq.LogLevelDebug)
        producers = append(producers, producer)
    }
    TopicProducers[topic] = producers
}

// PublishTopicMsg
// 向 topic 发送消息 会自动向每一个包含此 topic 的节点发送 集群模式
func PublishTopicMsg(topic string, msg []byte) (totalNode int, failedNode int, err error) {
    producers, ok := TopicProducers[topic]
    if !ok {
        return 0, 0, errors.New("PublishTopicMsg err topic not exists")
    }
    totalNode = len(producers)
    for _, producer := range producers {
        errPub := producer.Publish(topic, msg)
        if nil != errPub {
            failedNode++
        }
    }
    return
}

// 获取 topic 的所在的 nsqd 节点集合
func getTopicNodeAddrSet(topic string) (topicNodeAddrArr []string) {
    resp, _ := http.Get("http://127.0.0.1:4161/lookup?topic=" + topic)
    defer func() {
        _ = resp.Body.Close()
    }()

    bodyRaw, _ := ioutil.ReadAll(resp.Body)
    lookupTopicRes := &LookupTopicRes{}
    _ = json.Unmarshal(bodyRaw, &lookupTopicRes)

    for _, producer := range lookupTopicRes.Producers {
        topicNodeAddrArr = append(topicNodeAddrArr, producer.BroadcastAddress+":"+strconv.Itoa(producer.TcpPort))
    }

    return topicNodeAddrArr
}

nsqConsumer
Use all nsqd nodes that automatically obtain topic + channel through lookupd , and subscribe for consumption.

package main

import (
    "flag"
    "github.com/nsqio/go-nsq"
    "log"
    "os"
    "os/signal"
    "syscall"
)

type nsqMessageHandler struct{}

// HandleMessage implements the Handler interface.
func (h *nsqMessageHandler) HandleMessage(m *nsq.Message) error {
    if len(m.Body) == 0 {
        // Returning nil will automatically send a FIN command to NSQ to mark the message as processed.
        // In this case, a message with an empty body is simply ignored/discarded.
        return nil
    }

    // do whatever actual message processing is desired
    log.Println("HandleMessage nsqd:", m.NSQDAddress, "msg:", string(m.Body))

    // Returning a non-nil error will automatically send a REQ command to NSQ to re-queue the message.
    return nil
}

func main() {
    var topic string
    var channel string
    var count int
    var consumerGroup []*nsq.Consumer

    flag.StringVar(&topic, "topic", "test", "topic name default test")
    flag.StringVar(&channel, "channel", "test", "channel name default test")
    flag.IntVar(&count, "count", 1, "consumer count default 1")
    flag.Parse()

    // Instantiate a consumer that will subscribe to the provided channel.
    config := nsq.NewConfig()
    config.MaxInFlight = 10 // 一个消费者可同时接收的最多消息数量

    for i := 0; i < count; i++ {
        consumer, err := nsq.NewConsumer(topic, channel, config)
        if err != nil {
            log.Fatalln("NewConsumer err:", err.Error())
        }

        // Set the Handler for messages received by this Consumer. Can be called multiple times.
        // See also AddConcurrentHandlers.
        consumer.AddHandler(&nsqMessageHandler{})
        // 并发模式的消费处理 消费者会启用 n 个协程处理消息
        consumer.AddConcurrentHandlers(&nsqMessageHandler{}, 10)
        consumer.SetLogger(&nsqServerLogger{}, nsq.LogLevelDebug)

        // Use nsqlookupd to discover nsqd instances.
        // See also ConnectToNSQD, ConnectToNSQDs, ConnectToNSQLookupds.
        // 会订阅所有包含当前 topic 的 nsqd 实例
        // 多用于集群模式时 生产者向多个含有topic的实例同时发送消息
        // 当其中部分实例挂到时 消费者仍可通过其它实例获得消息
        // !此处要做消息幂等处理！
        err = consumer.ConnectToNSQLookupd("localhost:4161")
        if err != nil {
            log.Fatalln("ConnectToNSQLookupd err:", err.Error())
        } else {
            log.Println("ConnectToNSQLookupd success topic:", topic, "channel:", channel)
        }

        consumerGroup = append(consumerGroup, consumer)
    }

    // wait for signal to exit
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
    sigMsg := <-sigChan
    log.Println("sigMsg", sigMsg)

    // Gracefully stop the consumer.
    for _, consumer := range consumerGroup {
        consumer.Stop()
    }
}

run

go run nsqConsumer.go -topic test_ha -channel chan_replic -count=2
go run nsqProducer.go -topic test_ha

Two consumers with chan each other load balancing constitute consumer group A, consumer group A in order to subscribe nsqd0, nsqd1 of chan_replic , producers to test_ha delivery cluster mode, nsqd0, nsqd1 after receiving the message, will respectively consumer group A message delivery time , Consumer group A depends on the load balancer as to which consumer consumes.