Help the community｜StreamNative Hub upgrade to help you develop Pulsar Connector quickly and efficiently!

About Apache Pulsar

Apache Pulsar is the top-level project of the Apache Software Foundation. It is the next-generation cloud-native distributed message flow platform. It integrates messaging, storage, and lightweight functional computing. It uses a separate architecture design for computing and storage to support multi-tenancy, persistent storage, Multi-machine room and cross-regional data replication, with strong consistency, high throughput and low latency, highly scalable streaming data storage characteristics.

Why choose Pulsar Connector?

Pulsar can be easily and quickly integrated with external systems through connector[1]. According to the 2021 Pulsar user survey report (to be released this month), 30% of Pulsar users use connectors, which is one of the most commonly used functions of Pulsar.
In order to promote the development of the connector and improve its ease of use, we launched StreamNative Hub[2] in 2020, which supports multiple operations such as finding, downloading, using, storing and sharing Pulsar related plug-ins, and providing extensive Pulsar integration. Since the Hub went online, a large number of connectors have been added. Popular Pulsar Connector plug-ins include AWS SQS connector[3], AWS Lambda connector[4], AMQP1_0 connector[5], IoTDB connector[6], etc.
Compared with the early days of Hub launch, the development and use of Pulsar connector is easier to learn. This article mainly introduces the recent updates of StreamNative Hub, so that you can develop and use the connector more easily!

About StreamNative Hub

StreamNative Hub[7] is an application store that provides a large number of Pulsar plug-ins and integrations for developing event streaming applications. The key components of

• Connector : Allows users to migrate streaming data in and out of Pulsar, helping enterprises to more easily introduce Pulsar into their existing infrastructure and simplify system integration. StreamNative Hub supports users to use all built-in connectors of Pulsar.
• Offloader : Allows users to unload historical data from BookKeeper to external remote storage, saving data storage costs, and can be easily expanded with the amount of data.
• protocol handler : Allow users to natively and dynamically support other messaging protocols (for example, Kafka, AMQP, and MQTT) when running Pulsar. Pulsar's enterprise-level functions can be used without modifying the code, which greatly simplifies the operation. .

This article mainly introduces the latest developments on the connector and how these improvements simplify the development and use of the Pulsar connector.
With the continuous increase in the number of Pulsar Connector users and contributors, we have been able to improve the ease of use of StreamNative Hub according to user needs. The following chapters detail the related improvements of StreamNative Hub.

Pulsar Connector New Development Manual

In order to simplify the integration of Pulsar and external systems, we have compiled a new version of the development guide-Pulsar Connector Development Manual [8] to provide a reference for developers' connector development work.

• Develop a new Connector : If you need to transfer data between Pulsar and other systems (these systems do not have a Pulsar connector for the time being), it is recommended that you read the Pulsar connector development manual [9]! You will learn how to develop connectors, how to contribute connectors to StreamNative Hub and other details. In order to simplify development, code and documentation templates are also provided in the manual.
• Promote the existing Connector : If you want to contribute the connector to the community, you can host it in any public repository and display it on StreamNative Hub. According to the instructions in the Pulsar connector development manual [10], only one line of code is required to synchronize the document to the StreamNative Hub.

StreamNative Hub future plans

We are always looking for new ways to improve StreamNative Hub, and we are also trying other upgrades, such as adding more comprehensive tests to improve the availability, reliability and performance of the connector; supporting the deployment and use of the connector on the cloud through GUI tools. stay tuned!

Contribute to Connector

If you are a connector developer, we recommend that you add the connector to the StreamNative Hub! In StreamNative Hub, your connector will be understood and used by more people, speeding up its development process, and at the same time, you will also contribute to the growing Pulsar ecosystem.

Join the Pulsar community

In addition to adding connectors, you can also contribute in other ways, such as:

• Improve documentation! All connector documents [11] on StreamNative Hub are open source. You can submit changes (correct spelling errors, add instructions, etc.) via GitHub at any time.
• Report a bug.
• Review the PR.
• Give feedback on proposed features, improvements or designs.
• Propose new features.
• Answer the questions in GitHub issue or WeChat group.

Participate now

Click to view the quick start guide [12], hurry up and participate!

Related Reading

• StreamNative announces open source Function Mesh: simplifying complex streaming tasks on the cloud
• Product News | StreamNative Announces Open Source AMQP 1.0 Connector
• Product News | StreamNative Announces Open Source AWS SQS Connector

Reference link
[1] connector: https://pulsar.apache.org/docs/en/next/io-overview/
[2] StreamNative Hub: https://streamnative.io/en/blog/tech/2020-05-26-intro-to-hub
[3] AWS SQS connector: https://streamnative.io/en/blog/tech/2021-03-17-announcing-aws-sqs-connector-for-apache-pulsar
[4] AWS Lambda connector: https://github.com/streamnative/pulsar-io-aws-lambda
[5] AMQP1_0 connector: https://streamnative.io/en/blog/tech/2021-04-26-announcing-amqp10-connector-for-apache-pulsar
[6] IoTDB connector: https://github.com/streamnative/pulsar-io-iotdb
[7] StreamNative Hub: https://streamnative.io/en/blog/tech/2020-05-26-intro-to-hub
[8] Pulsar Connector Development Manual: https://github.com/streamnative/pulsar-io-template/blob/master/README.md
[9] Pulsar connector development manual: https://github.com/streamnative/pulsar-io-template/blob/master/README.md
[10] Pulsar connector development manual: https://github.com/streamnative/pulsar-io-template/blob/master/README.md
[11] connector document: https://github.com/streamnative/pulsar-hub/tree/master/connectors
[12] Quick Start Guide: https://github.com/streamnative/pulsar-io-template/blob/master/README.md

About StreamNative StreamNative is an open source basic software company formed by the founding team of Apache Pulsar, a top project of the Apache Software Foundation, to build a next-generation cloud-native batch-streaming fusion data platform around Pulsar. As an Apache Pulsar commercialization company, StreamNative focuses on open source ecology and community construction, and is committed to innovation in cutting-edge technology. The founding team members have worked for Yahoo, Twitter, Splunk, EMC and other well-known large companies.

Follow the StreamNative public account to learn about the surrounding ecology of Pulsar and StreamNative products