部署
本节介绍生产就绪的部署解决方案。 一般来说,我们正在部署一个没有单点故障的弹性RocketMQ集群。
条件
在开始本节之前,请确保您已阅读快速入门部分,并熟悉RocketMQ的核心概念和组件。
Production-ready Deployment
Name Server
为了确保一个实例崩溃时群集仍可以正常工作,建议使用两个或更多名称服务器实例。 只要有一个名称服务器实例处于活动状态,整个集群就会保持服务状态。
名称服务器(Name Server)遵循无共享设计范例。 经纪商将心跳数据发送到所有名称服务器。 生产者和消费者可以在发送/消费消息时从任何可用的名称服务器查询元数据。
Broker
经纪人可根据其角色分为两类:主人和奴隶。 主代理提供RW访问,而从代理只接受读访问。
要部署没有单点故障的高可用性RocketMQ集群,应该部署一系列代理集。 代理集包含一个brokerId设置为0的主设备和几个具有非零brokerID的从设备。 所有在一个集合中的经纪人都有相同的brokerName。 在严重的情况下,我们应该至少有一个经纪人中的两个经纪人。 每个主题都驻留在两个或更多个经纪人中。
Configuration
部署RocketMQ集群时,推荐配置如下:
Broker configuration
| 属性名 | 默认值 | 详情 |
|---|---|---|
| listenPort | 10911 | listen port for client |
| namesrvAddr | null | name server address |
| brokerIP1 | InetAddress for network interface | Should be configured if having multiple addresses |
| brokerName | null | broker name |
| brokerClusterName | DefaultCluster | this broker belongs to which cluster |
| brokerId | 0 | broker id, 0 means master, positive integers mean slave |
| storePathCommitLog | $HOME/store/commitlog/ | file path for commit log |
| storePathConsumerQueue | $HOME/store/consumequeue/ | file path for consume queue |
| mapedFileSizeCommitLog | 1024 * 1024 * 1024(1G) | mapped file size for commit log |
| deleteWhen | 04 | When to delete the commitlog which is out of the reserve time |
| fileReserverdTime | 72 | The number of hours to keep a commitlog before deleting it |
| brokerRole | ASYNC_MASTER | SYNC_MASTER/ASYNC_MASTER/SLVAE |
| flushDiskType | ASYNC_FLUSH | {SYNC_FLUSH/ASYNC_FLUSH}. Broker of SYNC_FLUSH mode flushes each message onto disk before acknowledging producer. Broker of ASYNC_FLUSH mode, on the other hand, takes advantage of group-committing, achieving better performance. |
命令行管理工具(CLI Admin Tool)
RocketMQ提供了一个CLI(命令行界面)管理工具带来查询,管理和诊断各种问题。
How To Get
管理工具随RocketMQ一起发布。 你可以下载一个预先构建的二进制版本,或者自己编译源代码,你已经拥有它了。
如果您需要源代码,rocketmq-tools模块将包含其源代码。
Deployment
ON THIS PAGE
This section introduces production-ready deployment solution. Generally speaking, we are deploying a resilient RocketMQ cluster having no single point of failure.
Prerequisite
Before starting this section, make sure you have read Quick Start section, and are farmiliar with the core concepts and components of RocketMQ.
Production-ready Deployment
Name Server
To ensure the cluster can still function when one instance crashes, two or more name server instances are recommended. As long as there is one name server instance alive, the whole cluster remains in service.
Name server follows the share-nothing design paradigm. Brokers send heartbeat data to all name servers. Producers and consumers can query meta data from any of name servers available while sending / consuming messages.
Broker
Brokers can be divided into two categories according to their roles: master and slave. Master brokers provide RW access while slave brokers only accept read access.
To deploy a high-availability RocketMQ cluster with no single point of failure, a series of broker sets should be deployed. A broker set contains one master with brokerId set to 0 and several slaves with non-zero brokerIDs. All of the brokers in one set have the same brokerName. In serious scenarios, we should have at least two brokers in one broker set. Each topic resides in two or more brokers.
Configuration
When deploying a RocketMQ cluster, recommended configuration is listed below:
Broker configuration
| Property Name | Default value | Details |
|---|---|---|
| listenPort | 10911 | listen port for client |
| namesrvAddr | null | name server address |
| brokerIP1 | InetAddress for network interface | Should be configured if having multiple addresses |
| brokerName | null | broker name |
| brokerClusterName | DefaultCluster | this broker belongs to which cluster |
| brokerId | 0 | broker id, 0 means master, positive integers mean slave |
| storePathCommitLog | $HOME/store/commitlog/ | file path for commit log |
| storePathConsumerQueue | $HOME/store/consumequeue/ | file path for consume queue |
| mapedFileSizeCommitLog | 1024 * 1024 * 1024(1G) | mapped file size for commit log |
| deleteWhen | 04 | When to delete the commitlog which is out of the reserve time |
| fileReserverdTime | 72 | The number of hours to keep a commitlog before deleting it |
| brokerRole | ASYNC_MASTER | SYNC_MASTER/ASYNC_MASTER/SLVAE |
| flushDiskType | ASYNC_FLUSH | {SYNC_FLUSH/ASYNC_FLUSH}. Broker of SYNC_FLUSH mode flushes each message onto disk before acknowledging producer. Broker of ASYNC_FLUSH mode, on the other hand, takes advantage of group-committing, achieving better performance. |
CLI Admin Tool
RocketMQ provides a CLI(command-line interface) admin tool belt to query, manage and diagnose various issues.
How To Get
The admin tool is shipped along with RocketMQ. Either you download a pre-built binary version or build from source by yourself, you already have it.
In case you need the source code, the rocketmq-tools module contains its source code.
How to use
管理工具非常易于使用。 在这里,为了演示目的,假设*nix环境。
将目录更改为$ {PACKAGE} / bin,命令bash mqadmin,您应该看到以下帮助菜单:
The most commonly used mqadmin commands are:
updateTopic Update or create topic
deleteTopic Delete topic from broker and NameServer
updateSubGroup Update or create subscription group
deleteSubGroup Delete subscription group from broker
updateBrokerConfig Update broker's config
updateTopicPerm Update topic perm
topicRoute Examine topic route info
topicStatus Examine topic Status info
topicClusterList get cluster info for topic
brokerStatus Fetch broker runtime status data
queryMsgById Query Message by Id
queryMsgByKey Query Message by Key
queryMsgByUniqueKey Query Message by Unique key
queryMsgByOffset Query Message by offset
queryMsgByUniqueKey Query Message by Unique key
printMsg Print Message Detail
sendMsgStatus Send msg to broker
brokerConsumeStats Fetch broker consume stats data
producerConnection Query producer's socket connection and client version
consumerConnection Query consumer's socket connection, client version and subscription
consumerProgress Query consumers's progress, speed
consumerStatus Query consumer's internal data structure
cloneGroupOffset Clone offset from other group
clusterList List all of clusters
topicList Fetch all topic list from name server
updateKvConfig Create or update KV config
deleteKvConfig Delete KV config
wipeWritePerm Wipe write perm of broker in all name server
resetOffsetByTime Reset consumer offset by timestamp(without client restart)
updateOrderConf Create or update or delete order conf
cleanExpiredCQ Clean expired ConsumeQueue on broker.
cleanUnusedTopic Clean unused topic on broker
startMonitoring Start Monitoring
statsAll Topic and Consumer tps stats
syncDocs Synchronize wiki and issue to github.com
allocateMQ Allocate MQ
checkMsgSendRT Check message send response time
clusterRT List All clusters Message Send RT
有关特定命令的更多信息,请参阅“mqadmin help”。 如果你想得到更多关于像“clusterList”这样的特定命令的信息,只需键入bash mqadmin help clusterList,你将看到:
usage: mqadmin clusterList [-h] [-i <arg>] [-m] [-n <arg>]
-h,--help Print help
-i,--interval <arg> specify intervals numbers, it is in seconds
-m,--moreStats Print more stats
-n,--namesrvAddr <arg> Name server address list, eg: 192.168.0.1:9876;192.168.0.2:9876
复制模式
为了确保没有成功发布的消息将会丢失,RocketMQ提供了复制模式,通过两种复制方式获得更强的耐用性和更高的可用性:同步和异步。
复制: 同步 / 异步 Broker
与许多复制系统一样,同步代理等待,直到提交日志被复制到从属服务器之后才进行确认。 相反,异步代理在主服务器上处理消息后立即返回。
How to configure
RocketMQ在conf文件夹中的发行版本有三种预置的配置供您参考:
2m-2s-sync
2m-2s-async
2m-noslave
注意:所有配置使用ASYNC_FLUSH。
部署
以2m-2s-sync的部署为例。 首先,启动两个名称服务器,如“快速入门”部分所示。 假设他们的IP是192.168.0.2和192.168.0.3。
然后启动代理(假设二进制RocketMQ在/ home / rocketmq / dist)
>cd /home/rocketmq/dist/bin
>bash mqbroker -c ../conf/2m-2s-sync/broker-a.properties -n 192.168.0.2:9876,192.168.0.3:9876
>bash mqbroker -c ../conf/2m-2s-sync/broker-a-s.properties -n 192.168.0.2:9876,192.168.0.3:9876
>bash mqbroker -c ../conf/2m-2s-sync/broker-b.properties -n 192.168.0.2:9876,192.168.0.3:9876
>bash mqbroker -c ../conf/2m-2s-sync/broker-b-s.properties -n 192.168.0.2:9876,192.168.0.3:9876
How to verify
Execute the following command to verify according to the CLI section:
> bash mqadmin clusterlist