mongo监控及性能调优

转载请注明出处 http://www.paraller.com
原文排版地址 http://www.paraller.com/2016/10/22/mongo%E7%9B%91%E6%8E%A7%E5%8F%8A%E6%80%A7%E8%83%BD%E8%B0%83%E4%BC%98/

监控策略

mongo监控的策略主要有以下两种:

• 一系列的工具能够报告实时信息
• 数据库命令能够返回当前数据库的详细精确信息

mongostat

mongostat captures and returns the counts of database operations by type (e.g. insert, query, update, delete, etc.). These counts report on the load distribution on the server.

Use mongostat to understand the distribution of operation types and to inform capacity planning. See the mongostat manual for details.

root@db94dfee43b5:/# mongostat <sleeptime>
insert query update delete getmore command dirty used flushes vsize   res qrw arw net_in net_out conn                time
    *0    *0     *0     *0       0     2|0  0.0% 0.1%       0  270M 59.0M 0|0 0|0   160b   45.5k    2 Aug 29 08:48:07.749
    *0    *0     *0     *0       0     1|0  0.0% 0.1%       0  270M 59.0M 0|0 0|0   157b   44.8k    2 Aug 29 08:48:08.749

每个字段代表的意思

• inserts
  The number of objects inserted into the database per second. If followed by an asterisk (e.g. *), the datum refers to a replicated operation.
• query
  The number of query operations per second.
• update
  The number of update operations per second.
• delete
  The number of delete operations per second.
• getmore
  The number of get more (i.e. cursor batch) operations per second.
• command
  The number of commands per second. On slave and secondary systems, mongostat presents two values separated by a pipe character (e.g. |), in the form of local|replicated commands.
• flushes
  Changed in version 3.0.

For the WiredTiger Storage Engine, flushes refers to the number of WiredTiger checkpoints triggered between each polling interval.
For the MMAPv1 Storage Engine, flushes represents the number of fsync operations per second.

• dirty
  New in version 3.0.

Only for WiredTiger Storage Engine. The percentage of the WiredTiger cache with dirty bytes, calculated by wiredTiger.cache.tracked dirty bytes in the cache / wiredTiger.cache.maximum bytes configured.

• used
  New in version 3.0.

Only for WiredTiger Storage Engine. The percentage of the WiredTiger cache that is in use, calculated by wiredTiger.cache.bytes currently in the cache / wiredTiger.cache.maximum bytes configured.

• mapped
  Changed in version 3.0.

Only for MMAPv1 Storage Engine. The total amount of data mapped in megabytes. This is the total data size at the time of the last mongostat call.

• vsize
  The amount of virtual memory in megabytes used by the process at the time of the last mongostat call.
• res
  The amount of resident memory in megabytes used by the process at the time of the last mongostat call.
• faults
  Changed in version 3.0.

Only for MMAPv1 Storage Engine. The number of page faults per second.
Changed in version 2.1: Before version 2.1, this value was only provided for MongoDB instances running on Linux hosts.

• lr
  New in version 3.2.

Only for MMAPv1 Storage Engine. The percentage of read lock acquisitions that had to wait. mongostat displays lr|lw if a lock acquisition waited.

• lw
  New in version 3.2.

Only for MMAPv1 Storage Engine. The percentage of write lock acquisitions that had to wait. mongostat displays lr|lw if a lock acquisition waited.

• locked
  Changed in version 3.0: Only appears when mongostat runs against pre-3.0 versions of MongoDB instances.

The percent of time in a global write lock.

• idx miss
  Changed in version 3.0.

Only for MMAPv1 Storage Engine. The percent of index access attempts that required a page fault to load a btree node. This is a sampled value.

• qr
  The length of the queue of clients waiting to read data from the MongoDB instance.
• qw
  The length of the queue of clients waiting to write data from the MongoDB instance.
• ar
  The number of active clients performing read operations.
• aw
  The number of active clients performing write operations.
• netIn
  The amount of network traffic, in bytes, received by the MongoDB instance.

This includes traffic from mongostat itself.

• conn
  The total number of open connections.
• set
  The name, if applicable, of the replica set.

Add Fields to mongostat Output

root@db94dfee43b5:/# mongostat -O 'host,version,network.numRequests=network requests'
insert query update delete getmore command dirty used flushes vsize   res qrw arw net_in net_out conn                time            host version network requests
    *0    *0     *0     *0       0     2|0  0.0% 0.1%       0  271M 59.0M 0|0 0|0   158b   44.8k    2 Aug 29 09:45:11.850 localhost:27017   3.4.2             6319
    *0    *0     *0     *0       0     2|0  0.0% 0.1%       0  271M 59.0M 0|0 0|0   158b   44.9k    2 Aug 29 09:45:12.847 localhost:27017   3.4.2             6323
    *0    *0     *0     *0       0     1|0  0.0% 0.1%       0  271M 59.0M 0|0 0|0   157b   44.7k    2 Aug 29 09:45:13.849 localhost:27017   3.4.2             6327
    *0    *0     *0     *0       0     1|0  0.0% 0.1%       0  271M 59.0M 0|0 0|0   157b   44.7k    2 Aug 29 09:45:14.850 localhost:27017   3.4.2             6331

mongotop

racks and reports the current read and write activity of a MongoDB instance, and reports these statistics on a per collection basis.

Http Console

mongo:
  image: mongo:latest
  volumes:
    - /Users/zhidaliao/bak_dir/mongo:/data/db
  ports:
    - "27017:27017"
    - "1000:1000"
    - "28017:28017"
  command: ["mongod", "--rest"]

--httpinterface 还可以增加参数进行接口操作

在浏览器中访问 http://localhost:28017/ 可以查看实时信息，不推荐在生产环境中使用，安全性和稳定性。

命令行方式获取

db.currentOp()

显示当前的用户操作。

db.serverStatus()

详细的Mongo服务器信息。主要关注的点

• connections 当前连接和可用连接数，听过一个同行介绍过，mongodb最大处理到2000个连接就不行了（要根据你的机器性能和业务来设定），所以设大了没意义。设个合理值的话，到达这个值mongodb就拒绝新的连接请求，避免被太多的连接拖垮。
• indexCounters:btree:misses 索引的不命中数，和hits的比例高就要考虑索引是否正确建立。你看我的”missRatio” : 3.543930204420982e-7，很健康吧。所以miss率在mongostat里面也可以看

db.stats()

> db.stats()
{
    "db" : "yea",
    "collections" : 31,
    "views" : 0,
    "objects" : 45827,
    "avgObjSize" : 345.1348768193423,
    "dataSize" : 15816496,
    "storageSize" : 7372800,
    "numExtents" : 0,
    "indexes" : 54,
    "indexSize" : 2985984,
    "ok" : 1
}

db.collection.stats()

mongo数据库的集合详细信息。

replSetGetStatus

The replSetGetStatus command (rs.status() from the shell) returns an overview of your replica set’s status.

profiler

类似于MySQL的slow log, MongoDB可以监控所有慢的以及不慢的查询。
Profiler默认是关闭的，你可以选择全部开启，或者有慢查询的时候开启。

> use test
switched to db test
> db.setProfilingLevel(2);
{"was" : 0 , "slowms" : 100, "ok" : 1} 
> db.getProfilingLevel()
2

slowms:代表输出 操作大于100毫秒的语句

To change the threshold, pass two parameters to the db.setProfilingLevel() helper in the mongo shell. The first parameter sets the profiling level for the current database, and the second sets the default slow operation threshold for the entire mongod instance.

设置0代表当前的数据库禁止profiler功能，但是这个Mongo实例下的数据库，只要等级是1的，都会使用20ms的配置

db.setProfilingLevel(0,20)

查看Profile日志

> db.system.profile.find().sort({$natural:-1})
{"ts" : "Thu Jan 29 2009 15:19:32 GMT-0500 (EST)" , "info" :
"query test.$cmd ntoreturn:1 reslen:66 nscanned:0 query: { profile: 2 } nreturned:1 bytes:50" ,
"millis" : 0} ...

3个字段的意义

• ts：时间戳
• info：具体的操作
• millis：操作所花时间，毫秒

Database Profiler¶

第三方监测工具

MongoDB Monitoring Service : MongoDB Monitoring Service(MMS)是Mongodb厂商提供的监控服务，可以在网页和Android客户端上监控你的MongoDB状况。

以及其他第三方链接：
Third Party Tools

性能调优

场景：现实blog的首页-我们希望现实最近发布的10条posts。ts为时间字段。

db.posts.find().sort({ts:-1}); 

创建索引

db.posts.ensureIndex({ts:1});

数据库就可以基于索引信息排序，不会直接查看每个document

限定结果

MongoDB游标返回一组document，我们叫这个为chunks.

articles = db.posts.find().sort({ts:-1}).limit(10); 

查询特定的字段

db.oauth_clients.find({},{clientId:1,clientSecret:1})

如果你选择了要查询的字段，那么返回的就是部分对象。这个对象并不能直接进行更新

读写分离

如果读写都在主节点的话，从节点就一直处在空置状态，这是一种浪费。对于报表或者搜索这种读操作来说完全可以在从节点实现，因此要做的是在 connection string 中设置成 secondarypreferred。

Capped Collections

Capped Collections¶

Capped collections 是固定大小的集合，基于插入的顺序，支持高速的读写文档的操作， Capped collections 工作机制于循环缓存类似: 一旦一个集合填充了他被分配的空间，对于新插入的文档他会覆盖旧文档的空间

表现形式

Insertion Order

Capped collections 保证插入顺序.所以查询的时候不需要根据索引返回结果，而是通过插入顺序返回,没有索引， capped collections 可以支持高速插入操作，类似于数据库。

自动删除旧文档

为了给新文档创建空间， capped collections 在集合中自动删除老旧的文档，不需要执行额外的脚本和操作。

举个例子，oplog.rs collection 在副本形式下保存的操作日志就是使用capped collection。以下的潜在的场景：

• 在高速系统中储存日志，或者在队列形式的事件机制中保持顺序。
• capped collections缓存少量的数据，因为缓存的读操作别写操作更频繁，你可以确保你的集合总是保存在你的工作区（比如内存），或者你能接受一些需要保持顺序的写操作。

_id Index

Capped collections have an _id field and an index on the _id field by default.

优缺点

Updates

如果你准备更新文档，创建了索引所以这些更新操作不需要 经过集合扫描。

Document Size

Changed in version 3.2.

文档大小是固定的，如果 更新和替换操作，改变了文档大小，操作将会失败

Document Deletion

你不能删除文档，只能使用drop()方法删掉集合并重新创建新的内容。

Sharding 分片

capped collection 不支持分片操作

Query Efficiency

使用自然的排序去获取最近插入集合的元素是更有效率的，类似诶于在日志文件中tail 的命令

函数 $out

$out 管道操作对capped collection无效

程序操作

Create a Capped Collection 创建

db.createCollection()命令创建，但是要指定参数预分配空间，单位字节。

db.createCollection( "log", { capped: true, size: 100000 } )

如果空间小于或者等于4096, 集合将会划分出 4096 bytes. MongoDB规划的大小以256的倍数字节扩增

另外，我们可以指定max参数，用来限制最大的文档数量(不是集合的空间大小，而是数量)

db.createCollection("log", { capped : true, size : 5242880, max : 5000 } )

Query a Capped Collection 查询

• 当你使用find函数没有指定排序的时候，mongodb会保证按照插入顺序返回。如果要按照插入顺序的逆序返回结果，find()函数 要紧接着 sort()函数，使用natural参数指定为-1

db.cappedCollection.find().sort( { $natural: -1 } )

• 检查是否是Capped 集合

db.collection.isCapped()

• Convert a Collection to Capped ，普通集合转换成 Capped类型

db.runCommand({"convertToCapped": "mycoll", size: 100000});

This command obtains a global write lock and will block other operations until it has completed.

• TTL

Setting TTL. These indexes allow you to expire and remove data from normal collections using a special type, based on the value of a date-typed field and a TTL value for the index.
TTL Collections are not compatible with capped collections.

• Tailable Cursor

You can use a tailable cursor with capped collections. Similar to the Unix tail -f command, the tailable cursor “tails” the end of a capped collection. As new documents are inserted into the capped collection, you can use the tailable cursor to continue retrieving documents.

IMPORTANT
size参数总是必须的，即使你指定了max数量，mongoDb会移除更旧的文档，如果你的集合到达了指定的最大空间限制，在最大的条数限制条数之前。

参考网站

MongoDB 优化

Monitoring for MongoDB¶