当前位置： 首页 > article >正文

DB-MongoDB-00002--Workload Generator for MongoDB

article 2025/9/10 18:38:20

##

DB-MongoDB-00002–Workload Generator for MongoDB

1、介绍

Workload Generator for MongoDB was designed to help MongoDB users effortlessly generate data and simulate workloads for both sharded and non-sharded clusters. The generated workloads include standard CRUD operations, reflecting real-world usage patterns of a MongoDB environment.

Additionally, the tool supports the creation of realistic workloads by incorporating all CRUD operations through a set of queries that simulate common usage scenarios. Users can also define custom queries to run against collections created by the tool, further enhancing its flexibility and applicability.

While the tool provides extensive configuration capabilities, it requires minimal setup — only basic connection details are needed. It is user-friendly, does not require compilation, and offers a high level of flexibility.

The application was developed and tested using Python 3. As such, Python 3 is recommended for optimal compatibility. If upgrading is not feasible, modifications to the scripts may be necessary to support older Python versions.

The tool is optimized to leverage as many available CPU cores as you wish and supports configuration of an arbitrary number of threads among, enabling high parallelism and making it ideal for generating large-scale workloads and conducting effective stress tests on MongoDB clusters.

一句话：另一个YCSB

2、安装
[root@mongodb01 ~]# python --version
Python 3.9.18
[root@mongodb01 ~]# pip --version
-bash: pip: command not found
[root@mongodb01 ~]# pip3 --version
-bash: pip3: command not found
Error: Unable to find a match: install
[root@mongodb01 ~]# dnf install -y python3-pip[root@mongodb01 ~]# pip3 install faker joblib pymongo
Collecting fakerDownloading faker-37.3.0-py3-none-any.whl (1.9 MB)|████████████████████████████████| 1.9 MB 382 kB/s
Collecting joblibDownloading joblib-1.5.0-py3-none-any.whl (307 kB)|████████████████████████████████| 307 kB 10.2 MB/s
Collecting pymongoDownloading pymongo-4.13.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (937 kB)|████████████████████████████████| 937 kB 564 kB/s
Collecting tzdataDownloading tzdata-2025.2-py2.py3-none-any.whl (347 kB)|████████████████████████████████| 347 kB 11.2 MB/s
Collecting dnspython<3.0.0,>=1.16.0Downloading dnspython-2.7.0-py3-none-any.whl (313 kB)|████████████████████████████████| 313 kB 1.9 MB/s
Installing collected packages: tzdata, dnspython, pymongo, joblib, faker
Successfully installed dnspython-2.7.0 faker-37.3.0 joblib-1.5.0 pymongo-4.13.0 tzdata-2025.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead:                     https://pip.pypa.io/warnings/venv[root@mongodb01 ~]# wget https://github.com/Percona-Lab/mongoDB-workload-generator/archive/refs/tags/0.1.zip
--2025-05-18 21:10:06--  https://github.com/Percona-Lab/mongoDB-workload-generator/archive/refs/tags/0.1.zip
Connecting to 192.168.5.18:7890... connected.
Proxy request sent, awaiting response... 302 Found
Location: https://codeload.github.com/Percona-Lab/mongoDB-workload-generator/zip/refs/tags/0.1 [following]
--2025-05-18 21:10:07--  https://codeload.github.com/Percona-Lab/mongoDB-workload-generator/zip/refs/tags/0.1
Connecting to 192.168.5.18:7890... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘0.1.zip’0.1.zip                                            [ <=>                                                                                                ]  23.48K  --.-KB/s    in 0.06s2025-05-18 21:10:09 (420 KB/s) - ‘0.1.zip’ saved [24040][root@mongodb01 ~]# mv 0.1.zip mongoDB-workload-generator-0.1.zip
[root@mongodb01 ~]# unzip mongoDB-workload-generator-0.1.zip
[root@mongodb01 ~]# cd mongoDB-workload-generator-0.1/
3、配置

基本配置就需要配置mongodbCreds.py这个文件就可以了

[root@mongodb01 mongoDB-workload-generator-0.1]# cat mongodbCreds.py
# Configuration for MongoDB connection
dbconfig = {"username": "login_here","password": "password_here","port": 27017,"hosts": ["mongos00","mongos01"],"serverSelectionTimeoutMS": 5000, # We need this to fail faster, otherwise the default is 30 seconds"connectTimeoutMS": 3000,  # Example timeout setting"maxPoolSize": 500, # Example pool setting# Leave replicaSet: None if connecting to mongos. Enter the appropriate replicaSet name if connecting to replicaSet instead of Mongos"replicaSet": None,"authSource": "admin",  # Adjust for authentication"tls": "false",  # Example tls setting
}[root@mongodb01 mongoDB-workload-generator-0.1]#
3.1、常见错误

如果连接信息配置错误或者在MongoDB没有开启认证的时候，就会出现这个错误

CRITICAL:root:Unable to connect to MongoDB, please make sure your configuration is correct.
Error: mongos01:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 3000.0ms, connectTimeoutMS: 3000.0ms),mongos00:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 3000.0ms, connectTimeoutMS: 3000.0ms), Timeout: 5.0s, Topology Description: <TopologyDescription id: 6829e152d0087848897bc0a8, topology_type: Unknown, servers: [<ServerDescription ('mongos00', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('mongos00:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 3000.0ms, connectTimeoutMS: 3000.0ms)')>, <ServerDescription ('mongos01', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('mongos01:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 3000.0ms, connectTimeoutMS: 3000.0ms)')>]>
3.2、配置无认证模式

如果MongoDB没有开启认证，如何连接数据库呢

3.2.1、配置mongodbCreds.py文件

旧文件

dbconfig = {"username": "login_here","password": "password_here","port": 27017,"hosts": ["mongos00","mongos01"],

新文件，把username/password删了，并且修改hosts

[root@mongodb01 mongoDB-workload-generator-0.1]# cat mongodbCreds.py
# Configuration for MongoDB connection
dbconfig = {"port": 27017,"hosts": ["mongodb01"],"serverSelectionTimeoutMS": 5000, # We need this to fail faster, otherwise the default is 30 seconds"connectTimeoutMS": 3000,  # Example timeout setting"maxPoolSize": 500, # Example pool setting# Leave replicaSet: None if connecting to mongos. Enter the appropriate replicaSet name if connecting to replicaSet instead of Mongos"replicaSet": None,"authSource": "admin",  # Adjust for authentication"tls": "false",  # Example tls setting
}
3.2.2、配置app.py文件

旧连接串

[root@mongodb01 mongoDB-workload-generator-0.1]# more app.py
# Start building the base connection URI
connection_uri = f"mongodb://{dbconfig['username']}:{dbconfig['password']}@{hosts}"

新连接串，这个地方还可以加localhost

[root@mongodb01 mongoDB-workload-generator-0.1]# more app.py
# Start building the base connection URI
#connection_uri = f"mongodb://{dbconfig['username']}:{dbconfig['password']}@{hosts}"
connection_uri = f"mongodb://{hosts}"
3.3、负载测试
[root@mongodb01 mongoDB-workload-generator-0.1]# ./mongodbWorkload.py
2025-05-18 21:38:28 - INFO - Configuring Workload
2025-05-18 21:38:28 - INFO -Duration: 60 seconds
CPUs: 1
Threads: (Per CPU: 4 | Total: 4)
Collections: 1
Configure Sharding: False
Insert batch size: 10
Optimized workload: False
Workload ratio: SELECTS: 60% | INSERTS: 10% | UPDATES: 20% | DELETES: 10%
Report frequency: 5 seconds
Report logfile: None===================================================================================================================Workload Started
===================================================================================================================2025-05-18 21:38:38 - INFO - AVG Operations last 5s (1 CPUs): 264.20 (SELECTS: 154.20, INSERTS: 31.00, UPDATES: 50.40, DELETES: 28.60)
2025-05-18 21:38:43 - INFO - AVG Operations last 5s (1 CPUs): 303.40 (SELECTS: 187.40, INSERTS: 30.80, UPDATES: 54.80, DELETES: 30.40)
2025-05-18 21:38:48 - INFO - AVG Operations last 5s (1 CPUs): 243.00 (SELECTS: 138.00, INSERTS: 30.00, UPDATES: 53.00, DELETES: 22.00)
2025-05-18 21:38:53 - INFO - AVG Operations last 5s (1 CPUs): 326.00 (SELECTS: 201.40, INSERTS: 29.40, UPDATES: 60.40, DELETES: 34.80)
2025-05-18 21:38:58 - INFO - AVG Operations last 5s (1 CPUs): 271.20 (SELECTS: 162.60, INSERTS: 30.00, UPDATES: 53.20, DELETES: 25.40)
2025-05-18 21:39:03 - INFO - AVG Operations last 5s (1 CPUs): 322.80 (SELECTS: 191.80, INSERTS: 29.40, UPDATES: 67.60, DELETES: 34.00)
2025-05-18 21:39:08 - INFO - AVG Operations last 5s (1 CPUs): 276.20 (SELECTS: 167.60, INSERTS: 28.80, UPDATES: 49.20, DELETES: 30.60)
2025-05-18 21:39:13 - INFO - AVG Operations last 5s (1 CPUs): 260.40 (SELECTS: 149.20, INSERTS: 30.60, UPDATES: 51.20, DELETES: 29.40)
2025-05-18 21:39:18 - INFO - AVG Operations last 5s (1 CPUs): 290.20 (SELECTS: 172.20, INSERTS: 28.40, UPDATES: 59.60, DELETES: 30.00)
2025-05-18 21:39:23 - INFO - AVG Operations last 5s (1 CPUs): 282.60 (SELECTS: 168.00, INSERTS: 29.20, UPDATES: 54.60, DELETES: 30.80)
2025-05-18 21:39:28 - INFO - AVG Operations last 5s (1 CPUs): 245.80 (SELECTS: 145.00, INSERTS: 29.00, UPDATES: 48.40, DELETES: 23.40)
2025-05-18 21:39:33 - INFO - AVG Operations last 5s (1 CPUs): 266.60 (SELECTS: 158.20, INSERTS: 26.40, UPDATES: 54.40, DELETES: 27.60)
2025-05-18 21:39:38 - INFO - AVG Operations last 5s (1 CPUs): 266.60 (SELECTS: 158.20, INSERTS: 26.40, UPDATES: 54.40, DELETES: 27.60)
2025-05-18 21:39:38 - INFO -
===================================================================================================================Workload Finished
===================================================================================================================2025-05-18 21:39:38 - INFO -
================================================================================
|                               Collection Stats                               |
================================================================================
|            Name         |     Sharded      |      Size      |    Documents   |
================================================================================
|         flights_1       |      False       |   115.40 MB    |      47407     |
================================================================================2025-05-18 21:39:38 - INFO -
===================================================================================================================Workload Stats (All CPUs Combined)
===================================================================================================================
Workload Runtime: 1.17 minutes
CPUs Used: 1
Total Operations: 16762 (SELECT: 9978, INSERT: 1765, UPDATE: 3284, DELETE: 1735)
AVG QPS: 238.79 (SELECTS: 142.15, INSERTS: 25.14, UPDATES: 46.78, DELETES: 24.72)
Documents Inserted: 17650, Matching Documents Selected: 9865, Documents Updated: 2995, Documents Deleted: 1573
===================================================================================================================
4、Ref

https://www.percona.com/blog/mongodb-benchmarking-made-effortless-with-mongodb-workload-generator/

https://github.com/Percona-Lab/mongoDB-workload-generator

http://www.lryc.cn/news/2384105.html

相关文章：

  • 3.8.1 利用RDD实现词频统计
  • Spring Ioc和Aop，Aop的原理和实现案例，JoinPoint，@Aspect，@Before，@AfterReturning
  • [解决conda创建新的虚拟环境没用python的问题]
  • 【优秀三方库研读】在 quill 开源库 LogMarcos.h 中知识点汇总及讲解
  • jvm安全点(五)openjdk17 c++源码垃圾回收之安全点阻塞状态线程在安全点同步中无需调用block函数的详细流程解析
  • C++ 中的 **常变量** 与 **宏变量** 比较
  • 【C++】控制台小游戏
  • 配合本专栏前端文章对应的后端文章——从模拟到展示：一步步搭建传感器数据交互系统
  • React中常用的钩子函数:
  • springboot IOC
  • java面试每日一背 day2
  • Ajax01-基础
  • （37）服务器增加ipv6配置方法
  • 生成树协议（STP）配置详解：避免网络环路的最佳实践
  • 面向 C 语言项目的系统化重构实战指南
  • 网络层——蚂蚁和信鸽的关系VS路由原理和相关配置
  • Python Pandas库简介及常见用法
  • 第十六届蓝桥杯复盘
  • 【已解决】HBuilder X编辑器在外接显示器或者4K显示器怎么界面变的好小问题
  • 直线型绝对值位移传感器：精准测量的科技利刃
  • 解决服务器重装之后vscode Remote-SSH无法连接的问题
  • AI 招聘系统科普：如何辨别真智能与伪自动化
  • Ansible模块——管理100台Linux的最佳实践
  • 从0开始学习大模型--Day09--langchain初步使用实战
  • C++中的菱形继承问题
  • 订单越来越到导致接口列表查询数据缓慢解决思路
  • word格式相关问题
  • 网络-MOXA设备基本操作
  • 飞桨paddle import fluid报错【已解决】
  • 测试工程师要如何开展单元测试