当前位置： 首页 > news >正文

论，物联网日志系统架构如何设计？

news 2025/8/15 7:58:51

使用双引擎：

  • mongodb存储用户行为日志
  • Elasticsearch存储设备状态日志、app系统日志（打印调试）。

在这里插入图片描述

@startuml
!theme plain
skinparam componentStyle rectangle
skinparam nodesep 30
skinparam ranksep 30
skinparam defaultFontName Helvetica
skinparam defaultFontSize 14
skinparam classFontSize 14
skinparam BackgroundColor #F5F5F5title 物联网日志系统架构left to right direction' ====== 设备层 ======
package "设备层" #D5E8D4 {rectangle "4G无线终端" as device <<(D,#55AA55)>> {rectangle "MCU" as mcu #A9D18Erectangle "4G模组" as modem #A9D18E}rectangle "执行器" as actuators {rectangle "LED控制器" as led #A9D18Erectangle "指纹识别" as finger #A9D18Erectangle "音频模块" as audio #A9D18Erectangle "窗帘" as curtain #A9D18Erectangle "空调" as airConditioning #A9D18E}mcu --> led : GPIOmcu --> finger : uartmcu --> audio : I2Smcu --> modem : AT指令mcu --> airConditioning: 红外发射管mcu --> curtain: pwm
}' ====== 通信层 ======
package "通信层" #FFF2CC {rectangle "MQTT Server" as mqtt <<(M,#FF7700)>> {rectangle "TCP接入" as tcp #FFD966rectangle "主题路由" as router #FFD966rectangle "消息队列" as queue #FFD966rectangle "认证中心" as auth #FFD966tcp --> router : 消息分发router --> queue : 持久化auth --> tcp : 设备认证}
}' ====== 服务层 ======
package "服务层" #DAE8FC {rectangle "RESTful API服务" as api_svc <<(S,#4472C4)>> {rectangle "设备管理" as device_mgr #BDD7EErectangle "指令处理" as cmd_handler #BDD7EErectangle "会话管理" as session_mgr #BDD7EE}rectangle "行为日志服务" as action_svc #BDD7EErectangle "日志采集器" as log_agent #BDD7EE
}' ====== 数据存储层 ====== [重点补充]
package "数据存储层" #E2F0D9 {database "MongoDB集群" as mongo <<(DB,#70AD47)>> {rectangle "用户行为日志" as action_logs #A9D18Erectangle "设备元数据" as device_meta #A9D18Erectangle "用户配置" as user_config #A9D18E}database "Elasticsearch集群" as es <<(DB,#548235)>> {rectangle "应用日志索引" as app_logs #A9D18Erectangle "设备状态索引" as device_status #A9D18Erectangle "运行指标" as metrics #A9D18E}database "Redis缓存" as redis <<(DB,#FF6600)>> {rectangle "会话缓存" as sessions #FFCC99rectangle "设备状态缓存" as device_cache #FFCC99rectangle "消息队列" as msg_queue #FFCC99}database "对象存储" as s3 <<(DB,#0070C0)>> {rectangle "固件文件" as firmware #8DB4E2rectangle "音频资源" as audio_files #8DB4E2rectangle "日志备份" as log_backup #8DB4E2}
}' ====== 客户端层 ======
package "客户端层" #E1D5E7 {rectangle "移动APP" as app <<(C,#9666CC)>> {rectangle "控制界面" as ctrl_ui #D5C6E0rectangle "实时监控" as monitor #D5C6E0}
}' ====== 监控分析层 ======
package "分析层" #FCE4D6 {rectangle "Kibana" as kibana <<(V,#339999)>> #FCE4D6rectangle "Grafana" as grafana <<(V,#339999)>> #FCE4D6rectangle "管理后台" as admin <<(V,#990099)>> #FCE4D6
}' ====== 完整连接关系 ======
' 设备通信
modem --> mqtt : 4G TCP\n<&signal>publish
mqtt --> modem : subscribe\n<&message>' 服务间通信
mqtt --> api_svc : 内部转发\n(消息总线)
api_svc --> mqtt : 发布指令\n<&message>' 数据处理流
api_svc --> action_svc : 用户行为事件\n<&document>
action_svc --> mongo : 写入行为日志\n<&data>
api_svc --> log_agent : 输出应用日志\n<&log>
log_agent --> es : 发送日志\n<&transfer>
api_svc --> redis : 缓存会话\n<&save>
api_svc --> s3 : 固件下载\n<&download>' 客户端连接
app --> api_svc : HTTPS/WebSocket\n<&phone>
api_svc --> app : 实时状态推送\n<&notification>' 监控分析
kibana --> es : 日志分析\n<&search>
grafana --> es : 仪表盘\n<&dashboard>
grafana --> redis : 实时监控\n<&eye>
admin --> mongo : 行为分析\n<&graph>
admin --> s3 : 资源管理\n<&folder>' ====== 数据存储层内部关系 ======
es --> s3 : 冷数据归档\n<&archive>
redis --> mongo : 持久化同步\n<&sync>
mongo --> s3 : 备份导出\n<&export>' ====== 图例说明 ======
legend right<b>存储层组件</b>|<#70AD47> MongoDB | 行为日志/元数据 ||<#548235> Elasticsearch | 日志/状态数据 ||<#FF6600> Redis | 缓存/消息队列 ||<#0070C0> 对象存储 | 文件/备份 |
endlegend@enduml

Mongodb集群架构：

@startuml
!theme plain
skinparam componentStyle rectangle
skinparam nodesep 30
skinparam ranksep 30
skinparam defaultFontName Helveticatitle MongoDB 分片副本集群package "分片集群" {database "分片1" as shard1 {component "副本集" as rs1 {node "主节点" as primary1node "从节点" as secondary1node "从节点" as secondary2primary1 --> secondary1 : 复制primary1 --> secondary2 : 复制}}database "分片2" as shard2 {component "副本集" as rs2 {node "主节点" as primary2node "从节点" as secondary3node "从节点" as secondary4}}database "配置服务器" as config_servers {node "cfg1"node "cfg2"node "cfg3"}
}package "查询路由器" {[mongos1] as mongos1[mongos2] as mongos2
}package "客户端应用" {[设备控制服务] as api_svc[行为分析服务] as analytics_svc
}' ==== 连接关系 ====
api_svc --> mongos1 : 写入请求
analytics_svc --> mongos2 : 查询请求mongos1 --> config_servers : 获取元数据
mongos1 --> shard1 : 路由请求
mongos1 --> shard2 : 路由请求config_servers --> shard1 : 管理分片
config_servers --> shard2 : 管理分片' ==== 分片策略 ====
note bottom of shard1**分片键**：sh.shardCollection("iot.actions", {"user_id":1, "timestamp":-1})
end note' ==== 性能指标 ====
legend left<b>性能指标</b>| 写入吞吐 | 10,000 ops/s || 查询延迟 | <50ms p95 || 可用性 | 99.99% |
end legend
@enduml

mongodb文档格式：

@startuml
!theme plain
skinparam componentStyle rectangle
skinparam nodesep 20
skinparam ranksep 20
skinparam defaultFontName Helvetica
skinparam BackgroundColor #F5F5F5title MongoDB 集群数据结构package "MongoDB集群" {component "用户行为日志" as action_logs {[用户操作记录][设备交互事件][执行结果]}component "设备元数据" as device_meta {[设备型号][固件版本][硬件配置]}component "用户配置" as user_config {[偏好设置][通知规则][自动化场景]}
}' ==== 数据结构注释 ====
note top of action_logs**数据结构**：{"user_id": ObjectId,"device_id": String,"action": "led_on/vibrate/play_sound","params": {"brightness":80},"timestamp": ISODate()}
end notenote right of device_meta**数据结构**：{"device_id": "DEV_2024X","model": "SmartController","firmware": "v2.1.5","capabilities": ["led","vibrate","audio"]}
end notenote bottom of user_config**数据结构**：{"user_id": ObjectId,"led_brightness": 70,"auto_off": {"enabled":true,"delay_min":30},"alerts": ["vibration_detected"]}
end note' ==== 关系说明 ====
action_logs --> device_meta : 关联\ndevice_id
user_config --> action_logs : 基于用户行为\n的配置@enduml

ES集群架构：

@startuml
!theme plain
skinparam componentStyle rectangle
skinparam nodesep 20
skinparam ranksep 20
skinparam defaultFontName Helvetica
skinparam BackgroundColor #F5F5F5
skinparam rectangle {BackgroundColor WhiteBorderColor Black
}title Elasticsearch 集群架构（热-暖-冷三阶设计）left to right direction' ====== 节点角色 ======
package "热层（SSD）" #FFEBEE {rectangle "热节点1" as hot1 #FF5252rectangle "热节点2" as hot2 #FF5252rectangle "热节点3" as hot3 #FF5252hot1 -[hidden]- hot2hot2 -[hidden]- hot3
}package "暖层（HDD）" #FFF8E1 {rectangle "暖节点1" as warm1 #FFC107rectangle "暖节点2" as warm2 #FFC107
}package "冷层（对象存储）" #E3F2FD {rectangle "对象存储" as cold #2196F3
}' ====== 数据分片 ======
rectangle "分片P1主" as shard_p1 #4CAF50
rectangle "分片P1副" as shard_p1r #81C784
rectangle "分片P2主" as shard_p2 #4CAF50
rectangle "分片P2副" as shard_p2r #81C784' ====== 分片分配 ======
shard_p1 --> hot1 : 热层
shard_p1r --> hot2 : 热层
shard_p2 --> warm1 : 暖层
shard_p2r --> warm2 : 暖层' ====== 协调层 ======
rectangle "协调节点1" as coord1 #9C27B0
rectangle "协调节点2" as coord2 #9C27B0' ====== 管理组件 ======
rectangle "Kibana" as kibana #4CAF50
rectangle "Logstash" as logstash #FF9800
rectangle "Filebeat" as filebeat #795548' ====== 数据流 ======
filebeat -> logstash : 发送日志
logstash -> coord1 : 写入数据
coord1 --> shard_p1 : 路由到热分片
coord1 --> shard_p2 : 路由到暖分片' ====== 数据迁移流 ======
shard_p1 --> shard_p2 : 热→暖迁移\n(3天后)
shard_p2 --> cold : 暖→冷迁移\n(30天后)' ====== 查询流 ======
kibana -> coord2 : 查询请求
coord2 --> shard_p1 : 查询热数据
coord2 --> shard_p2 : 查询暖数据
coord2 --> cold : 查询冷数据' ====== 集群管理 ======
rectangle "主节点" as master #795548
master --> hot1 : 管理
master --> warm1 : 管理' ====== 索引生命周期 ======
note right of warm1**ILM策略**：phases: {hot: { actions: { rollover: "50gb" } },warm: { min_age: "3d", allocate: "hdd" },cold: { min_age: "30d", snapshot: "s3" }}
end note' ====== 图例说明 ======
legend bottom<color:#FF5252>▉ 热节点</color> : SSD存储, 最新数据<color:#FFC107>▉ 暖节点</color> : HDD存储, 近期数据<color:#2196F3>▉ 冷存储</color> : 对象存储, 归档数据<color:#4CAF50>▉ 主分片</color> <color:#81C784>▉ 副本分片</color><color:#9C27B0>▉ 协调节点</color>
end legend@enduml
http://www.lryc.cn/news/620711.html

相关文章：

  • AI增强SEO关键词表现
  • Postman 平替 技术解析：架构优势与实战指南
  • 考研408《计算机组成原理》复习笔记，第五章(2)——CPU指令执行过程
  • 使用 Docker 部署 PostgreSQL
  • 考研408《计算机组成原理》复习笔记，第四章(3)——指令集、汇编语言
  • Java设计模式之《策略模式》
  • Effective C++ 条款41：理解隐式接口和编译期多态
  • 应用系统连达梦数据库报“服务器模式不匹配”的根源与修复方案
  • 使用colmap自制3DGaussian_Splatting数据集
  • BotCash：GPT-5发布观察 工程优化的进步，还是技术突破的瓶颈？
  • GoLand 项目从 0 到 1：第八天 ——GORM 命名策略陷阱与 Go 项目启动慢问题攻坚
  • 通过机器学习框架实现Android手写识别输入功能
  • Spring Boot 3中JWT密钥安全存储方案
  • Python训练营打卡Day32-神经网络的训练
  • 【Golang】Golang内存泄漏问题排查（二）
  • OpenCv（三）——图像平滑处理
  • 8. 函数简介
  • OpenCV中对图像进行平滑处理的4种方式
  • HarmonyOS AI辅助编程工具（CodeGenie）智慧调优
  • 力扣（LeetCode) ——225 用队列实现栈(C语言）
  • 信息vs知识：人类学习与AI规则提取
  • 异步编程的 8 种实现方式：疑难点与注意事项解析
  • 《疯狂Java讲义（第3版）》学习笔记ch4
  • 安全加固4（K8S最小化微服务安全）
  • C++ 中的元控制流与概念化类型擦除
  • Elasticsearch 中如何配置 RBAC 权限-实现安全的访问控制
  • 论郑和下西洋元素融入课件编辑器的意义与影响​
  • 智能门锁：安全与便捷的现代家居入口
  • UE小：编辑器模式下「窗口/鼠标不在焦点」时仍保持高帧率
  • UE5配置MRQ编解码器输出MP4视频