System Monitoring with Node Red
This post will cover the use of Node Red as a system monitoring tool. Why is this useful? It gives system status at a glance. With some advanced configuration, Node Red can provide trend analytics or automated responses.
Here is a comparison of two methods to gather metrics from systems and present them in Node Red.
The first method is with the SNMP node. This is centralized because Node Red is the single point that is gathering metrics from multiple end points. The types of metrics available are limited by what each end point supports with their OIDs. Another downside of the SNMP standard is it is being phased out. Reasons for this is from it being a network service that must run on each end point. The problem here is that not only does the polling device have access, so do other network end points. Firewalls and ACLs can be put in place to address this issue but they add more management overhead. Fewer devices support it and Windows has discontinued it. This post shows the process on how to install it on Windows 10 or 11, https://theitbros.com/snmp-service-on-windows-10/
The second method is with the MQTT node. This is de-centralized because Node Red can subscribe to any number of MQTT brokers that have published content from various IoT devices. Because of this it provides extensible metrics. If something can produce data, that data can become a metric in Node Red. There is no need to allow access to each end point. Those end points publish their data to an available broker, so they only need to have egress network access.
Here is what a typical SNMP flow would look like.
It consists of the following nodes
Inject – this repeats a message every 15 seconds.
SNMP – this queries a host running a snmp service for the value of an OID, 1.3.6.1.2.1.25.1.1.0 being uptime
Change – this changes the message payload to the extract the value from the snmp query, msg.payload to msg.payload[0].value
Function – this converts the message payload from milliseconds to seconds.
Function (Seconds to DD:HH:MM:SS) – this converts the message payload from seconds into a formatted Day, Hours, Minutes, and Seconds format.
Text – this presents the value on the Node Red Web UI
Here is the JavaScript of the SNMP flow.
[
{
"id": "b1a92a500b15a762",
"type": "inject",
"z": "c4f01f6d02da6930",
"name": "Timer (15 Seconds)",
"props": [
{
"p": "payload"
}
],
"repeat": "15",
"crontab": "",
"once": false,
"onceDelay": 0.1,
"topic": "",
"payloadType": "str",
"x": 340,
"y": 200,
"wires": [
[
"77ce38016a91b29d"
]
]
},
{
"id": "77ce38016a91b29d",
"type": "snmp",
"z": "c4f01f6d02da6930",
"host": "IP Address Of SNMP End Node",
"community": "public",
"version": "1",
"oids": "1.3.6.1.2.1.25.1.1.0\n",
"timeout": "1",
"name": "Uptime",
"x": 580,
"y": 200,
"wires": [
[
"220992a38a5609b6"
]
]
},
{
"id": "220992a38a5609b6",
"type": "change",
"z": "c4f01f6d02da6930",
"name": "",
"rules": [
{
"t": "set",
"p": "payload",
"pt": "msg",
"to": "payload[0].value",
"tot": "msg"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 800,
"y": 200,
"wires": [
[
"6e8b0c3ae9399993"
]
]
},
{
"id": "6e8b0c3ae9399993",
"type": "function",
"z": "c4f01f6d02da6930",
"name": "Convert to Seconds",
"func": "var millis = msg.payload;\nvar S = millis / 100;\nvar seconds = Math.floor(S);\nmsg.payload = seconds;\nreturn msg;\n\n",
"outputs": 1,
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 1030,
"y": 200,
"wires": [
[
"201f23e773ba9814"
]
]
},
{
"id": "201f23e773ba9814",
"type": "function",
"z": "c4f01f6d02da6930",
"name": "Seconds to DD:HH:MM:SS",
"func": " var totalNumberOfSeconds = msg.payload;\n var days = parseInt( totalNumberOfSeconds / 86400 );\n var hours = parseInt (( totalNumberOfSeconds - ( days * 86400 )) / 3600 );\n var minutes = parseInt ((totalNumberOfSeconds - ((hours * 3600)+( days * 86400 ))) / 60 );\n var seconds = parseInt(totalNumberOfSeconds - ((hours * 3600) + (minutes * 60)+( days * 86400 )));\n var result = (days < 10 ? \"0\" + days : days) + \" Days \" + (hours < 10 ? \"0\" + hours : hours) + \":\" + (minutes < 10 ? \"0\" + minutes : minutes) + \":\" + (seconds < 10 ? \"0\" + seconds : seconds);\n msg.payload=result;\n return msg;",
"outputs": 1,
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 1320,
"y": 200,
"wires": [
[
"291d964ba00e8c83"
]
]
},
{
"id": "291d964ba00e8c83",
"type": "ui_text",
"z": "c4f01f6d02da6930",
"group": "e867fdf0c079680f",
"order": 6,
"width": 0,
"height": 0,
"name": "",
"label": "Uptime",
"format": "{{msg.payload}}",
"layout": "row-spread",
"className": "",
"x": 1560,
"y": 200,
"wires": []
},
{
"id": "e867fdf0c079680f",
"type": "ui_group",
"name": "SNMP End Node",
"tab": "4afc27d31e945b9d",
"order": 1,
"disp": true,
"width": "7",
"collapse": false,
"className": ""
},
{
"id": "4afc27d31e945b9d",
"type": "ui_tab",
"name": "Systems",
"icon": "dashboard",
"order": 6,
"disabled": false,
"hidden": false
}
]
Here is what a typical MQTT flow would look like.
It consists of the following nodes
MQTT In – this subscribes to a topic from a broker
Trigger – this creates a blank message payload if no messages are received for a set timeout period
Function (Null Events) – this sets the message payload to zero if the received payload is blank, else it forwards the payload.
Function (Seconds to DD:HH:MM:SS) – this converts the message payload from seconds into a formatted Day, Hours, Minutes, and Seconds format.
Text – this presents the value on the Node Red Web UI
Here is the JavaScript of the MQTT flow.
[
{
"id": "5cae056a4a858529",
"type": "mqtt in",
"z": "0ac50cf87502b864",
"name": "",
"topic": "Host/upTime",
"qos": "2",
"datatype": "auto",
"broker": "54b01280.e7076c",
"nl": false,
"rap": true,
"rh": 0,
"inputs": 0,
"x": 480,
"y": 400,
"wires": [
[
"49e92b1ddff4a2d0",
"e394c83d9247c567"
]
]
},
{
"id": "3c383f47f53ed436",
"type": "ui_text",
"z": "0ac50cf87502b864",
"group": "5cd5d53c0624d828",
"order": 1,
"width": 0,
"height": 0,
"name": "",
"label": "Uptime",
"format": "{{msg.payload}}",
"layout": "row-spread",
"className": "",
"x": 1560,
"y": 400,
"wires": []
},
{
"id": "e394c83d9247c567",
"type": "function",
"z": "0ac50cf87502b864",
"name": "Null Events",
"func": "if(msg.payload.length === 0){\n\tmsg.payload = \"0\"\n\t}\nreturn msg;",
"outputs": 1,
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 1050,
"y": 400,
"wires": [
[
"51bb587b2ac4bcb5"
]
]
},
{
"id": "49e92b1ddff4a2d0",
"type": "trigger",
"z": "0ac50cf87502b864",
"name": "30 Second Timeout",
"op1": "",
"op2": "",
"op1type": "nul",
"op2type": "str",
"duration": "30",
"extend": true,
"overrideDelay": false,
"units": "s",
"reset": "",
"bytopic": "all",
"topic": "topic",
"outputs": 1,
"x": 790,
"y": 360,
"wires": [
[
"e394c83d9247c567"
]
]
},
{
"id": "51bb587b2ac4bcb5",
"type": "function",
"z": "0ac50cf87502b864",
"name": "Seconds to DD:HH:MM:SS",
"func": " var totalNumberOfSeconds = msg.payload;\n var days = parseInt( totalNumberOfSeconds / 86400 );\n var hours = parseInt (( totalNumberOfSeconds - ( days * 86400 )) / 3600 );\n var minutes = parseInt ((totalNumberOfSeconds - ((hours * 3600)+( days * 86400 ))) / 60 );\n var seconds = parseInt(totalNumberOfSeconds - ((hours * 3600) + (minutes * 60)+( days * 86400 )));\n var result = (days < 10 ? \"0\" + days : days) + \" Days \" + (hours < 10 ? \"0\" + hours : hours) + \":\" + (minutes < 10 ? \"0\" + minutes : minutes) + \":\" + (seconds < 10 ? \"0\" + seconds : seconds);\n msg.payload=result;\n return msg;",
"outputs": 1,
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 1300,
"y": 400,
"wires": [
[
"3c383f47f53ed436"
]
]
},
{
"id": "54b01280.e7076c",
"type": "mqtt-broker",
"name": "Broker Name",
"broker": "Broker IP Address",
"port": "1883",
"clientid": "",
"autoConnect": true,
"usetls": false,
"protocolVersion": "3",
"keepalive": "60",
"cleansession": true,
"birthTopic": "",
"birthQos": "0",
"birthPayload": "",
"birthMsg": {},
"closeTopic": "",
"closePayload": "",
"closeMsg": {},
"willTopic": "",
"willQos": "0",
"willPayload": "",
"willMsg": {},
"sessionExpiry": ""
},
{
"id": "5cd5d53c0624d828",
"type": "ui_group",
"name": "Host",
"tab": "4afc27d31e945b9d",
"order": 5,
"disp": true,
"width": "7",
"collapse": false,
"className": ""
},
{
"id": "4afc27d31e945b9d",
"type": "ui_tab",
"name": "Systems",
"icon": "dashboard",
"order": 6,
"disabled": false,
"hidden": false
}
]
Setting up the SNMP agent on a Linux system requires these steps.
- Install using the “sudo apt-get install snmpd snmp” command
- Edit the SNMP config file /etc/snmp/snmpd.conf
agentAddress udp:161 rocommunity public <Single IP Address>/32 sysLocation City, State sysContact Admin <admin@local.local>
- The restart the SNMP service with this command “/etc/init.d/snmpd restart”
Windows is a bit more involving, see https://theitbros.com/snmp-service-on-windows-10/ for details. Here is an outline of those steps.
- From a Powershell prompt, enter ‘Add-WindowsCapability -Online -Name “SNMP.Client~~~~0.0.1.0“’
- Then navigate to Settings > Apps > Apps & Features > Manage optional feature > Add Feature and select Simple Network Management Protocol (SNMP)
- The SNMP service should now be in the Services Console
- Edit the SNMP servie Agent and Security settings
- Restart the service to apply those changes.
Another layer is the firewall settings, which will not be covered in detail here. But suffice it to say the host running the snmp service should only allow access for the intended system(s) making queries. This is why the SNMP method can become daunting when scaled to larger deployments.
Setting up the MQTT agent on a Linux system requires these steps.
- sudo apt install -y mosquitto-clients
That’s it, nothing more needs to be done except to start a bash script that publishes the readings. Here is an example script that publishes system host resources.
#!/bin/bash
# cloudacm.com
# LocalSysToMQTT.sh
while true;
do
sysname=$(hostname -s)
upTime=$(awk '{print $1}' /proc/uptime)
cpuUser=$(top -bn1 | awk '/Cpu/ { print $4}')
cpuSystem=$(top -bn1 | awk '/Cpu/ { print $6}')
loadAve=$(uptime | tr -d , | awk '/load average/ { print $8}')
memUsage=$(free -m | awk '/Mem/{print $3}')
diskUsage=$(df -h | tr -d % | awk '/sda1/{print $5}')
mosquitto_pub -h <Broker> -m "$upTime" -t $sysname/upTime
mosquitto_pub -h <Broker> -m "$cpuUser" -t $sysname/cpuUser
mosquitto_pub -h <Broker> -m "$cpuSystem" -t $sysname/cpuSystem
mosquitto_pub -h <Broker> -m "$loadAve" -t $sysname/loadAve
mosquitto_pub -h <Broker> -m "$memUsage" -t $sysname/memUsage
mosquitto_pub -h <Broker> -m "$diskUsage" -t $sysname/diskUsage
sleep 10;
done
Windows is a little indirect, but not nearly as complicated as its SNMP counterpart. There is an installer available here, https://mosquitto.org/download/. Once downloaded, I was able to install it on a sandbox system and gather these binaries for use as a portable install.
- mosquitto_sub.exe
- mosquitto_pub.exe
- mosquitto.dll
- libssl-3-x64.dll
- libcrypto-3-x64.dll
After placing these on another host and adding the path to the environmental variable, they are ready for this batch file.
@echo off
: windows batch file
: cloudacm.com
: LocalSysToMQTT.cmd
:publish
@for /f "skip=1" %%a in ('wmic computersystem get Name /VALUE') do (
for /F "tokens=2 delims==" %%b in ("%%a") do if not defined SysHostName set "SysHostName=%%b"
)
for /f "tokens=4" %%a in ('net statistics workstation ^| find "Statistics since"') do if not defined SysUpTime set "SysUpTime=%%a"
for /f "tokens=3" %%a in ('net statistics workstation ^| find "Statistics since"') do if not defined SysUpDate set "SysUpDate=%%a"
@for /f "skip=1" %%a in ('wmic cpu get loadpercentage /VALUE') do (
for /F "tokens=2 delims==" %%b in ("%%a") do if not defined SysCPULoad set "SysCPULoad=%%b"
)
@for /f "skip=1" %%a in ('wmic OS get FreePhysicalMemory /VALUE') do (
for /F "tokens=2 delims==" %%b in ("%%a") do if not defined SysFreeMemory set "SysFreeMemory=%%b"
)
@for /f "skip=1" %%a in ('wmic logicaldisk get FreeSpace /VALUE') do (
for /F "tokens=2 delims==" %%b in ("%%a") do if not defined SysFreeDisk set "SysFreeDisk=%%b"
)
mosquitto_pub -h <Broker> -m "%SysHostName%" -t %SysHostName%/hostName
mosquitto_pub -h <Broker> -m "%SysUpTime%" -t %SysHostName%/bootTime
mosquitto_pub -h <Broker> -m "%SysUpDate%" -t %SysHostName%/bootDate
mosquitto_pub -h <Broker> -m "%SysCPULoad%" -t %SysHostName%/cpuLoad
mosquitto_pub -h <Broker> -m "%SysFreeMemory%" -t %SysHostName%/memFree
mosquitto_pub -h <Broker> -m "%SysFreeDisk%" -t %SysHostName%/diskFree
TIMEOUT /T 10
goto publish
Happy monitoring.