ICT – Domotics – Blog – [20220903] – Configuring Telegraf and InfluxDB [V2] for nous Wall Plugs

Back to ICT – Domotics – Blog

In this section I will, hopefully briefly, describe configuring my Telegraf instance (as well as InfluxDB [V2]) for use with the Tasmota [nous] wall plugs.

Starting point

A nous device (configured to be) connected via WLAN and fixed IPv4 address and host name (though DHCP/DNS configuration in the router or access point). In this particular case nous-01.jdj. Note that my DHCP/DNS setup is local.
Running MQTT Broker (Mosquitto) on my Ethernet.
Routing/Firewall setup between WLAN en Ethernet (so the nous can reach the MQTT Broker; also handy for Tasmota configuration).
MQTT enabled on the nous device.

Notes:

In my setup, Telegraf is exclusively used to filter and mangle incoming data from MQTT and send the result to InFluxDB V2.
In theory, the nous plug does not need Internet access, but in this case it has just to make sure that the time on the plug does not go haywire. (Update: I intend to block Internet access later, since I override the measurement time of the telemetry.)
I use tele/nous-01/SENSOR (and others that are not relevant here), or in general tele/<host-without-domain>/SENSOR as MQTT topic.

Resources

https://www.influxdata.com/blog/mqtt-topic-payload-parsing-telegraf/ [checked 20220903]
https://www.influxdata.com/blog/how-parse-json-telegraf-influxdb-cloud/ [checked 20220903]
https://www.blakehartshorn.com/using-tasmota-smart-plugs-with-telegraf/ [checked 20220903]
https://gstephens.org/telegraf/influxdb/mqtt/tasmota/2022/01/14/telegraf-mqtt-tasmota.html [checked 20220903]

JSON Structure

Here’s the (obviously JSON) nous-01 message on topic tele/nous-01/SENSOR with purple highlighting of the stuff I’m interested in at the moment:

{
  "Time":"2022-09-03T09:44:59",
  "ENERGY":
  {
    "TotalStartTime":"2022-09-02T13:48:43",
    "Total":0.099,
    "Yesterday":0.075,
    "Today":0.024,
    "Period": 0,
    "Power": 3,
    "ApparentPower": 7,
    "ReactivePower": 7,
    "Factor":0.35,
    "Voltage":261,
    "Current":0.027
  }
}

Do not proceed until you see the relevant JSON output from the MQTT Broker!

Command to View the JSON Structure

# mosquitto_sub -d -h www.jdj -p 1883 \
-t tele/nous-01/SENSOR

You can run this on any machine, actually.

The Time Field

Well, come to think of it, the Time field is not that useful since in the end the nous Wall Plugs will run without Internet access, and I don’t feel much for setting up local NTP for the plugs. So, I might as well override the measurement time.

Topic Parsing [[inputs.mqtt_consumer]] – nous

/etc/telegraf/telegraf.conf [snippet]:

[[inputs.mqtt_consumer]]
    servers = ["tcp://127.0.0.1:1883"] # mqtt url
    topics = ["tele/+/SENSOR"] # filters topics from mqtt
    [[inputs.mqtt_consumer.topic_parsing]] # parses topic
        topic = "tele/+/SENSOR" # topic filter for section
        # measurement = "measurement/_" # dunno yet
        tags = "_/sensor_id/_" # extract sensor id as tag
                               # from topic name

In the end, this part of the configuration proved close enough. I decided to keep the provisional topic scheme tele/<host>/SENSOR for the nous plugs.

Payload Parsing [[inputs.mqtt_consumer]] – nous

/etc/telegraf/telegraf.conf [snippet]:

# DOES NOT WORK
data_format = "json_v2" # invokes the parser --
                        # lines following are parser config
[[inputs.mqtt_consumer.json_v2]]
  [[inputs.mqtt_consumer.json_v2.field]]
    path = "ENERGY.Power" # select the Power field
    # rename = "power_W" # placeholder; maybe later

Well, the idea was close, but this part did not work!

I suspect that for some unknown reason, the data_format line in between the [[inputs.mqtt_consumer.topic_parsing]] and [[inputs.mqtt_consumer.json_v2]] caused trouble… I moved that line to the global section [[inputs.mqtt_consumer]], and things started to work.

In other words, the configuration below did not work, see the error output from telegraf.

# DOES NOT WORK
[[inputs.mqtt_consumer]]
    servers = ["tcp://127.0.0.1:1883"]
    topics = ["tele/+/SENSOR"]
    [[inputs.mqtt_consumer.topic_parsing]]
        topic = "tele/+/SENSOR"
        # measurement = "measurement/_"
        tags = "_/sensor_id/_"
    data_format = "json_v2"
    [[inputs.mqtt_consumer.json_v2]]
        [[inputs.mqtt_consumer.json_v2.field]]
            path = "ENERGY.Power"
            rename = "power_W"

Sep 03 12:17:44 www.jdj telegraf[371348]: 2022-09-03T10:17:44Z E! [inputs.mqtt_consumer] Error in plugin: metric parse error: expected tag at 1:79: "{\"Time\":\"2022-09-03T11:17:44\",\"ENERGY\":{\"TotalStartTime\":\"2022-09-02T13:48:43\",\"Total\":0.102,\"Yesterday\":0.075,\"Today\":0.027,\"Period\": 0,\"Power\": 2,\"ApparentPower\": 7,\"ReactivePower\": 7,\"Factor\":0.30,\"Voltage\":260,\"Current\":0.027}}"

Command to Obtain this Error Output

# /usr/bin/telegraf \
  --debug \
  -config /etc/telegraf/telegraf.conf \
  -config-directory /etc/telegraf/telegraf.d

Aggregate [[inputs.mqtt_consumer]] Configuration – nous

Well, at around 14:00 I reached a milestone: Data started to enter InFluxDB! Here’s the Telegraf config /etc/telegraf/telegraf.config that made it start to work:

[[inputs.mqtt_consumer]]
    servers = ["tcp://127.0.0.1:1883"]
    topics = ["tele/+/SENSOR"]
    data_format = "json_v2"
    [[inputs.mqtt_consumer.topic_parsing]]
        topic = "tele/+/SENSOR"
        # measurement = "measurement/_"
        tags = "_/sensor_id/_"
    [[inputs.mqtt_consumer.json_v2]]
        [[inputs.mqtt_consumer.json_v2.field]]
            path = "ENERGY.Power"
            rename = "power_W"

The Telegraf output from [output.plugin] was as follows:

mqtt_consumer,host=www.jdj,sensor_id=nous-01,topic=tele/nous-01/SENSOR power_W=2 1662206324829319614

At first I thought the output was useless, but it proved to be OK. Later, I added an option to the Telegraf configuration to remove the host tag, regenerated a key for the telegraf_mqtt bucket in InfluxDB, and added the Total and Voltage metrics (the latter because it is a tad optimistic about my mains voltage).

Consolidation

After some more testing, I decided that everything was working and started to consolidate.

I replaced /etc/telegraf/telegraf.conf with its pristine version and moved sections to /etc/telegraf/telegraf.d/ in order to make life easier upon Telegraf updates while retaining the heavily documented /etc/telegraf/telegraf.conf. Upon Telegraf updates, the pristine /etc/telegraf/telegraf.conf will (should) be replaced with the new default, while still using the sections in /etc/telegraf/telegraf.d/. This avoids the creation of an .rpmnew file. (At least, this is how Fedora set this up; YMMV.)

By the way, Telegraf will only pick up configuration files in /etc/telegraf/telegraf.d/ with file name ending in .conf.

/etc/telegraf/telegraf.d/telegraf.inputs.mqtt_consumer.nous.conf

[[inputs.mqtt_consumer]]
    servers = ["tcp://127.0.0.1:1883"]
    topics = ["tele/+/SENSOR"]
    data_format = "json_v2"
    tagexclude = ["host"]
    [[inputs.mqtt_consumer.topic_parsing]]
        topic = "tele/+/SENSOR"
        # measurement = "measurement/_"
        tags = "_/sensor_id/_"
    [[inputs.mqtt_consumer.json_v2]]
        [[inputs.mqtt_consumer.json_v2.field]]
            path = "ENERGY.Total"
            rename = "energy_kWh"
        [[inputs.mqtt_consumer.json_v2.field]]
            path = "ENERGY.Power"
            rename = "power_W"
        [[inputs.mqtt_consumer.json_v2.field]]
            path = "ENERGY.Voltage"
            rename = "voltage_V"

/etc/telegraf/telegraf.d/telegraf.outputs.influxdb_v2.telegraf_mqtt.conf

[[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:8086"]
  token = "<token>"
  organization = "<organization>"
  bucket = "telegraf_mqtt"
  namepass = ["mqtt_consumer"]

Note the namepass setting preventing the telegraf_mqtt bucket from being cluttered with (e.g.) “cpu”, “disks”, etc., measurements. Since we have multiple [[outputs.influxdb_v2]] sections in our configuration (see below), we need to decide which measurement goes where. The namepass array lists the measurement names to be considered (this is, in fact, an educated guess!).

/etc/telegraf/telegraf.d/telegraf.outputs.influxdb_v2.telegraf_hoststats.conf

[[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:8086"]
  token = "<token>"
  organization = "<organization>"
  bucket = "telegraf_hoststats"
  namepass = ["cpu","disk","diskio","kernel","mem","net","processes","swap","system"]

This is a legacy configuration to get essential statistics on a host into InfluxDB [V2].

/etc/telegraf/telegraf.d/telegraf.outputs.file.stdout.conf.disabled

# For debugging; end file name in '.conf' to enable.
[[outputs.file]]
  files = ["stdout"]

This is a configuration for debugging, which I normally disable (through its file-name).