Metrics is having problems with messages that have whitespace.

[hrant]

The node sends something like the following to the gateway (just an example)

Code Select Expand

radio.sendWithRetry(GATEWAYID, "Foobar: Register", len)

Or

Code Select Expand

radio.sendWithRetry(GATEWAYID, "Foobar: Error,  received invalid data", len)

(len is properly calculated)

If I set the metrics to this

Code Select Expand

foobar_register : {
	name : 'register',
	regexp : /Foobar\:/i,
	value : 'register'
}

Either message is seen. and of course the value is 'register'
This is expected as the regex is just matching Foobar:

Now let's try matching Foobar: Register
I would expect the regex to be:

Code Select Expand

regexp : /Foobar\:\sRegister/i,

But that fails miserably,
so does

Code Select Expand

regexp : /Foobar\: Register/i,

The only way I could get around it is by eliminating the space from the sent message.

Code Select Expand

radio.sendWithRetry(GATEWAYID, "Foobar:Register", len)

Or

Code Select Expand

radio.sendWithRetry(GATEWAYID, "Foobar:Error,  received invalid data", len)

But even then, if I try to capture the Error message, to display it as value

Code Select Expand

regexp : /Foobar\:(Error.*)/i

The value I get is:
Error,

It looks like whitespace is treated as terminating character.

Is this a bug or a limitation in the metrics processor? or am I doing something wrong?

I hate to eliminate all the whitespace in the node messages to get around this issue.

Any guidance / feedback is greatly appreciated.

[Felix]

Whitespace is a separator, naturally it will get split in the gateway token parser. You can change that behavior if you'd like but by default the expected format is:

"TOKEN1 TOKEN2 TOKEN3 ... TOKENn"

[hrant]

Thanks for the feedback Felix

So if it gets split in the gateway token parser, how does one use it, perhaps there is an intended use that I'm missing?
otherwise where do I change this behavior?

Thanks

[perky]

Is it escapable? Could you use a backslash in front of the space?
Mark.

[Felix]

Not escapable, this is split by javascript regex.
Consider using another separator within the same token, not whitespace, computers don't care.

[hrant]

OK I tried both approaches,
Using a different separator ":", or keeping the space and handling them as multiple tokens.

Still couldn't get what I was trying to ultimately achieve.
Consider the following example.
Mote Name: FooMote
It needs to send multiple numeric values as a set, they should be grouped together and are meaningless in isolation.
Let's say values for A,B,C,D

What I want to graph is either a single dot similar to Doorbell ring, where the value is 1 and the label (hover over) of the dot displaying something like
A:123,B:456,C:768,D:90

Or alternately I'm ok with having 4 vertical dots at the same time slot, at different heights, (values 1, 2, 3, 4)
each displaying one of the metrics as it's label (hover over)
A:123 at vertical 4
B:456 at vertical 3
C:768 at vertical 2
D:90 at vertical 1

of course these number would change each time the node reports, hence can't be hardcoded and need to be extracted from the received message.

If I start with the  following Sketch code example with spaces:

Code Select Expand

...
sprintf(buff, "FooMote:A:%lu FooMote:B:%d FooMote:C:%d FooMote:D:%d", Aval, Bval, Cval, dVal);
reportToGW(buff);
...
boolean reportToGW(char *msg)
{
  byte len = strlen(msg);
  return radio.sendWithRetry(GATEWAYID, msg, len);
}

metrics.js code.

Code Select Expand

foomote_a:{name:'Events',regexp:/FooMote\:A\:(\d+)/i,label:'A',graphValPrefix:'A:',unit:'',pin:1,graph:1,graphOptions:{legendLbl:'A',lines:{show:false,fill:false},points:{show:true,radius:5,lineWidth:5},grid:{backgroundColor:{colors:['#000','#000']}},yaxis:{ticks:0},colors:['#a40']}},
foomote_b:{name:'Events',regexp:/FooMote\:B\:(\d+)/i,label:'B',graphValPrefix:'B:',unit:'',pin:1,graph:1,graphOptions:{legendLbl:'B',lines:{show:false,fill:false},points:{show:true,radius:5,lineWidth:5},grid:{backgroundColor:{colors:['#000','#000']}},yaxis:{ticks:0},colors:['#a40']}},
foomote_c:{name:'Events',regexp:/FooMote\:C\:(\d+)/i,label:'C',graphValPrefix:'C:',unit:'',pin:1,graph:1,graphOptions:{legendLbl:'C',lines:{show:false,fill:false},points:{show:true,radius:5,lineWidth:5},grid:{backgroundColor:{colors:['#000','#000']}},yaxis:{ticks:0},colors:['#a40']}},
foomote_d:{name:'Events',regexp:/FooMote\:D\:(\d+)/i,label:'D',graphValPrefix:'D:',unit:'',pin:1,graph:1,graphOptions:{legendLbl:'D',lines:{show:false,fill:false},points:{show:true,radius:5,lineWidth:5},grid:{backgroundColor:{colors:['#000','#000']}},yaxis:{ticks:0},colors:['#a40']}}

Is it even possible to achieve what I'm trying to get.
The above code produces really weird results
C always comes up as temperature with the degree unit at the end, even though I explicitly set the unit to ''
As if it is picking up the weathershield metrics.
The name is different, the fields are different, yet it's picking that up, and no dot is displayed in any of the graphs.

If I try to not use spaces and use another separator (let's say

Code Select Expand

sprintf(buff, "FooMote:A:%lu:B:%d:C:%d:D:%d", Aval, Bval, Cval, dVal);

And do a regex something like this

Code Select Expand

egexp : /FooMote\:(.*)/i

This doesn't work either
expecting to capture everything after FooMote:

If I use this regex

Code Select Expand

regexp : /FooMote\:(A\:\d+)\:(B\:\d+)\:(C\:\d+)\:(D\:\d+)/i

How do you even specify which capturing group gets assigned to value / label ...

Am I out of luck or doing something wrong?
Thanks

[Felix]

I would not waste so much bandwidth with repeated tokens like "FooMote".
You can setup that in your Gateway and assign that token in the node name or description, based on the Node ID.

Your "C:numeric-value" is probably matching the default Celsius temperature metric from metrics.js :

Code Select Expand

C : { name:'C', regexp:/C\:([-\d\.]+)/i, value:'', unit:'°', pin:1, graph:1, graphValSuffix:'C', graphOptions:{ legendLbl:'Temperature' }},

If you really need FooMote in your tokens, then use another separator (like - or _, to avoid matching existing metrics).
You could also put your definition before the above default, and the first match would have priority, however in the new v8.8 release I recommend users to NOT touch metrics.js and instead load their own custom metrics under ~/gateway/userMetrics as shown in the example from that directory.

For instance: FooMote_C:value

Then your matching regex for that token could be:

Code Select Expand

/FooMote-C\:([-\d\.]+)/i

This would match FooMote-C, colon, decimal integer, (/i = case insensitive). Note the capturing parentheses which is required if you want to capture the actual numeric value.

Try out your expressions at regexr.com , a great tool for testing your regex matching expressions.

[hrant]

Thanks Felix for taking the time to respond.

I don't have difficulty writing regular expressions, thanks for the link though, it's a very good one, I usually use https://regex101.com/ which is another good one.

What I'm having difficulty with is getting a good grasp of how metrics is working.
Yes I am using the latest 8.8, and indeed I have all my node customization broken into individual files (one for each custom node type) and placed in userMetrics, thanks for that, makes it really easy to update without worrying to do a merge, great feature by the way.
It also makes it easy to share code without asking users to edit metrics.js.

I see that the temperature metrics expression is too loose (what I mean is that it can easily match an unintended string), it would match any c:numeric which also happens to be in my data.
That is why I was trying to prefix mine with FooMote: to make it more strict,
I suppose a better match is not selected, but rather, first match.
How do I place files in userMetrics folder and have it higher priority than what is defined in metrics.js ?
I was under the impression that metrics.js has lower priority than what's in userMetrics, I guess not.

I can change the format of my message so that it doesn't collide with the temperature metrics, but such loose regex-es should not have such high priority.
No where in my message I can have c:numeric or p:numeric or f or ... (you get the drift)

I also see that it stops on first match, so how do I extract multiple fields from a single message?
short of sending multiple messages, I don't see a way.
Unless multiple capturing groups work (which I doubt)

Here's another issue I'm facing.
If the value is a string specified in the metrics example GarageMote value 'OPEN' or in DoorbellMote ring value 'RING'
then that string is displayed in the graph.
However if the value is not specified, then it's supposed to pick the value from the regex capture, which works fine on the node list page.
but does not in the node metrics page.
Instead of showing the captured value it shows the logValue.

Also if the captured value contains a string, then nothing is displaying on the graph
METRIC NOT NUMERIC, logging skipped...
Even though a logValue is specified (1)

As previously states, my metrics is a group of data, which needs to be displayed / logged as a set.
So either one string containing the entire string is fine as long as the node metrics shows it as a label
Or multiple dots at the same time at different heights each capturing one metric, but viewed as a set because they all align vertically at a single time point.
I suppose the former is easier, assuming that captured strings can be displayed as point label
The latter as it stands can only work if I send multiple messages from the node.
But still would require a captured value being displayed as label and not metric.

I see that you can set graphValSuffix
Can you also set graphValPrefix ?

Many thanks in advance.

[Felix]

Thanks for the feedback and details are welcome but I will have to slowly digest your long message before I can get back to you on every point.

Below is a sample that illustrates graphValPrefix, and also how to capture and log a non numeric value (node simply reports STS:CLOSED, and logger will log value 0):

Code Select Expand

closed : { name:'Status', regexp:/(?:STS\:)?(CLS|CLOSED)/i, value:'CLOSED', pin:1, graphValPrefix:' Door: ', graph:1, logValue:0 },

[Felix]

Ok let me try to address your questions:

What I'm having difficulty with is getting a good grasp of how metrics is working.

First step is splitting of messages (single line, whitespace separated) into individual tokens.
Second step is trying to match each token against a metric.
The parser goes through each metric definition in metrics. When a token matches a definition, the parser stops trying to match. It logs any data if needed and moves to next token.

One important aspect is that any metrics defined in userMetrics will override anything defined in metrics.js. This is powerful and allows you to redefine any metric if you'd like. For instance you could redefine a metric like the C:value if you feel that's too loose, make it more strict so it would not match your own tokens, etc.
I have to agree with you some of these metrics could be massaged a little to avoid mismatching. The C:value could include word boundaries (\b), something like this:

Code Select Expand

/\bc:([-\d\.]+)\b/ig

I also see that it stops on first match, so how do I extract multiple fields from a single message?

See answer above. Each token is evaluated individually against each metric, first metric match stops further matching on that token.

Capturing groups: there is 1 group that matters - the first - that should capture a numeric value, which becomes the logged value, unless overriden with logValue. This is illustrated in the many given examples.

Here's another issue I'm facing.
If the value is a string specified in the metrics example GarageMote value 'OPEN' or in DoorbellMote ring value 'RING'
then that string is displayed in the graph.
However if the value is not specified, then it's supposed to pick the value from the regex capture, which works fine on the node list page.
but does not in the node metrics page.
Instead of showing the captured value it shows the logValue.

Also if the captured value contains a string, then nothing is displaying on the graph
METRIC NOT NUMERIC, logging skipped...
Even though a logValue is specified (1)

I don't really know what you mean. Look at the given examples below, they illustrate how to take a non-numeric metric, and log numeric values for it. A garage has several different states, yet each can log a different value, and in the node display the string while logging a numeric value. In the graph it will show the string again, unless you omit the value:

Code Select Expand

  //GarageMote
  //NOTE the \b word boundary is used to avoid matching "OPENING" (ie OPEN must be followed by word boundary/end of word)
  open : { name:'Status', regexp:/(?:STS\:)?(OPN|OPEN)\b/i, value:'OPEN', pin:1, graph:1, logValue:2, graphOptions:{ legendLbl:'Garage door events', yaxis: {ticks:0}, colors:['#4a0'], /*lines: { lineWidth:1 }*/}},
  opening : { name:'Status', regexp:/(?:STS\:)?(OPNING|OPENING)/i, value:'OPENING..', pin:1, graph:1, logValue:1 },
  closed : { name:'Status', regexp:/(?:STS\:)?(CLS|CLOSED)/i, value:'CLOSED', pin:1, graphValPrefix:' Door: ', graph:1, logValue:0 },
  closing : { name:'Status', regexp:/(?:STS\:)?(CLSING|CLOSING)/i, value:'CLOSING..', pin:1, graph:1, logValue:1.1 }, //1.1 to avoid a match with "OPENING"
  unknown : { name:'Status', regexp:/(?:STS\:)?(UNK|UNKNOWN)/i, value:'UNKNOWN!', pin:1, graph:1, logValue:0.5 },

As previously states, my metrics is a group of data, which needs to be displayed / logged as a set.
So either one string containing the entire string is fine as long as the node metrics shows it as a label

Again, not 100% sure what you're after, but each token is treated separately, has its own log and graph. You can obviously have any number of metrics per node. There is no feature yet to display multiple metrics on 1 graph, perhaps that's what you want. It's something i'd like to add in the future, a way to group metrics on a graph or a container "virtual" node.

If you have more questions let me know.

[hrant]

Thanks again Felix, for going into details and explaining how it works, much appreciated.
I'm beginning to believe that I won't be able to achieve what I'm try to do.
Perhaps picture(s) are worth 1000 words, hope it explains my goals better.

As previously stated, my message has multiple data to be treated as a single set, the values after each letter changes from message to message, hence the need for capturing group

If I send to the gateway as a single message, with no spaces.
I get the following on the node display


This is all good, that's exactly what I want to see
For reference, this is the regex
regexp : /FooMote\:(A_\d+_B_\d+_C_\d+_D_\d+)/i

I'm using DoorbellMote style graph
and am setting the logValue to 1
value is not set (captured)
I get the following on the graph

Notice how it shows the logValue as the label instead of the captured value from the regex (as it was displayed on the node)
Compare this to Doorbell mote
which displays the following


In doorbell metrics the value is hardcoded to 'RING' and the logLevel to 1
and the graph shows the string 'RING' as label

Why is it that when the value is captured, instead of displaying the captured value it decides to display the logValue?
That's the issue I'm struggling with.

That's why I even entertained the idea of displaying  4 dots similar to this

each representing one metric, and the regex only capturing the numeric
Still no go, the minute you don't define the value, and even if the captured value is numeric, it decides to show the logValue as a label


Hope this clarifies the problem I'm encountering and hope there's a workaround.

Thanks again.

[Felix]

The problem is the value you log is always 1, so thats what goes to the disk, not your actual values. For RING, the value is constant 1, in fact we don't even care what it is, but for it to be logged (remember the timestamp when a RING happened) the value has to be NUMERIC, that's because of the data point format (timestamp + numeric value), so we just hardcode a constant value (1 or whatever).

For you to log actual changing values to disk you should be splitting into separate metrics and capture the value and log that instead of a hardcoded constant. There's no way to log a "string". If you want you can fork my work and do that but it will be a lot slower to store strings.

In doorbell metrics the value is hardcoded to 'RING' and the logLevel to 1
and the graph shows the string 'RING' as label

Why is it that when the value is captured, instead of displaying the captured value it decides to display the logValue?
That's the issue I'm struggling with.

See the RING definition:

Code Select Expand

ring : { name:'RING', regexp:/RING/i, value:'RING', pin:1, graph:1, logValue:1, graphValSuffix:'!'

If you don't show me your metric I cannot really tell what's going on so please include your latest.
Everything you need is doable except graphing all on 1 graph is not yet supported. But you have to capture separate metrics.