OIBus North connector Data Rate

OIBus sends values ​​to a target application via the REST API North connector (OIConnect, OIAnalytics … ). This connector offers two sending modes: a File endpoint mode and a Values ​​endpoint mode. The volumes to be taken into account can be estimated according to the data to be sent and the sending mode selected. These estimates can also be used to size the amount of cache storage needed to ensure the store and forward under good conditions.

Sending values ​​with File endpoint (CSV)

We will focus on data in the form of CSV files. In this case the volume will depend on several parameters:

  • The data sampling frequency
  • The file sending frequency
  • The timestamp format
  • The data format: number of characters used, so the precision
  • The size of the data references
  • The file format: in lines or in columns

In the following examples, we will look at how much space occupies the CSV file generated by OIBus. We took the following assumptions:

  • The sampling frequency: one point per minute.
  • The frequency of sending the file: one file every 30 minutes.
  • The timestamp format: ISO 8601 format, 24 bytes in size.
  • Data format: 3 digits with a separator for the decimal places. The data in the following examples therefore have a size of 4 bytes.
  • The size of the data references: DataXXX , where XXX represents three numbers characters. The references of the following examples therefore have a size of 7 bytes.

Column Files

This format is particularly suitable for data repeated on the same timestamp, which saves space compared to a “lines” format.

Timestamp Data001 Data002 Data003
2020-02-01T20: 04: 00.000Z 12.0 10.0 10.0
2020-02-01T20: 05: 00.000Z 10.0 19.0 10.0
2020-02-01T20: 06: 00.000Z 10.0 10.0 14.0
.. .

The size of the header here is 10 + 1 + 7 + 1 + 7 + 1 + 7 + 1 = 35 bytes
The size of a line here is 24 + 1 + 4 + 1 + 4 + 1 + 4 + 1 = 40 bytes
(the 1 correspond to column separators and newlines)

The number of lines depends on the frequency of the data, here one line every minute. With a file sent every 30 minutes, it will therefore have a size of 35 40 x 30 = 1235 bytes. Over a day, there will be 48 files, a total of 59,280 bytes or 58 kB.

Row Files

This format is particularly suitable when the different data transmitted do not have the same sampling frequency. In the example we assume that all data has the same sample rate.

Timestamp Reference Value
2020-02-01T20: 04: 00.000Z Data001 12.0
2020-02-01T20: 04: 00.000Z Data002 10.0
2020- 02-01T20: 04: 00.000Z Data003 10.0
2020-02-01T20: 05: 00.000Z Data001 10.0
2020-02-01T20: 05: 00.000Z Data002 19.0
2020-02-01T20: 05: 00.000Z Data003 10,0
2020-02-01T20: 06: 00.000Z Data001 10.0
2020-02- 01T20: 06: 00.000Z Data002 10.0
2020-02-01T20: 06: 00.000Z Data003 14.0

The size of the header here is 10 + 1 + 9 + 1 + 6 + 1 = 28 octets
The size of a line here is 24 + 1 + 7 + 1 + 4 + 1 = 38 bytes
(the 1s correspond to the characters separating columns and to the newlines)

The number of lines depends on the frequency of the data and the number of references, here one line every minute multiplied by 3 references. Consider 3 lines per minute. With one file sent every 30 minutes, it will therefore have a size of 28 + 38 x 30 x 3 = 3448 bytes. Over a day, there will be 48 files, a total of 165,504 bytes or 162 kB.

Column Row Files

This format has the advantage of the column file and allows the pooling of data identifiers ( 001 , 002 , 003 ) with the references if there are several, which is not the case here since only Data is used. This allows you to obtain the references Data001 , Data002 , Data003 .

Timestamp Reference 001 002 003
2020-02-01T20: 04: 00.000Z Data 12,0 10,0 10,0
2020-02-01T20: 05: 00.000Z Data 10.0 19.0 10.0
2020-02-01T20: 06: 00.000Z Data 10.0 10.0 14 , 0
… < / td>

The size of the header here is 10 + 1 + 9 + 1 + 3 + 1 + 3 + 1 + 3 + 1 = 33 bytes
La taille d’une ligne est ici de 24 + 1 + 4 + 1 + 4 + 1 + 4 + 1 + 4 + 1 = 45 bytes
(the 1s correspond to column separator characters and newlines)

The number of lines depends on the frequency of the data and the number of references, here a line every minute multiplied by a reference. Or one lines per minute. With one file sent every 30 minutes, the file to be sent will therefore have a size of 33 45 x 30 = 1,383 bytes. Over a day, there will be 48 files, a total of 66,384 bytes or 65 kB.

Sending values ​​with Values ​​endpoint (JSON)

When values ​​are retrieved by the North connector then sent by a Values ​​endpoint, they are formatted in an array like this:

[
{" timestamp ":" 2020-02-01T20: 04: 00.000Z "," pointId ":" Data001 "," data ": {" value ":" 12.0 "," quality ":" 192 "}},
{"timestamp": "2020-02-01T20: 04: 00.000Z", "pointId": "Data002", "data": {"value": "10.0", "quality": "192"}},
{"timestamp": "2020-02-01T20: 04: 00.000Z", "pointId": "Data003", "data": {"value": "10.0", "quality": "192"}} ,
]

  • timestamp : indicates the timestamp of the value in ISO 8601 format
  • pointId : reference of the value
  • The data field is an object containing the recorded value ( value ) and the quality ( quality )
  • .

We will focus on data in this form of JSON file. In this case the volumetry will depend on several parameters:

  • The data sampling frequency
  • The number of points grouped by sending (Defined by the parameter Group Count )
  • The sending frequency (Defined by the Send Interval parameter)
  • The format of data and quality: number of characters used, therefore the precision
  • The size of the data references

It is then possible to estimate the space occupied by a value.

  • The timestamp is represented by "timestamp": "2020-02-01T20: 00: 00.000Z" , or 39 bytes
  • The pointId is represented by "pointId": "DataXXX" , i.e. 13 bytes plus the number of bytes of the reference (here the 7 bytes of DataXXX )
  • The data field represented by "data": {…} is 10 bytes plus its content
    • The value field represented by "value": "10.0" , i.e. 11 bytes plus the variable number of bytes on which is encoded the value (here 4 bytes)
    • The quality field represented by "quality": "192" , i.e. 13 bytes plus the variable number of bytes on which the quality is coded (here 3 bytes)

The size of the object representing a value can be broken down into:

  • The fixed object size = 39 + 13 + 10 + 11 + 13 + 6 = 92 bytes (the 6 corresponding to the separators of the different elements)
  • The size of the reference = 7 bytes
  • The size of the value = 4 bytes
  • The size of the quality = 3 bytes

We therefore have 106 bytes for a value sent.

If we start again with a sampling frequency of 1 point per minute and 3 data, if we consider and that Group Count is equal to 1000 and Send Interval 1000ms, then OIBus will transmit a JSON every minute with 3 data or 318 bytes. Over one day, this will represent 318 x 24 x 60 = 457,920 bytes, or 447 kB.

Conclusion and Comparison

Under the conditions defined in the example, it appears that the mode of transmission and the format of the data will have a significant impact on the volumes transmitted. This will be all the more critical as the number of data and their sampling frequency will be higher than what is described in this example.

CSV
Columns
CSV
Rows
CSV
Columns
JSON
Daily volume 58 kb 162 kb 65 kb < / td> 447 kb
Volume by value 13.7 o 38.3 o 15.4 o 106 o