FTP/SFTP Integration

The process of sending data to the platform consists of publishing the data in the SCADA system through the MQTT protocol. The cloud SCADA solution adopted by thingable! performs all the work of storing historical data and making the data available in real time. A thingable! does not yet automatically acquire data from an FTP/SFTP server, so it becomes necessary to make use of Logic Builder to fetch this data and republish it on the platform using MQTT, our standard protocol.

In sections 1.1 and 1.2 is presented all the necessary process to install and use the FTP/SFTP nodes in Logic Builder. Section 1.3 presents the final process of republishing data on the platform.

1.1- Logic Builder - Installation of nodes

Initially it is necessary to install the libraries "node-red-contrib-advanced-ftp" and "@ xst/node-red-contrib-sftp" in the Logic Builder tool. If they are already installed, continue from the Section 1.2.

Below is the step-by-step for the installation and use of the mentioned tool:

Figure 1: Access to the featured tool "Logic Builder"

Access the "Manage Palette" menu:

Figure 2: "Logic Builder" tool menu, "Manage Palette" option

After accessing the "Manage Palette" menu, click on "Palette" in the "Install" option to search for "node-contrib-advanced-ftp" and click on "install" and wait for the installation process (figure 3). Repeat the process for the "@ xst/node- red-contrib-sftp" library. Once completed, you can return to the node and logic creation page:

Figure 3: Installation screen of ftp and sftp libraries

If the installation is correct it will appear in the palette of nodes referring to ftp and sftp (figure 4)

Figure 4 – Featured features of the “Advanced ftp” and “SFTP” library

1.2 - Logic Builder - Node Configuration

In the logic development tool, as mentioned above, it is possible to configure a ftp or sftp server, it is at the user's choice. In this tutorial we will explain how to configure for both cases.

1.2.1- FTP

To configure a server on the platform, we will use the "Advanced ftp" library, it allows us to perform some important functions in an FTP area, which are visualization, transfer and deletion of files. The main operations are:

• LIST

• GET

• PUT

• DELETE

For more information on library usage and how to configure nodes, visit the link: https://flows.nodered.org/node/node-red-contrib-advanced-ftp

1.2.1.1 - FTP - Configuration

To configure the node, simply access drag the node to the Logic Builder free desktop and click on the node (figure 5):

Figure 5: Highlighted, the Advanced FTP node configuration parameters

To configure the node it is necessary to fill in the fields:

• FTP Server: fill in the credentials of the server where the data will come from. It is possible that the user configures the access of a server of his choice or the server of the thingable!, to contract the use of our server contact support requesting an access to the FTP area.

When accessing the pencil on the right side of the "FTP Server" field, a window will open with the information regarding the FTP server that must be parameterized (figure 7). The main ones are:

• Host: Server Address

• Port: Server Port

• Username: Server access user

• Password: Password to access the server

The others are default settings for node operation, no need to change them.

Figure 6: Configuration parameters for Advanced FTP node server

• Operation: define which function will be performed by that node. The main functions have already been described: LIST, get, PUT and DELETE.

• Data Type: What type of data format will be delivered in the query return string to

  fTP area, can be a binary or ascii type.

1.2.1.2.- FTP - Operations

1.2.1.2.1 – LIST

To use a LIST type function, simply select the previously configured server (figure 6), select the "Operation" field as "LIST" and in the "Working Directory" field choose the directory that will be searched in the FTP area (figure 7), the return payload will be an array with the list of files and folders found in the parameterized directory (figure 8).To view the node return message, simply place a debug node on the output of the Advanced FTP node, as shown in figure 9.

Figure 7: Configuration parameters referring to the LIST operation of the Advanced FTP node

Figure 8: The return of the LIST operation is highlighted

1.2.1.2.2 – GET

To use a GET type function, simply select the previously configured server (figure 6), select the "Operation" field as "GET" and in the "File Name" field choose the file you want to download. In your configuration it is necessary to pass the name of the directory field + the file name. In the "Local File Name" field, parameterize to which directory will be stored the downloaded file, in this case use the directory "/tmp" which is the only valid path on the platform to store the files. As return will come a payload with the status of the operation, it can be a positive return: "GET operation successful", or a negative return: "GET Failed". To view the node return message simply place a debug node on the output of the Advanced FTP node, as seen in figure 9.

Figure 9: Parameters of settings regarding the GET operation of the Advanced FTP node

Figure 10: Highlight the return of the GET OPERATION

To read the file downloaded and stored in the local directory, simply use a file reading function in a function node "Function" on the output of the Advanced FTP node (figure 11). The return of the file will be a payload in string format with the contents of the file.

Below is the code used to read the file contained in the function node:

return { payload: context.global.fs.readFileSync(msg.localFilename)}

The "msg.localFilename" object is automatically created in the FTP node output when configured with the get operation.

Figure 11: Highlight the return of the reading of the downloaded file in the GET OPERATION

1.2.1.2.3 – DELETE

To use a DELETE type function, simply select the previously configured server (figure 6), select the "Operation" field as "DELETE" and in the "File Name" field choose the file you want to delete on the server. In its configuration it is necessary to pass the directory field + the file (figure 12).

As return will come a payload with the status of the operation, it can be a positive return: "DELETE operation successful", or a negative return: "DELETE Failed". To view the node return message simply place a debug node on the output of the Advanced FTP node, as seen in figure 13.

Figure 12: Configuration parameters related to the DELETE operation of the Advanced FTP name

Figure 13: Highlight the return of the reading of the downloaded file in the DELETE OPERATION

1.2.1.2.4 – PUT

To use a PUT type function, simply select the previously configured server (figure 6), select the "Operation" field as "PUT" and in the "File Name" field choose the directory to add the file. In its configuration it is necessary to pass the directory field + the file (figure 14). In the "Local File Name" field, it is parameterized to which location the file will be searched, to be sent in the chosen directory in the "File Name". Contact the FTP service provider to find out which directory is valid for data storage.

Figure 14: Configuration parameters related to the PUT operation of the Advanced FTP node

To send the file stored in the local directory, simply use a file reading function on a function node, on the output of the Advanced FTP node (figure 16). The return of the file will be a payload in string format with the contents of the file, highlighted (figure 15).

Below is the code used to read the file contained in the function node.

return { payload: context.global.fs.writeFileSync(msg.localFilename) }

As a return, a payload with the status of the operation will be presented, it can be a positive return: "PUT operation successful", or a negative return: "PUT Failed". To view the node return message simply place a green debug node on the output of the Advanced FTP node, as seen in figure 15.

Figure 15: Highlight the return of PUT OPERATION

1.2.2 – SFTP

The main difference between using the SFTP server for FTP is the issue of security when transferring a file. To configure a server on the platform we will use the library "@ xst/node-red-contrib-sftp", it allows us to perform some important functions in an SFTP area, which are: viewing, transferring and delegating files. The main operations are:

• LIST

• GET

• PUT

• DELETE

For more information about library usage and how to configure nodes, visit the link: https://flows.nodered.org/node/node-red-contrib-sftp

1.2.2.1 – SFTP – Configuration

To configure the node, simply access drag the node to the Logic Builder free desktop and click on the node (figure 16).

Figure 16: Highlight SFTP node settings parameters

To configure the node it is necessary to fill in the fields:

• Server: fill in the credentials of the chosen server. It is possible that the user configures the access of a server of his choice or the server of the thingable!, to hire the use of our server contact support requesting an access to the SFTP area.

When accessing the pencil on the right side of the "Server" field, a window will open with the information regarding the SFTP server that must be parameterized (figure 17). The main ones are:

• Host: Server Address

• Port: Server Port

• Username: Server access user

• Password: Password to access the server

Figure 17: Configuration parameters for SFTP node server

• Method: define which function will be performed by that node, the main functions have already been described above: LIST, get, PUT and DELETE.

1.2.2.2 – SFTP – Operations

1.2.2.2.1 – LIST

To use a LIST type function, simply select the previously configured server (figure 17), select the "Method" field as "LIST" and in the "Remote Path" field choose the directory that will be searched in the SFTP area (figure 18). As a return, there will be a payload in the form of an array with the list of files and folders found in the parameterized directory (figure 19). To view the node return message simply place a debug node on the SFTP node output, as seen in figure 19.

Figure 18: Configuration parameters for SFTP node LIST OPERATION

Figure 19: The return of the LIST operation is highlighted

1.2.2.2.2 – GET

To use a GET type function, simply select the previously configured server (figure 17), select the "Method" field as "GET' and in the"Remote Path"field choose the file you want to download. In your configuration it is necessary to pass the directory+ file field. In the "Local Path" field, it is parameterized to which directory this downloaded file will be stored. For this SFTP node, it is not mandatory to configure this directory, and the contents of the files read are also available in the format of a buffers array in the "msg.payload" object. If you want to work with stored files use the "/tmp" directory.

This node has the differential of allowing the download of numerous files in a single request. Simply pass a function node, previously connected to the SFTP node, with a function describing an array of objects, in which the objects are the directory files (figure 21).

Below is the example code:

   let list = [“arquivo1”, “arquivo2”, “arquivo3”]
    return { remoteFilePath: list }    

Figure 20: Configuration parameters for SFTP node GET OPERATION

As a return, there will be a payload with an array containing a reading buffer of the files downloaded from the chosen directory. To view the node return message simply place a green debug node on the SFTP node output, as seen in figure 21.

Figure 21: Highlight the return of the GET OPERATION

Whenever possible choose to use SFTP servers and the GET method by fetching multiple files at once without writing the files to the “tmp” folder, using only data output in buffer format. In this way, your integration becomes faster, more efficient and safer.

1.2.2.2.3 – DELETE

To use a function of type DELETE simply select the server previously configured (figure 17), select the field "Method" as "DELETE" and in the field "Remote Path" choose the file you want to do the delete operation, to delete it from the selected directory. In its configuration it is necessary to pass the directory field + the file (figure 22).

Figure 22: Configuration parameters related to the DELETE operation of the SFTP node

As a return there will be a payload with the status of the operation. To view the node return message simply place a debug node on the SFTP node output, as seen in figure 23.

Figure 23: The return of the DELETE operation is highlighted

Passing the file name to be deleted because it is dynamically done by the "msg" object. RemoteFilePath".

1.2.2.2.4 – PUT

To use a PUT type function, simply select the previously configured server (figure 17), select the "Method" field with "PUT' and in the"Remote Path"field choose the target directory of the file, in the format directory + file name (figure 24). In the "Local Path" field, it is parameterized which location the file will be searched for, to be sent in the directory chosen in the "Remote Path", in this case the director "/tmp".

All going well we will have the return of success: 'Successfully PUT'. To view the node return message simply place a debug node on the SFTP node output.

Figure 24: Configuration parameters regarding the PUT operation of the SFTP node

1.3 – Publish data on the thingable platform!

For the specific case of data coming from an FTP/SFTP server, it is necessary to read this data and format the resulting buffer in json messages, which must follow the standard of the data schema used in the device that will receive this data. For more information about data schema see our targeted documentation Metadata and Schema.

The information coming from the FTP/SFTP server can come in either binary or ascii format, but in the end it basically consists of text. Below is an example of a return message from the server:

Figure 25: Return on SFTP node output

It is necessary to format the above data in the following format (json), using the javascript language to configure the function node:

Figure 26: Specific format for sending data to the MQTT device

• deviceType: Parameter referring to the type of device. It is configured at the time of registering the device. In the example, in figure 26, "TYPE".

• deviceld: Parameter referring to the "serialNumber". It is configured at the time of registration of the device.In the example, in figure 26, "STATION 04".

• payload: Object that will send the data to the equipment. In the example, in figure 26, team, PP, BATe amp.

The following is an example flow for reading data at the output of a get node and formatting it in a valid json message:

data = msg.payload[0].split(';')
deviceId = data[0].split(":")[1]
deviceType = "TYPE"
day = data[1].split(":")[1].slice(4,6)
month = data[1].split(":")[1].slice(2,4)
year = "20" + data[1].split(":")[1].slice(0,2)
hour = data[2].split(":")[1].slice(0,2)
minute = data[2].split(":")[1].slice(2,4)
second = data[2].split(":")[1].slice(4,6)
time = new Date(year, month-1, day, hour, minute, second)
PP = data[3].split(":")[1]
BAT = data[4].split(":")[1]
AMP = data[5].split(":")[1].split(")")[0]
 
return {
    deviceType,
    deviceId,
    payload:{ time, PP, BAT, AMP}
}

Note: Remembering that the above code is an example of a specific data treatment in "javascript" for an "array" of objects that contains a "string". The important thing is to keep the structure of the returno with the information of "deviceType", "deviceid" and "payload" to publish on the device when they are configured in the node "ibm iot out" (figure 28).For more information about the device, access the document regarding the creation of devices and also about the creation of schemas Create a Device Metadata and Schema.

Figure 27: Coding in the "Logic builder" to publish the data to the platform by MQTT

IBM IoT OUT node configuration:

Figure 28: "ibm iot out" node settings parameters

• Authentication: Form of authentication to be used. For our application choose "API Key".

• API Key: API Key registration to be used. For more details on this key see section obs1 at the end of the document.

• Output Type: has the following options: "device command" and "device event". In this case, "device event" is used to publish the data.

• Device Type: msg.deviceType to receive the device type parameter, or the type of device that will receive this message can be written directly.

• Device Id: msg.deviceid to receive the device ID parameter from the device or you can write directly to the serialNumber that will receive this message.

• Format: json. Format of the message that is received by name.

• QoS: Quality of Service. In it, it is parameterized how the receipt of the MQTT message will be. By default it is kept at 0.

To verify that the data arrived on the device correctly, simply check in the "Device Manager" tool, in the options column, in the network icon "Device Events" (figure 29). It will have the return in "event" of what was parameterized to send in msg.payload (figure 26).

It is important to note that in order for SCADA to correctly identify the information sent in "msg.payload" it is necessary that the json structure correctly obeys the key structure and type in the values defined in the schema of the device identified in "Device Type" and "Device Id".

For the example covered in this tutorial we have a set of objects: time, PP, BAT and AMP. Below is the correct way to publish the data in the SCADA according to the mentioned case:

{"time":"2022-08-08T23:05:00.000Z","PP":"0.0","BAT":"12.80","AMP":"0"}

For more details on how to configure a database access the document Metadata and Schema

Figure 29: Access for “Device Events” functionality

Figure 30: Access to the "Device Events" functionality

Note 1: By default, on thinkable! the "ibm it out" node is already configured with the API Key corresponding to the environment you are working on, if you are in doubt whether the credential is valid click on the credential edit menu to check the details, or create a new one in the Add new ibm iot option. If it is necessary to create a new credential configuration, it is necessary to pay attention to the fact that Logic Builder is a application, therefore requires application credentials (which are different from device credentials, available when we create a new one), these credentials can be obtained from the IBM Watson Credentials tab within the Services Configurator.

Figure 31: Where to get the application credentials

Note 2: If any of the fields in "IBM Watson Credentials" are empty, consult the thingable support!

Authors: Pedro Henrique

Angelo Guimarães