Guide to implementing standard interfaces
This guide is for developers intending to add HMIs for custom hardware or designing interfaces for network-enabled devices, particularly for low-spec hardware.
Checklist / Requirements
This document assumes that there are certain basics / best practices already implemented :
- Consistent calling convention
- Consistent error codes and handling across the entire program
- The program’s public interface uses minimal, distinct, and consistent function signatures
- Shared brokers or gateways
- Consistent and simplified lifecycle methods, eg: Addon::setup, Addon::loop, Addon::destroy
- Handling of IO timeouts and retry
- Message resilience for subsequent devices / IO
These basics make further binding to a protocol or intermediate abstraction relatively easy.
Protocols
This chapter is dedicated to the implementation of popular protocols. Some of these ideally don´t require additional changes in the application, and some can be supported by minimal tweaks or sufficient documentation about how to acquire those (The goal of this guide is to support all of them with minimal effort).
A minimum yet practical protocol should be chosen wisely and upfront. A custom protocol often comes with penalties that are hard to document, communicate, and to support unless this is part of the business, but as often seen, it does not pay out for the vendor. In many cases, a protocol as Modbus is sufficient since there are lots of existing client software one can refer to. It makes it easy to test the product without noise as custom client implementations. Other than software to test and be used as a reference, there should be enough off-the-shelf hardware to extend wire lengths and convert it to other protocols (there is a big market around Modbus or Canbus). The choice must be part of the product design and business strategy.
1. Serial (custom messaging and data structures)
The protocol (rfc1963) is being widely used for applications that need only a short cable and can be found in 3D-Printers, HiFi amplifiers and even larger machines as older CNC mills or lathes - mostly for configuration and servicing tasks. Due to its limitations in cable length (2.5m) and transport, it’s not meant to build larger networks, especially in environments with electromagnetic emissions (EMI), since data transmission is realized using low voltages, prone to external EMI (5V - 12V). The protocol supports basic parity checks (YTube - Ben Eater, but more reliable CRC checks have to be built upon in the application. Please see more about on YTube - Ben Eater.
Implementation
In the case of a simple application, it´s relatively easy to parse incoming messages and then call the right functions, but this can be fast and unmanageable as soon as the API grows.
To invoke global functions or class member functions from incoming strings, eg: MyClass::MemberFunction::Argument, a registry with function pointers can be used. This method works sufficiently enough as long function signatures don´t differ too much from each other since for each different signature, a function pointer has to be declared. Using reinterpret cast to cast a void* pointer (isocpp) is possible but can leave the application quickly in an undefined state. This is because some architectures and suited compiler features don´t support advanced casting.
Example of a class::fn registry (This is proof of concept code, don´t use this in production!)
Recommended data structure for requests
To facilitate debugging and human-readable messaging (ASCII code as GCode). The following message structure turned out to be reliable and versatile.
<< id,verb,flags,version,payload >>>
Encapsulating it with a prefix ‘<<’ and suffix ‘>>’ helps to distinguish these messages from other data on the same connection (ie: debug statements as Serial.print)
-
id: optional, as in TCP or other high-level protocols, this is a sort of transaction id to track the message through its lifecycle which enables to detect of dropped or lost messages -
verb: optional, the type of the call, application specific, but in more complex scenarios, this designates the message type. In particular, these types have been practical (security, compilation and implementation wise):
enum ECALLS
{
// global function
EC_COMMAND = 1,
// addon method
EC_METHOD = 2,
// external function
EC_FUNC = 3,
// user space (addons)
EC_USER = 10
};
flags : optional but useful for prototyping and development. Flags that can be used to alter the response and command behavior
enum MessageFlags
{
// local flag for the recipient
NEW = 1 << 1,
// local flag for the recipient
PROCESSING = 1 << 2,
// local flag for the recipient
PROCESSED = 1 << 3,
// sender flag, turn on debugging
DEBUG = 1 << 4,
// sender flag, instructing the recipient to send a receipt
RECEIPT = 1 << 5
};
version : optional but recommended as soon it’s clear that the product may go through different revisions. This can avoid issues when backporting hotfixes, despite firmware ideally should expose its version and compilation configuration via initial handshake or explicit call. Doing this on a message level also enables easier testing for different implementations per version.
payload : The actual data. In the case of bound class functions, eg: <<1;2;0;1;Power:on:1>>. It should be noted that using strings on low-spec devices comes with memory problems but also often missing basic implementations of string functions. This document will explore later on more optimized invocation methods using numbers only (as seen in Modbus).
Remarks
- Serial communication can be traced and debugged on modern scopes and with software(kernel, DLL hooking, or WinUSBCap). Obfuscating can be archived by hashing but will require significantly more resources; yet, it’s a matter of time to tap into it.
- The concepts above are only for educational purposes. Simple applications such as serial invocations of commands (GCode) can perfectly deal with simple return messages echoing the request (as in Modbus). However, it seemed a reasonable introduction to the nuts and bolts of advanced messaging, mandatory requirements for the next chapters.
- SO : Messing with member function pointers, SE : Fishy reninterpret_cast
2. Modbus RTU over RS485 (2 wire)
To implement Modbus, please consult first the Specification and Implementation Guide from Modbus.org - PDF
The challenges begin when multiple devices have to be managed on one device(often an I/O controller).
2.1. Example : Print head
This utilizes a VFD and PID controller via Modbus-RTU-RS485 and other components via analog or digital signals. Low level I/O handling happens ideally on the I/O controller. The ‘Remote Manager’ acts as client and consumes a high level interface via Modbus-TCP.
+--------------------+
| |
| Remote Manager |
| |
+---------+----------+
|
|
| Modbus RTU / or raw TCP
|
|
+---------+----------+
| |
| I/O Controller |
| |
+--+------+----------+
| |
| | Modbus RS485 (2Wire)
| |
+--------------+ | | +-----------------------+
| | | | | |
| Sensor +---+ +----+ Device Nr. 1 (VFD) |
| | | | | |
+--------------+ | | +-----------------------+
| |
+--------------+ | | +-----------------------+
| | | | | |
| Stepper +---+ +----+ Device Nr. 2 (PID) |
| | | | |
+--------------+ | +-----------------------+
|
| +-----------------------+
| | |
+----+ Device Nr. 3 (PID) |
| |
+-----------------------+
The VFD and the PIDs have their own vendor-specific interfaces. The controller reads and writes relevant registers to the application and then exposes a new interface on Modbus TCP or plain TCP sockets.
Most Modbus implementations offer only a very slim API to query a device(slave). A query (or request) can be a read- but also a write - command, one at a time. The application’s responsibility is to manage these queries in an ordered fashion. For this, a message queue is being utilized. A message queue can be shared among different owners (classes) where all subscribers can queue new requests.
The organization of functionality (classes and functions) should be designed and optimized for its usage. Equal devices should be grouped and managed by a parent class to assist, for instance, in broadcasting (eg: all on or all off). Furthermore, it’s wise to consider that multiple but equal devices will be added over time.
In our example, we’re using an unknown number of PID controllers and exposing them in a managed interface:
class TemperatureControl : public Addon, ModbusClient {
OmronPID pids[NUMBER_OF_PIDS];
stop();{ each pids => p => pid->stop} // stop all PIDS
run(); {each pids => p => pid->run} // run all PIDS
isHeating(); // returns true if any of the PIDs is heating, used for a feedback LED on the control panel
Modbus *rs485; // the Modbus RS485 gateway
Mudbus *modBusTCP; // the Modbus TCP gateway
}
This way, the OmronPID class doesn’t need to know anything about Modbus and acts merely as storage for values.
Printhead register design
The more important information as status, state, etc… should be easy to fetch using the READ_HOLDING_REGISTER Modbus command. The command accepts a number to specify an address range. The very range should be kept small to preserve bandwidth and avoid collisions. Populating the first 10 registers is a common practice.
Print head holding read registers
// System Globals
- 0x0 : This is normally unused
- 0x1 : System state : RUNNING|STOPPED|POWERED|ERROR - any sub device activity will raise the flags - used for feedback and main control
- 0x2 : Last Error Code
- 0x3 : Reserved
- 0x4 : Reserved
// Sub systems or devices
- 0x5 : VFD - State : RUNNING|STOPPED|ACCELERATING|DECELERATING|ERROR
- 0x6 : PID 1 - State : RUNNING|STOPPED|HEATING|ALARM|ERROR
- 0x7 : PID 2 - State : RUNNING|STOPPED|HEATING|ALARM|ERROR
- 0x8 : PID 3 - State : RUNNING|STOPPED|HEATING|ALARM|ERROR
- 0x9 : System Sensor Flags : ENCLOSURE_OPEN|OVERHEAT|...
That’s it, 10 registers with 2 bytes each - ideal to be fetched by clients frequently. Any less frequent requested details are placed in higher addresses. Those are ideally grouped in similar register pages:
// Holding registers 0xA (10) - 0xE (14) are used for the VFD details
- 0xA : VFD - Last Error Code
- 0xB : VFD - Target Frequency
- 0xC : VFD - Current Frequency
- 0xD : VFD - Current (Amperage)
- 0xE : VFD - Reserved
To gain more flexibility about address ranges for such groups, offsets, and macros can be used :
#define SYSTEM_MAIN_START 0x0
#define VFD_MAIN_OFFSET SYSTEM_MAIN_START + 10
// VFD HOLDING READ REGISTERS (0x03)
#define VFD_RREGISTER_LAST_ERROR VFD_MAIN_OFFSET + 1
Depending on the system size, these registers might go under duplicate checks, preferably at compile time. For C++, several macros are implemented to provide array access- at low cost. See ./examples/commons/macros.h#85. Those macros can also assist for a safe pin assignment.
Commonly used address range assignment for read registers:
Read Holding Registers Poll Priority Address Ranges
+-----------------------------+------------------+------------------+
| Main System Monitors | | |
| | 1 | 0 |
| Recommended size : 10 | | |
+------------------------------------------------+------------------+
| Main System Detail Registers| | |
| | 2 | 0x10 |
| Recommended size : 10 | | |
+------------------------------------------------+------------------+
| | | |
| Sub System 1 Main Monitors | On Demand | 0x20 |
+------------------------------------------------+------------------+
| | | |
| Sub System 2 Main Monitors | On Demand | 0x30 |
+-------------------------------------------------------------------+
| | | |
| Main Advanced Level | On Demand | 0x1000 |
| | | |
+-------------------------------------------------------------------+
| | | |
| Sub System Advanced Level | On Demand | 0x2000 |
| | | |
| | | |
| | | |
+-----------------------------+------------------+------------------+
- Message queue overview
- Race conditions
- Split feedback and control via read and write registers
Example References
Remarks
- Modbus for Dummies - YT
- All You Need to Know About Modbus RTU - YT. Watch his other videos, incl. the justified rant to sell MQTT as the new IoT (aka Internet of sh** - TW).
- Serial Communication RS232 & RS485 - YT
3. Modbus RTU - TCP
4. TCP (Custom)
After implementing Modbus RTU for TCP/Serial, a custom TCP protocol can be added for clients who don’t have a Modbus implementation (eg: ABB and other robot systems). This enables the inheritance of error codes & message resilience. It’s becoming a matter of documentation only.
In example, our simple print-head firmware manages RS485 devices (ie: VFD & PID controllers) and proxies them via Modbus-RTU on TCP using a simple server implementation (see ‘./examples/modbus-tcp’).
To determine the exact byte sequence to be sent to the I/O controller, a simple Modbus client software can be used :
In this particular case, we’re setting the target temperature of a PID controller to 100, using address 17, eg: 01 06 00 11 00 64 D8 24
01: slave id06: Modbus verb / function code, in this case WRITE HOLDING REGISTER11: address (17)00 64: value (100), 2 bytesD8 24: CRC, 2 bytes. Since it’s TCP, this isn’t evaluated and can be ignored, see TCP Specs
As next, a packet tracer is used to see what is actually send over the TCP wire and thus the TCP protocol overhead can be determined.
The image shows where the Modbus message begins and ends. Wireshark also enables investigating of preceding bytes and helps to understand those.
d4 95 00 00 00 06 01 06 00 11 00 64
To fake a Modbus message, all we need is 01 06 00 11 00 64, but we also have to prefix it with the TCP overhead (d4 95 00 00 00 06)
|-- TCP Overhead----- | -------- Modbus ---- |
d4 95 00 00 00 06 | 01 06 00 11 00 64
In example, we can send this now via Hercules to emulate a Modbus Message over TCP.
The TCP overhead (d4 95 00 00 00 06) is created as follow:
d4 95: Transaction identifier, 2 bytes00 00: Protocol identifier, 2 bytes00 06: Length of the message, 2 bytes
Viola
5. MQTT
6. Via Proxy : Profibus
7. Via Proxy : CAN
Preface
- State/register race conditions
- Wiring (max. cable length, max. devices)
- High spec protocols & low spec devices
- External protocol proxies
- Documentation and custom firmware configuration
- User space (extension points & callbacks)
- Security
- Feeback & debugging
Abstractions
- IO state persistence across different protocols
- States
- Registers and custom address mapping
- Calling conventions (incl. user function to select responses)
- Message queues