Documentation

The page describes to you how to use Embedded Proto; generate code based on *.proto files, access and set message variables and serialize the data. If you are looking for more elaborate examples you will find them on the Examples page. When your intention is to better understand the inner workings of Embedded Proto of start developing for the library you will find the source code documentation here.

1. How it works

You define the structure of your message in a .proto file. This .proto file is the universal language in protocol buffers. Click here to find more information on all the possibilities.

message request 
{
  uint32 A = 1;
  int64 B = 2;
  // More variables.
}

The .proto files are the input for protoc, the protocol buffer’s compiler. Protoc is a source code generator rather than a compiler. It converts the messages defined in the .proto files into the source code of the programming language you select. You can use this generated code in your project.

Protoc natively supports many different programming languages. You and a colleague working in a different programming language can use the same message definitions. When either one updates the message definition, the other person just reruns protoc and gets the latest source code.

Natively however protoc generates C++ code not suitable for micro controllers. The C++ generated is written for server and desktop processors. This is where Embedded Proto offers an alternative.

Embedded Proto is a plugin for protoc generating C++ code suitable for micro controllers. In this way Embedded Proto provides an easy to use interface to exchange data between embedded devices and the out side world. Specifying the data format between your IOT device and other devices, servers, apps or desktop applications in a standardized way.

2. Supported Features

The two tables below indicate the level of support for various variable types and features.

All features mentioned above are of version proto3. At this moment proto2 is not supported by Embedded Proto. With regard to proto2 support google states the following:

Prefer proto3 while proto2 will continue to be supported, we encourage new codes to use proto3 instead, which is easier to use and supports more languages.

For this reason, it is unlikely that Embedded Proto will support proto2 in the future.

3. Installation

What is required to be able to generate source files based on .proto files:

1. Python 3.6 1.
2. Pip
3. Virtualenv
4. Protobuf 3.6.1
5. Git if you do not have it already.
6. CMake 3.10.2 (only required to build the PC unit tests)

Install the required software and continue with checking out the repository. For PC unit testing gtest is used which is included as a submodule of this repository. If you intent to run the PC unit tests of Embedded Proto it is suggested that you pull in the submodules as well.

3.1 Linux

Install the required software and continue with checking out the repository. For PC unit testing gtest is used which is included as a git submodule. If you intent to run the PC unit tests of Embedded Proto it is suggested that you pull in the submodules as well.

git clone --recursive URL_TO_EMBEDDED_AMS

Next enter the folder and in that folder create a virtual environment called venv:

virtualenv venv

Activate the virtual environment

source ./venv/bin/activate

Besides the list of tools above additional python packages are required. These are listed in the requirements.txt file in this repository. It is advised to install these required packages using pip in a python virtenv. You can however install the requirements globally. To install the packages run the command:

pip install -r requirements.txt

You can now use the Embedded Proto protoc plugin in your projects. You are also ready to build the PC unit tests if you have installed CMake.

3.2 Windows

Clone the repository using your favourit git tool.

From the Protocol Buffers website download the desired release of protoc. Mind that the source is indicated with protobuf-XXX-A.B.C.zip. You are looking for protoc-A.B.C.-win64.zip. Unzip the file and install it according to the Readme file.

Next open up powershell and go to the Embedded Proto folder.

cd C:\some\dir\embeddedproto

If not already installed, install virtualenv using pip3.

pip3 install virtualenv 

Next create a virtual environment called venv:

virtualenv venv

Activate the virtual environment

.\venv\Scripts\activate

You should now see in your console the addition of (venv) in front of your location.

Next we will install all the python packages required for the plugin. These packages are contained by the virtualenv and will not interfere with other installations. The requirements file lists all packages to be installed using pip3.

pip3 install -r requirements.txt

You can now use the Embedded Proto protoc plugin.

At this time building the unit tests under Windows is not supported.

4. Generating source code

When working on your project you write your proto files defining the message structure. Next you would like to use them in your source code. This requires you to generate the code based upon the definitions you have written. This is done using our plugin for the protoc compiler protoc-gen-eams.py. To generate the code use the following command:

protoc --plugin=protoc-gen-eams -I./LOCATION/PROTO/FILES --eams_out=./generated_src PROTO_MESSAGE_FILE.proto

What happens is that protoc is tolled to use our plugin with the option --plugin. Next the the standard option -I includes a folder where your *.proto files are located. The option --eams_out specifies the location where to store the generated source code. Finally a specific protofile is set to be parsed.

As our plugin is a Python script and the protoc plugin should be an executable a small terminal script is included. This terminal script is called protoc-gen-eams and is used to execute python with the Embedded Proto python script as a parameter. The main take away is that this script should be accesable when running your protoc command.

After running protoc without any errors the generated source code is located in the folder specified by ---eams_out. This folder is to be included into your project. This is not the only folder to be included. The generated source files depend on other header and source files. These files can be found in EmbeddedProto/src. You are thus required to include this folder as well in you toolchain.

Various examples how to use and integrate Embedded Proto in your project are given in the Examples section.

5. Using a message

In the following sections we will look into how to use the generated message code in your own applications. We will look into setting and retrieving data from the message objects for various data types.

5.1 Basic variables

There are a number of basic numeric variables possible in Protobuf: floating point values and integers both signed and unsigned. Also a simple boolean is available. There are various methods in which these types can be serialized, such as base 127, ZigZag and fixed length. To accommodate these various methods Embedded Proto warps the base types in a class , a Field class.

The Field class is used to wrap the basic variable type together with functions to serialize and deserialize the value. Various operators are overloaded in these classes to easily perform operations on the base type, such as assignment and comparison. Take for instance the examples below:

// A field object.
EmbeddedProto::int32 int_field;

// A normal scalar variable.
int32_t normal_variable = 1;

// Assigning the scaler to the field object.
int_field = normal_variable;

// A field and scaler comparison.
if(int_field == normal_variable) 
{
  // etc.
}

When a field is integrated into a message class a number of functions are provided by that class to access the field. Assume we have the following message named Foo:

message Foo
{
  int32 Bar = 1;
}

The message holds one variable named Bar. When generating code for the message Foo various functions are created to access the member Bar. The interface for getting, setting and clearing Bar is depicted below:

class Foo
{
  public:
    EmbeddedProto::int32::FIELD_TYPE Bar() const;
    EmbeddedProto::int32::FIELD_TYPE get_Bar() const;
    void set_Bar(const EmbeddedProto::int32::FIELD_TYPE& value);
    void set_Bar(const EmbeddedProto::int32::FIELD_TYPE&& value);
    void clear_Bar();
};

As you can see there are two functions to obtain a constant copy of the value using the Bar() and get_Bar() functions. The set_Bar() function is executed in two versions, one with the normal reference and one with an rvalue reference. The last function is used to clear the value back to the protobuf default. At the moment of writing Embedded Proto is implemented for Proto3 meaning that all variable types are set by default to zero.

5.2 Nested messages

It is possible not just to have basic variables in a message, nested messages are also possible. Take as an example the two messages below:

message Bar 
{
  // Some variables.
  int32 x = 1;
}

Message Foo
{
  Bar nested_bar = 1;
}

The message Foo uses an instance of a Bar message as one of it’s variables. To access the variables in nested_bar the message Foo will generate the same functions as for a basic variable with the addition of one. For nested messages also a non const mutable function will be available.

Bar& mutable_bar();

With this function it is possible to access the non const setter functions of the class Bar like set_x() and clear_x().

5.3 Repeated fields

In micro controllers dynamic memory allocation might cause problems. When running bare metal code it is hard to catch exceptions caused by insufficient memory and other related errors. Repeated fields in Protobuf are by default of unknown length. Their implementations thus often make use of dynamic memory allocation.

In Embedded Proto this is not the case. Repeated fields are implemented as array’s with a static length. The array size is passed to the message by means of a template parameter. As an example we take the following .proto message definition:

message Foo
{
  repeated uint32 y = 1;
}

The class this will generate a template parameter with which you can set the size of the message. The definition of the class and the array object for variable y looks somewhat like:

template<uint32_t y_SIZE> class Foo
{
  private:
     ::EmbeddedProto::RepeatedFieldSize<EmbeddedProto::uint32, y_SIZE> y_;
};

To access the array and the elements in it various functions are generated in the message.

void add_y(const EmbeddedProto::uint32& value);
void set_y(uint32_t index, const EmbeddedProto::uint32& value);
void set_y(uint32_t index, const EmbeddedProto::uint32&& value);

::EmbeddedProto::RepeatedFieldSize<EmbeddedProto::uint32, y_SIZE>& mutable_y();

const EmbeddedProto::uint32& y(uint32_t index) const;
::EmbeddedProto::RepeatedFieldSize<EmbeddedProto::uint32, y_SIZE>& get_y() const;

void clear_y();

Setting values in the array can be done by adding data to the end of the array with add_y() or at a given index with set_y(). It is also possible to set the data via the mutable function mutable_y(). This function returns the whole array as a non constant reference.

This function could also be used to just retrieve data but it is better to use the const version in that case by calling get_y(). Accessing individual elements is also possible using the function y(). Please not that you have to check yourself if your index is within bound of the array size.

Finally you can clear the whole array with the clear_y() function.

5.4 Strings

Just like the repeated fields, String fields are of fixed length in Embedded Proto. You specify the length by means of a template parameter when creating the message object. Lets take a look at a simple example:

message text
{
  string txt = 1;
}

The code generated with hold a field object with the type FieldString which has a template parameter to specify the maximum length of the string. In the FieldString object an array is specified of characters. This is done to be able to work without dynamic allocated memory. This in contrast to std::string which does use dynamic allocation.

template<uint32_t txt_LENGTH>
class text final: public ::EmbeddedProto::MessageInterface
{
  private:
    ::EmbeddedProto::FieldString<txt_LENGTH> txt_;
}

The functions to access and manipulate the FieldString contains functions to obtain the character array and set the string.

FieldString<MAX_LENGTH>& operator=(const char* const &&rhs)
Error set(const DATA_TYPE* data, const uint32_t length)
const DATA_TYPE& operator[](uint32_t index) const

A working example could look like this:

// Define the message object specifying the maximum number 
// of chars in the array. Ten in this case.
text<10> msg;

// Set the string by means of assignment.
msg.mutable_txt() = "Foo Bar";

// Change the capitals to lower case by means of indeces.
msg.mutable_txt()[0] = 'f';
msg.mutable_txt()[4] = 'b';

// Acces the char array.
const char* text = msg.get_txt().get_const();

// Obtain the current number of chars in the string
const uint32_t N_chars = msg.get_txt().get_length();

// It is also possible to obtain the maximum number of chars 
// this string can hold. This would always return the value 
// you have set as the template parameter.
const uint32_t max_N_chars = msg.get_txt().get_max_length();

// You can clear the whole string with the clear function
msg.mutable_txt().clear();

5.5 Bytes

The field type bytes may contain any type of uint8_t data. It is an uniform data container. It works similar to String fields in that the FieldBytes class requires you to specify the maximum number of bytes which will be stored in the array using a template parameter. Also the same type of access functions are available. Lets take a look at an example:

message raw_bytes
{
  bytes b = 1;
}

// Declare the message object stating the bytes array 
// should hold ten bytes.
raw_bytes<10> msg;

// Set some elements, others are by default zero.
msg.mutable_b()[0] = 1;
msg.mutable_b()[9] = 10;

// There is no index out of bound exception as they 
// are not possible on small mcu's. Instead the last 
// element of the array will be returned and changes!
msg.mutable_b()[10] = 11;

// Assign the content of a std array.
const std::array<uint8_t, 10> data = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
msg.mutable_b().set(data.data(), 10);

// Loopoing over the array could be done like this
for(uint8_t i = 0; i < 10; ++i) {
  const auto x = msg.get_b()[i];
}

// And you can clear the whole array with the clear function.
msg.mutable_b().clear();

5.6 Enumerations

Enumerations are fully supported by Embedded Proto. They work just like regular C++ enumerations. An enumeration defined in a .proto file as given:

enum Test_Enum 
{
  ZERO  = 0;
  ONE   = 1;
  TEN   = 10;
}

Is simply converted to the following C++ code:

enum Test_Enum
{
  ZERO = 0,
  ONE = 1,
  TEN = 10
};

Accessing enum values is done using the same functions as basic variables.

5.7 Oneof

Oneof’s are really powerful in saving RAM in your mcu. If you have a message where you will only set one of multiple variables at a time the oneof field will only allocate memory for the largest variable of the lot. The Embedded Proto implementation makes use of an union to do this. This means it is only possible to use one of the variables at a time.

Lets take a look at an example:

message Foo 
{
  oneof Bar 
  {
    bool active = 1;
    int32 count = 2;
    double value = 3;
  }
  // Other variables
}

This message has an oneof named Bar. This oneof holds three variables of different sizes, respectively one byte, four and eight. Because of the union the C++ code will only allocate eight bytes.

Please note that there is some overhead to an oneof. It is required to store which variable is currently in use. This is done via the field tag value. Each message generated by Embedded Proto has an enum class called id. This enumeration holds a list of all variables defined in the message together with their field tag. Each oneof has an accompanying variable of the type id. accessible via the which function.

enum class id
{
  NOT_SET = 0,
  ACTIVE = 1,
  COUNT = 2,
  VALUE = 3
};

id get_which_Bar() const;

Each time one of the variables in an oneof is set the which variable changes. This holds true also when deserializing a messages. Allowing you to make a switch statement to select which value to get from the object.

5.8 Optional fields

As of protobuf version 3.15 optional fields have been reintroduced in proto3. This feature allows you to track if a field has been set. Even if the field is set to it’s default value. Marking a fields as optional will also serialize default values when a field is set. This in contrast to a regular field which will not be serialized when it’s value equals to the default.

Lets take a look at a simple example:

message Foo 
{
  optional int32 count = 1;
};

Here we marked an integer as optional. This will add tracking of the fields presence. To check if a field has been set, even to it’s default value, you can use the has_xxx() function:

Foo msg;
msg.set_count(2);
if(msg.has_count()) 
{
 // Do someting with the count field. 
}

// Set to default value
msg.set_count(0);
// Still true even thow count equals zero.
bool count_is_set = msg.has_count();

Even when you set the field to zero, the default for an integer, it is still marked as present. This will result in the field being serialized into the byte stream. This would not happen with regular fields not marked as optional.

// Set to default value
msg.set_count(0);
// Still true even thow count equals zero.
bool count_is_set = msg.has_count(); 

// Count will show up in the serialized data.
msg.serialize(buffer);

To clear the presence of a field, you call the clear_xxx() function of that field. This will reset the value of the field to it’s default.

msg.clear_count();
// Returns false
bool count_is_set = msg.has_count(); 

The table below states an overview of the effect the optional keyword has on different types of fields.

§ The google implementation tracks the presence of nested message fields when memory is allocated for that field. In Embedded Proto no memory is dynamically allocated and the presence of nested message fields is not tracked unless the fields is marked as optional.

† The presence of an oneof field can be tracked using the which_xxx() function.

6. Packages

Just like the Goolge C++ implementation of Protobuf, package translate into namespaces. When defining a message is a package, the generated message class will be scoped a namespace with the same name as the package.

Lets take the following .proto where we have placed a simple message in a package:

package foo.bar;

message Dummy
{
  int32 data = 1;
};

The message Dummy will in this case be placed in the namespace foo::bar. An object instantiation of a Dummy message would now look like:

foo::bar::Dummy my_dummy_message;

7. Serialization

The data in a message has been set and it is ready to be transmitted over your communication line. So how you serialize the message?

Each message object has a function serialize() derived from one of it’s base classes. When calling this function the data in the message is serialized using the Protocol Buffers format. This function is similar to the Protobuf function SerializeToString()of the C++ implementation.

However there is a difference between the two. Where SerializeToString()outputs into a std::string this is not suitable for embedded implementations. The standard string used dynamic memory allocation which is something we are trying to avoid.

This is where the interface class WriteBufferInterface comes in. This base class is passed to the message serialzie() function. It is used to set the output in. But as the name suggests it is an interface. The library provides no actual implementation for this class.

It is up to the user of this library to write an implementation of WriteBufferInterface specific for their situation. Implementations could for instance be a simple array or you could write the data directly to a uart bus. The interface is simple and ealisy implemented. You can find more documentation on the required functions here.

8. Deserialization

With respect to desrialization the almost same applies as for serialization. Each message generated has a deserialize() function which takes an object derived from ReadBufferInterface as input.

Again the user of this library is to write an implementation for the ReadBufferInterface, tailored to there needs. The class should hold the whole serialized message as received over your specific communications method. For instance it could be all the packages combined after you received the data over TCP. If you need more documentation on the ReadBufferInterface class you can find it here.