Repeated fields

In microcontrollers, dynamic memory allocation might cause problems. When running bare metal code it is hard to catch exceptions caused by insufficient memory and other related errors. Repeated fields in Protobuf are by default of unknown length. Their implementations thus often make use of dynamic memory allocation.

In Embedded Proto this is not the case. Repeated fields are implemented as arrays with a static length. The array size is passed to the message by means of a template parameter (option 1) or through a custom option in the .proto file (option 2). We will discuss both in the following paragraphs.

Option 1: Set the length using a template parameter

As an example we take the following .proto message definition:

message Foo
{
  repeated uint32 y = 1;
}

The class this will generate a template parameter with which you can set the size of the message. The definition of the class and the array object for variable y looks somewhat like this:

template<uint32_t y_SIZE> class Foo
{
  private:
     ::EmbeddedProto::RepeatedFieldSize<EmbeddedProto::uint32, y_SIZE> y_;
};

Option 2: Set the length in the .proto file

As of version 3.0.0, it is also possible to fix the length of a repeated field in the .proto file. This also applies to String and Bytes fields. You will not get the template option but an array of the size you specify in your .proto file. This is done by using a mechanism called custom options provided by Google Protobuf. There are two advantages to doing this:

  1. It will reduce the number of template parameters in the code.
  2. You could use the custom options to communicate the field length to other languages. Allowing them to know the limits of the embedded code. Read up on custom options if you wish to know how to do this.

The custom options are defined in the file embedded_proto_options.proto, which comes with Embedded Proto. When you use Embedded Proto as a submodule, run the install script again after updating it. The plugin needs a generated copy of the options and checks that it is up to date. Please refer to the installation manual on how to do this.

Next, let us modify your .proto files to include two additional elements:

import "embedded_proto_options.proto"; // Number 1

message Foo
{
  repeated uint32 y = 1 [(EmbeddedProto.options).maxLength = 10]; // Number 2
}

Number 1: Include the definitions of the Embedded Proto custom options file. This file is part of the EmbeddedProto Python package. Inside the submodule you find it in the EmbeddedProto folder.

Number 2: Add the custom option to each repeated field for which you wish to set the length. In this example, the length is set to 10 elements.

Finally, generate the code. The embeddedproto command includes the folder with the options file for you:

embeddedproto -I LOCATION/PROTO/FILES --eams_out=GENERATED/SRC/DIR PROTO_MESSAGE_FILE.proto

When you call protoc yourself, add the folder with the options file with an additional -I, as described on the generating source code page.

The generated code will now have the length filled in. So where in the templated option you will see something like y_SIZE, when using the custom option this will be replaced by a fixed number.

Special case

Starting from version 3.6.0 there is an additional option for repeated string or byte fields. It is now possible to specify the size of the array separately from the size of the string or bytes held by the repeated field. Below you see the example. Please note that the extra option to name is `nestedMaxLength`.

In this example, the array will have three elements, each holding a bytes field with ten bytes in each.

import "embedded_proto_options.proto";

message Foo
{
  repeated bytes array_of_bytes = 1 [(EmbeddedProto.options).maxLength = 3, 
  								 	 (EmbeddedProto.options).nestedMaxLength = 10];
}

Option 3: Set the length in an options file

As of version 4.0.0 the same options can be set from a JSON file, without editing the .proto file. This is useful when the .proto file is shared with other teams or comes from a library. The file is passed to the plugin when generating the code. See the options file page.

Usage of repeated fields

To access the array and the elements in the repeated field various functions are generated in the message. 

void add_y(const EmbeddedProto::uint32& value);
void set_y(uint32_t index, const EmbeddedProto::uint32& value);
void set_y(uint32_t index, const EmbeddedProto::uint32&& value);

::EmbeddedProto::RepeatedFieldSize<EmbeddedProto::uint32, y_SIZE>& mutable_y();

const EmbeddedProto::uint32& y(uint32_t index) const;
::EmbeddedProto::RepeatedFieldSize<EmbeddedProto::uint32, y_SIZE>& get_y() const;

void clear_y();

Setting values in the array can be done by adding data to the end of the array with add_y() or at a given index with set_y(). It is also possible to set the data via the mutable function mutable_y(). This function returns the whole array as a non-constant reference.

This function could also be used to just retrieve data but it is better to use the const version in that case by calling get_y(). Accessing individual elements is also possible using the function y(). Please note that you have to check yourself if your index is within the bounds of the array size.

Finally, you can clear the whole array with the clear_y() function. As of version 4.0.0 a single element can be removed with mutable_y().erase(index). The elements behind it shift down one place.

Is the array too large for your RAM? A repeated field can also be streamed through callbacks, without storing the elements in the message. See callback storage.

As of version 4.0.0, a repeated field of fixed32, fixed64, float or double is copied to and from the buffer as one block on little-endian targets. When you write your own buffer class, or build for a big-endian target, read about packed fixed-width fields.