Packed fixed-width fields

A repeated field of a numeric type is packed on the wire: one tag and one length, followed by all the elements. For most types each element still has to be encoded on its own, as a varint takes more or fewer bytes depending on its value. Six types are different. The elements of fixed32, sfixed32, float, fixed64, sfixed64 and double always take four or eight bytes, and protobuf stores them little-endian.

On a little-endian processor, which most microcontrollers are, the array in memory is therefore already the wire format. As of version 4.0.0 Embedded Proto makes use of this. The whole array of such a repeated field is written to the buffer with a single call, and read back with a single call. Before, a repeated fixed32 with a hundred elements meant four hundred separate calls to push() on the write buffer. Now it is one. A user streaming sensor data in real time had to fall back to a bytes field to get this. That workaround is no longer needed.

The same applies to a single field of one of these six types. Its four or eight bytes are pushed as one block instead of one at a time.

Repeated fields of varint types, such as int32, uint32, sint64, bool and enumerations, are not affected. They are encoded per element, as before. When serializing in parts each varint element is encoded first and pushed as one block, so that an element which does not fit is retried whole in the next buffer.

Endianness

The byte order of the target is checked at compile time in WireFormatter.h. The library reads the __BYTE_ORDER__ macro of the compiler, and treats _WIN32, _M_IX86, _M_X64, _M_ARM and _M_ARM64 as little-endian. On a big-endian target, or when the detection fails, the library uses the per element path and the wire format stays correct. It is only slower.

Does your compiler not provide the byte order macros? Then define EMBEDDED_PROTO_LITTLE_ENDIAN as 1 or 0 when building. This is the only case in which you need it.

Your own buffer classes

The gain only exists when the buffer accepts a block efficiently. The buffers which come with the library, WriteBufferFixedSize, ReadBufferFixedSize and ReadBufferSection, do. When you wrote your own, please note the following:

  1. WriteBufferInterface::push(const const_bytes_view& bytes) is pure virtual, so your write buffer already implements it. Make sure it copies the block in one go, and does not loop over the single byte push(). It must also append all bytes or none: when the block does not fit, return false without writing a part of it. Serializing in parts relies on this to retry a split element cleanly.
  2. WriteBufferInterface::get_available_size() must never report more bytes than a following push() accepts. When serializing in parts a tag and its value are written as a pair on the strength of this number. Once the tag is in the buffer the value has to fit, there is no way to take the tag back.
  3. ReadBufferInterface::pop(const bytes_view& dest) is virtual with a default implementation which loops over the single byte pop(). Override it with a block copy to benefit when deserializing.
  4. ReadBufferInterface::peek(const bytes_view& dest) is the same for a block which stays in the buffer. A bytes or string callback field fills its window with it, and only advances the buffer by what the callback accepted. Override it as well when you use those fields.

When the block is used

Serializing always writes the block. Deserializing reads the block when the length of the packed data is a whole multiple of the element size, the elements fit in the remaining capacity of the repeated field, and all bytes are present in the buffer. In every other case the library falls back to reading element by element, with the same lenient behaviour as before. Elements received are appended after the elements already in the field, as a packed field may appear more than once in a message.

Two exceptions. When serializing in parts is enabled, these fields are still written one element per call, so that serialization can resume after a full buffer. And a repeated field with callback storage emits its elements one at a time by design.