Migrating from 3.x to 4.0

Version 4.0.0 changes how Embedded Proto is installed and adds maps, streaming through callbacks, an options file and support for protobuf editions. Your proto files and the code you wrote against the generated messages stay as they are. What changes is in your toolchain, and, in the rare case that you wrote your own buffer class, in that class. Below is the list, in the order you will run into it.

Toolchain

  1. Compile with C++17. Version 3.x compiled with C++11 and up. Version 4.0.0 uses if constexpr, so C++17 is required.
  2. Python 3.11 or newer. Protoc does not have to be installed anymore, the protobuf compiler comes with the package. The --include parameter of the install script has no function anymore.
  3. The setup script is now the install script. When you use the submodule, replace python setup.py with python install.py. Alternatively, drop the submodule and install with pip, as described on the installation page.
  4. Regenerate every message with the 4.0 plugin. A generated header now records the version of the plugin which made it and checks it against the library when compiling. Mixing 3.x code with the 4.0 library stops the compiler with Major version mismatch between generated code and library. So you cannot forget this step.
  5. Replace the src folder, and drop the three source files. The library is header only. Fields.cpp, MessageInterface.cpp and ReadBufferSection.cpp no longer exist, so remove them from your build. The src folder stays an include path.
  6. Change your include lines. The headers moved into a folder of their own, so that Errors.h cannot collide with a header of yours. Where you wrote #include "WriteBufferFixedSize.h", write #include <EmbeddedProto/WriteBufferFixedSize.h>. Or include everything at once with <EmbeddedProto.h>. Generated code is regenerated anyway, and includes the new way.
  7. The custom options file moved. In 3.x, embedded_proto_options.proto was in the generator folder of the submodule, and your protoc command included it with -I./generator. It is now part of the Python package, in the EmbeddedProto folder of the submodule. Easier is to switch to the embeddedproto command, which includes the folder for you. See generating source code.

Your own buffer classes

Did you implement the ReadBufferInterface yourself, for instance to read straight from a UART? The interface has one new pure virtual function:

virtual bool peek(const uint32_t n_bytes, uint8_t& byte) const = 0;

It returns the byte n_bytes ahead of the current position without advancing. It returns false when that many bytes are not available. The library uses it to read a varint without consuming the buffer when the varint is incomplete. Have a look at ReadBufferFixedSize.h for a reference implementation. You may also override pop(const bytes_view& dest) and peek(const bytes_view& dest), which pop or peek a block of bytes at once. Their default implementations go byte by byte.

The WriteBufferInterface has one new pure virtual function, the block push:

virtual bool push(const const_bytes_view& bytes) = 0;

It appends all the bytes in the view or, when they do not fit, none of them and returns false. The 3.x form push(const uint8_t* bytes, uint32_t length) still exists on the interface, forwards to the view form and is marked deprecated, so a buffer class of your own only has to implement the view form.

Arrays are passed as views

Every function which took an array as a pointer and a separate length now takes an array_view, a small struct from Defines.h which holds the pointer and the number of elements together. Four names are ready made: bytes_view and string_view for arrays the library writes into, const_bytes_view and const_string_view for arrays it only reads.

buffer.push(::EmbeddedProto::const_bytes_view{data, n});
buffer.pop(::EmbeddedProto::bytes_view{data, n});
msg.mutable_payload().set(::EmbeddedProto::const_bytes_view{data, n});
msg.mutable_name().set(::EmbeddedProto::const_string_view{text, n});
msg.mutable_values().set_data(::EmbeddedProto::array_view<const int32_t>{values, n});

The pointer and length forms still compile, but they are marked deprecated and will be removed in a later major release. The library and the generated code no longer use them. Please note that std::array::size() returns a size_t, so give it a static_cast<uint32_t> when you build a view from it.

Serializing fields by hand

Most users never call the serialize function of a single field, only of the whole message. If you did, the function serialize_with_id() is removed from Field, MessageInterface, FieldStringBytes and RepeatedField. Its replacement is:

Error serialize_len(const uint32_t field_number, const uint32_t size, WriteBufferInterface& buffer, const bool optional) const;

It writes the tag, the size and the data. The size is serialized_size() for a message, get_length() for a string or bytes field and serialized_size_packed() for a repeated field. On scalar fields serialize_with_id() still exists, but it now skips the default value unless optional is true, as the message code always did.

If you derived from RepeatedField: serialize() now writes the packed elements without tag and size, the functions are no longer final, and there is a new virtual function erase(index) which returns INDEX_OUT_OF_BOUND unless you implement it.

Behaviour to be aware of

  1. When deserialize() returns an error, the message remembers the field it was working on. A second call continues with that field instead of reading a new tag. Call clear() first when you want a fresh parse.
  2. To make this possible, every message holds two extra members: a field number and a wire type. Count on a few bytes more RAM per message object.
  3. Five values are added to the Error enumeration: STATE_MISMATCH, NESTING_TOO_DEEP, CALLBACK_NOT_SET, CALLBACK_SEQUENCE and CALLBACK_SIZE_MISMATCH. A switch statement listing every value will warn about the new ones.
  4. The memory layout of strings is unchanged. The new NULL_TERMINATED_STRINGS define, which reserves room for a terminator, is off by default. See strings.
  5. The destructors of Field, ReadBufferInterface and WriteBufferInterface are protected and no longer virtual. Deleting a message or buffer through a pointer to one of these interfaces, for instance a std::unique_ptr<MessageInterface>, no longer compiles. Define VIRTUAL_DESTRUCTORS_ENABLED to get the 3.x behaviour back. See code size.
  6. The bytes on the wire are identical. A 3.x device and a 4.0 device understand each other.

What is new

The features 4.0.0 adds are listed on the page what is new in 4.0. None of them require a change to existing code.