Callback storage

Every field in Embedded Proto has a fixed size, so the RAM a message takes is known at compile time. Sometimes the data is larger than the RAM you have. Think of a firmware image in a bytes field, or thousands of samples in a repeated field. As of version 4.0.0 such a field can be streamed instead of stored. With callback storage the field is not stored in the message at all. The library asks your code for each element while serializing, and hands each element to your code while deserializing.

Mark a field with the callbackStorage option:

import "embedded_proto_options.proto";

message Samples
{
  repeated int32 values = 1 [(EmbeddedProto.options).callbackStorage = true];
}

The option is available for repeated scalar and enum fields, for string and bytes fields, for maps, and for repeated message fields when the message encoding is delimited, see protobuf editions. It is not available for members of a oneof, for singular scalars and messages, for optional string and bytes fields, and for repeated string or bytes fields. A callback field adds no template parameter to the message, as nothing is stored.

The field object has two callbacks. The source is called during serialization, once per element, until it returns false. The sink is called during deserialization, once per element received. Each callback is held in a Functional, a small non-owning wrapper around a function, a function with a context pointer, or an object with a call operator. Let us take a look at an example:

using Field = ::EmbeddedProto::RepeatedFieldCallback<::EmbeddedProto::int32>;

// Produces three values, one per call, and returns false when done.
struct Producer
{
  int32_t values[3] = {10, 20, 30};
  uint32_t index = 0;

  bool operator()(::EmbeddedProto::int32& element)
  {
    const bool more = index < 3;
    if(more)
    {
      element.set(values[index]);
      ++index;
    }
    return more;
  }
};

// Receives one value per call. Returning an error aborts deserialization.
struct Collector
{
  ::EmbeddedProto::Error operator()(const ::EmbeddedProto::int32& element)
  {
    // Store element.get() somewhere.
    return ::EmbeddedProto::Error::NO_ERRORS;
  }
};

// Serialize.
Producer producer;
Field::SourceCallback source;
source.set(producer);

Samples out;
out.mutable_values().set_source(source);
out.serialize(write_buffer);

// Deserialize.
Collector collector;
Field::SinkCallback sink;
sink.set(collector);

Samples in;
in.mutable_values().set_sink(sink);
in.deserialize(read_buffer);

You only bind the side you use. A device which only sends binds a source, a device which only receives binds a sink. Without a source the field serializes as empty, and without a sink the received elements are dropped. When you prefer an error over silence, call set_strict(true) on the field. A missing callback is then reported as CALLBACK_NOT_SET.

String and bytes fields work differently. One byte per call would be fine for a small value, but a firmware image is better moved in blocks. A flash write or a DMA transfer wants a buffer, not a byte. So a string or bytes field streams in chunks. Take a bytes field marked with the option:

message Update
{
  bytes firmware = 1 [(EmbeddedProto.options).callbackStorage = true];
}

The chunks are staged in memory you provide, a window. Bind it with set_firmware_window() as a view holding a pointer and a size. The size is the largest chunk your code is handed per call. The message holds only the view, so the window can be a static array, a DMA buffer or a scratch region shared between fields, and its size is a choice of your platform rather than of the schema. One window serves both directions, a field never serializes and deserializes at the same time. A window of one byte gives you byte by byte streaming when that is all you want.

Both directions use the same callback shape, a function taking a bytes_view (or a string_view for a string field) and returning a number of bytes. The view holds a pointer and a size. While deserializing your code is handed each parsed chunk and returns how many bytes it accepted. While serializing your code is handed a writable window, fills it, and returns how many bytes it produced. No offset is passed, your code keeps track of where it is. Let us take a look at an example:

using Firmware = ::EmbeddedProto::BytesStringCallback<uint8_t>;

// The window, 64 bytes per chunk.
static uint8_t window[64];

// Receive: write every chunk straight to flash.
uint32_t written = 0;
auto to_flash = [&](::EmbeddedProto::bytes_view chunk) -> uint32_t
{
  flash_write(FLASH_BASE + written, chunk.data, chunk.size);
  written += chunk.size;
  return chunk.size;
};
Firmware::ChunkCallback sink;
sink.set(to_flash);

Update in;
in.set_firmware_window({window, 64});
in.set_firmware_on_deserialize_chunk(sink);
in.deserialize(read_buffer);

// Send: declare the length, then fill each window from flash.
uint32_t sent = 0;
auto from_flash = [&](::EmbeddedProto::bytes_view window) -> uint32_t
{
  const uint32_t n = std::min(window.size, image_size - sent);
  flash_read(FLASH_BASE + sent, window.data, n);
  sent += n;
  return n;
};
Firmware::ChunkCallback source;
source.set(from_flash);

Update out;
out.set_firmware_window({window, 64});
out.set_firmware_length(image_size);
out.set_firmware_on_serialize_chunk(source);
out.serialize(write_buffer);

Serializing needs the total length up front, as it is written in front of the data. So you call set_firmware_length() once before serializing, and the source must then produce exactly that many bytes. The length is also what serialized_size() reports, no data is pulled for it. Returning zero before the length is reached ends serialization with CALLBACK_SIZE_MISMATCH.

Deserializing needs no length, it is read from the wire. Accepting fewer bytes than offered stops the field and deserialization returns CALLBACK_STOPPED. Only the accepted bytes are consumed. With partial deserialization the field can be resumed afterwards, so a sink that is temporarily out of room can take the rest later. A window never spans two buffers there, the end of a buffer simply ends the chunk.

Please note two limits. A callback field can only be used in a message which is serialized directly, or nested through a delimited message. A message nested with a length prefix needs its size up front, which a source cannot give, and serialization returns CALLBACK_SEQUENCE. Also, max_serialized_size() of a message with a callback field returns UINT32_MAX, as the size is not known.

Would you rather keep the data in the message, but in a storage class of your own? See custom storage.