Dynamic Software Updates using CRDTs

The DSU approach should apply to any distributed application. We will use a CRDT to replicate the application code amongst the different nodes of the distributed application. Even though the application code is stored within the CRDT, the CRDT design should not impose any restriction on the type of application that can be updated.

In other words, the CRDT should be independent of the application code it replicates, ensuring compatibility with distributed applications regardless of their implementation.

All communication between the CRDT replicas that propagate software updates should be transparent to the implementation of the application itself. The application code should not need to contain any logic that checks whether it is up-to-date, and it should not be concerned with distributing updates across different components of the application. The CRDT implementation itself hides the complexity of the update distribution logic. However, the DSU approach should also ensure that application-specific messages sent between the different components of the distributed application do not break the overall application execution, even if some components have not been yet updated and run an older version of the code.

As a CRDT will manage all the application’s code updates, an application does not have control over when it might be updated. When the application needs to be updated, the DSU should offer a way to migrate the application’s state from an old code version to a new one.

However, the logic of state migration is application-dependent and cannot be inferred from code changes(Tesone et al., 2018). The DSU approach should thus provide the means for developers to specify what needs to be migrated.

There should be a clear distinction between the application code and the logic needed to update the application itself. The implementation of the application should not be mixed with the logic required to migrate the state, for example. Although migrating the state of an application is inherently intertwined with the application itself, the developer should be able to separate the logic of updating their application from the actual implementation itself.

We model code updates with a state-based CRDT in which the payload is the application code.

We model the application code as closures, but other representations are possible ( e.g. JavaScript uses strings, Julia uses symbolic expressions, etc.). We chose a state-based CRDT design as it requires fewer assumptions from the underlying framework or programming language. Modelling code updates with an operation-based CRDT design requires the programming language to reify code changes and dynamically apply them to a running system. This is only possible in some languages with advanced DSU support, such as Smalltalk or CLOS.

To allow the CRDT to differentiate code updates, we associate a version with the application code. Versions are assumed to be totally orderable. Our approach uses semantic versions¹¹1https://semver.org to represent application versions, although another scheme may be used.

A semantic version consists of three subversions: major, minor, and patch. An increment to the patch version represents fixes to the application that do not provide new functionality. A new minor version indicates the addition of a new feature while still being backwards compatible. Finally, a major version increment represents a backwards-incompatible API change. Semantic versions allow us to define a total order of code updates, enabling the CRDT to decide what updates to apply.

We will refer to our state-based CRDT as the App CRDT from now on. Algorithm 1 shows its specification. As mentioned, the App CRDT payload is the application code and its version.

The initial payload consists of version ”0.0.0”, and nil, i.e. we do not provide any application code.

The update operation brings up to date the payload with the given version and application code. The precondition ensures that the given version is higher than the current one.

Note that the CRDT can either be updated with a new application code locally, which will then be propagated to other replicas, or it can receive a code update from another replica. In both cases, the new version is assumed to be greater than ”0.0.0”. Our approach also handles the teardown of the old code and the initialisation of the new code to enable state migration between the old and new application instances. This is detailed in the following section.

The getValue operation just returns the application code. Ultimately, the CRDT represents application code, and the version is only metadata used to correctly merge code updates.

The compare operation takes two states as arguments and returns true if the first state’s version is smaller than or equal to the second state’s version. If a compare operation returns false, it implies the first state has a strictly lower version than the second state.

The merge procedure takes a new state N and updates the local state to conform to the new state if it has an appropriate version. Concretely, it selects the application state with the newer version.

This means if the replica receives a newer version, the local state will be updated to this newer version. Otherwise, we will not perform an update.

Updating an application instance with new code requires careful management of resources and the application state to ensure a correct transition. Our approach supports this process by allowing applications to define a teardown handler. This handler is invoked when a newer version of the application is being deployed.

The teardown handler can release resources held by the current version and optionally return data that needs to be migrated to the new code version. Developers need to define an init handler that takes this optional state and initialises the application.

It is up to the application developers to ensure that the handlers behave consistently across instances. While the application state does not always need to be globally consistent, certain scenarios may require maintaining invariants across instances. In such cases, developers must design the teardown and init handlers to uphold these invariants during the migration process.

Although the App CRDT ensures that all application code eventually converges, code updates are not applied instantly. Since the system synchronizes replicas asynchronously, some application instances may be updated sooner. As a result, some instances may run an outdated code version. This is problematic when there are backwards-incompatible software updates. For example, consider that all of the application instances in the system are running version 1.0.0 of an application. The developer now publishes a new version of the application, tagged 2.0.0, with a backwards-incompatible API change. While the instances are being updated, some could run the new version 2.0.0 while the rest are still running version 1.0.0. If these application instances interact, this could lead to errors.

To enable application instances to interact with others without requiring all updates to be compatible, we introduce a API-compatible message delivery layer.

All application-level messages are transmitted with the code version. Our system delivers a message to the application instance immediately or temporarily buffers it, depending on the compatibility of the code versions.

Algorithm 2 outlines the interplay between the message delivery layer and the App CRDT at an application instance. When an application receives an application message $m$ for an object it hosts $r$, the receive event on that instance is triggered. The deliver function is provided by the platform to invoke the application operation corresponding to $m$ on the receiver object $r$.

The majorVersion function extracts the major versions of the message $v_{m}$ so that it can be compared with the local application instance version $V$.

We distinguish three cases:

• •

  $v_{m}=V$ If the version of the message $v_{m}$ equals the version of the local application code $V$, the sender application instance runs the same version of the code as the receiving one, so the message can be delivered. Similarly, the system delivers messages which do not carry information about a version, i.e. unversioned message. An example of an unversioned message is the update message being propagated between App CRDT replicas.

• •

  $v_{m}<V$ If the major version of the message is lower than the major version of the local application code, the application instance received a message from another instance running an older code.

  Although the API between these replicas is incompatible, the developer might be able to provide a function that transforms the received message into one compatible with the local API. This transformation can be implemented in the updateMessage function provided in the application code. Given a message $m$, updateMessage returns a new message compatible with the version of the local application instance.

• •

  $v_{m}>V$ If the major version of the message is higher than the major version of the local application code, the system does not deliver the message, as this might cause errors.

  Since we assume eventual-reliable point-to-point channels, the code update to the higher version will eventually be propagated to the local App CRDT. While the App CRDT does not receive the update ( and the local application instance does not run the higher code version), the system buffers the application messages received with the $v_{m}$ version in $Q$. More concretely, the messages are first assigned a local message counter and then stored in $Q$ together with the receiver (line 12 in Algorithm 2).

Once the application instance has been updated, the appVersionUpdated event is triggered, which triggers receive (and eventually deliver) for any messages that can be delivered in the order they were stored (lines 17-18). The event will be triggered directly after the initialisation of the new app version (i.e., after init in Algorithm 1).

Our DSU approach ensures that application instances with incompatible APIs can still coexist on the same network while being updated. The extension also ensures that the system’s delivery guarantees remain unaffected. Since the AppCRDT will be updated eventually, the messages stored in $Q$ will also be delivered.

Algorithm 1 shows the core of the AmbientTalk implementation of our approach. It directly corresponds with the previously described specification, dealing with application versioning and state migration.

The code defines the AppCRDT prototype class, which can be used by application developers to implement their own App CRDT-based applications. The init method (lines 6-18) takes care of initialising the application, and ensuring that the application CRDT goes online. It also takes care of installing a special AmbientTalk mirror using setVersionMirror which will ensure that the asynchronous messaging layer of AmbientTalk is extended with the API versioning logic. We will discuss this in more detail in Section 4.2.

The essence of the logic is embedded in the merge method, from lines 20 to 31. This method is invoked whenever code updates arrive and ensures the teardown of the existing state, and initialisation of the new code corresponding to the specification defined in section 3.1.

We implemented API-compatible message delivery by installing a new message delivery protocol in AmbientTalk. The essence of the code for this can be seen in Listing 2. Messages between replicas are tagged with the code version so that the receiver actor know if it either needs to deliver the message or buffer it. If a message is not tagged, the message is always delivered. This is needed to support messages with the UnVersionedMessage tag. If the message is tagged, we implemented the three possibilities described in 3.3.

We can achieve this behaviour by levering on AmbientTalk’s meta-object protocol throughmirrors. Mirrors are metaobjects that allow developers to inspect and manipulate application objects (called base-level objects) and their runtime behaviour. In our case, we use them to inspect messages when they are delivered to an actor’s mailbox (lines 13-31). With this code actors receiving messages can decide whether to deliver those messages or to buffer them until the correct application version is active. Although adding version information to every message sent by an actor may seem like a complex change, the meta-object protocol allows us to implement this behaviour in a single call. At line 34 in Listing 2 we show how the send method is defined in the mirror installed by the extension.

The send method is a part of the meta-object protocol and is called whenever an actor sends a message to another actor. In our extension, we first check whether the message is tagged with an UnVersionedMessage type tag. If this tag is present, we do not add version information to the message. This allows us to make sure some messages are delivered regardless of the implementation version. If the tag is not present, we add a ”version” field to the message containing the version of the implementation used by the sender, and we tag the message with the Versioned tag. This allows the receiving actor to easily detect whether a message contains version information or not.

To intercept messages when they arrive, we implement the receive method on line 14. This method will be called when the actor receives a message. Here we follow the specification in Algorithm 2, and put messages that have a too new version inside of a queue, to be eventually dequeued when the application version is high enough.

Because the CRDT decides when to update the application, it manages the lifecycle of the replicated application by design. To start an application, the app CRDT can be provided with an initial implementation. The CRDT will then start this application and discover other replicas, updating them or receiving updates itself if needed. The implementation provided to the CRDT can be seen as a specification for starting the application. Practically, the implementation needs to be an object with the following fields:

• •

  version: This field should contain a semantic version object that represents the version of the implementation. This information is used by the app CRDT to decide how to merge other implementations.

• •

  init: This field should contain a method that initializes the application. This function has special semantics in AmbientTalk: whenever we create a new object by calling the new method, AmbientTalk will clone the object and initialize its state by calling the init function. This is also what will happen in our implementation: when the app CRDT wants to start an implementation, it will create a new app object by calling new. Treating the implementation as an object is useful since it allows us to associate the state with the implementation. This can be used later when we want to clean up its resources.

• •

  teardown: Whenever the app CRDT receives an implementation with a higher version, it needs to update the application that is running locally with the new implementation. To shut down the old implementation and clean up any resources, the implementation can define a teardown function. This is useful when working with external libraries, like GUI frameworks. If an application requires a GUI, the logic for closing the GUI can be provided in the teardown function.

Whenever the CRDT wants to create a new application, it will create a new app object by calling the new method on the implementation object. One of our design goals was to provide the ability to migrate the state of an application during an update. This migration logic depends on the application and needs to be provided by the application developer. More specifically, migration logic can be implemented in the initialization method. If the app CRDT is updating an existing application, it will pass the old application object as an argument to the initialization function. This gives the developer access to the old state of the application, allowing it to migrate local state if needed.

Note that this migration logic can be as simple or as complicated as needed. If the application does not have any local state, or if it does not need to be transferred, the initialization function can initialize the new app object as normal. If some state does need to be transferred, the old application object also contains its version. This can be used to customize the migration logic based on the version of the old application.

Listing 3 shows an example of how the implementation of an application can be replicated using the CRDT.

We start by creating our application specification on line 4. Instead of creating an ordinary object, we create an isolated object using the isolate: method. This is related to how AmbientTalk sends objects to other VMs. In our example, we want to send this app object to other replicas. There, the CRDT can use the version field to decide whether to update the implementation or not. If an update is required, it can create an instance of the new application by calling the new method.

However, if we send an ordinary object to another VM, the other replica will receive a Far Reference instead. A far reference represents a reference to an object from another actor. This is because each actor has its own, isolated state. Far references do not allow synchronous access to the object they reference. Instead, we can use a far reference to send an asynchronous message to the object. This allows us to easily send objects to other actors or VMs without breaking the actor system. However, this is not what we want in our app CRDT.

When we send the implementation to another VM, we want to receive an ordinary object and not a far reference. The goal of replicating the implementation is that each replica has the implementation locally. We can’t use a far reference to instantiate a new application. We can only send messages to it, which will be processed by the actor that contains the object. Fortunately, we can create an isolated object instead. The difference between a regular and an isolated object is that the latter is sent by value. This means that other replicas will receive a copy of the implementation, instead of a reference to the implementation object on another VM.

Our implementation object contains the required version, init, and teardown methods. We also create a state field which the application will use to store some local state. In the initialization function, we can provide our migration logic. If the argument of init is not nil, it will contain the old application object. If we are updating the application, we make sure that the local state is copied correctly. Otherwise, we initialize the state with a random number.

After creating our application specification object, we can start it by creating a new app CRDT and providing it with our implementation object. The CRDT will manage the lifecycle of the application, initializing it on startup and updating it whenever a newer version is received.

This implementation demonstrates that our three design goals have been achieved. First of all, the logic related to communicating with other replicas to receive and send implementation updates is hidden from the user of the CRDT. The specification object of the application only needs to contain the version and the code, while the app CRDT takes care of distributing and updating the implementation. Our second design goal was to provide CRDT users with a way to migrate the local state of an application when it is updated. This is enabled by the initialization method, which can contain complex state migration logic. Our final design goal was a separation of concerns. Although our example is small, the idea is that the core functionality of the application can be located in separate files or modules. For example, we could implement our GUI in a separate object that is created by the initialization method. The GUI implementation does not need to know that its implementation is distributed across multiple replicas. This allows us to easily integrate existing applications with our CRDT.

Although our CRDT can keep an implementation consistent across multiple replicas, it poses some risks when it comes to security. The next section introduces an extension that allows the CRDT to check whether an update was produced by a trusted party.

In networks with no authentication or authorization policies, a malicious VM could start distributing updates to other replicas with implementations containing untrusted code. To prevent this from happening, our App-CRDT implementation incorporates code signing functionality.

When the code signing extension is enabled on the CRDT, it will check whether every received update is correctly signed by a trusted party. If the signature is valid, the update is applied (if needed). Otherwise, the message is ignored. We can use asymmetric cryptography with a public-private key pair to generate a signature. We can add a signature by hashing the implementation and encrypting the hash using the private key. Replicas that receive the update can then hash the received implementation, and then decrypt the signature with the public key. If the signature is valid and the implementation has not been tampered with, the decrypted string will be equal to the hash.

This extension allows the replicas to be certain that an implementation was created by a trusted party. A malicious replica could try to change the implementation in a Man In The Middle attack. However, this would cause the signature verification to fail on the replica that receives the update. Since the implementation has been modified, its hash will be different, which will cause a mismatch between the hashed implementation and the decrypted signature.

Even though the implementation is distributed in a peer-to-peer fashion, the replicas can verify that an update message is genuine by checking the signature.

Looking back at listing 3, it shows a very basic example of using the app CRDT. More complex applications can similarly be integrated with the app CRDT. An interesting example of an application whose implementation can be replicated uses CRDTs itself. For example, we could use the app CRDT to replicate the implementation of a state-based counter CRDT. This is discussed in the next section.

We also implemented our CRDT in LuAT(Bauwens, 2017), an ambient-oriented programming framework for Lua. Whereas AmbientTalk can be regarded as a programming language for research purposes, Lua is currently used in real-world systems. Extending our implementation to Lua shows our approach applicability for real-world applications.

As LuAT also implements an ambient-oriented programming model, it shares many features with AmbientTalk. The foundation of AmbientTalk is asynchronous message passing between actors, but Lua does not directly support actors. However, LuAT enables actors to be implemented by providing a set of concurrency-related functions that use Lua coroutines under the hood. As in AmbientTalk, LuAT provides methods to export and discover objects on a local network by connecting to other LuAT programs in a peer-to-peer fashion. We can also create objects in LuAT. Objects can contain state and methods and are transformed into far references when sent to a different actor.

However, we cannot create isolated objects in LuAT. This means that objects sent to other actors are always sent by reference, and cannot be sent by value. A solution to this problem is to send Lua tables instead of LuAT objects. Because a table is seen as a primitive value by the serializer, it is sent by value to other replicas.

Ultimately, LuAT is a framework implemented on top of Lua, while AmbientTalk is a full-fledged programming language. But for our discussion on extending our code replication CRDT to other languages, LuAT is an ideal choice since it also supports the ambient-oriented programming paradigm. In the next section, we will discuss how the app CRDT can be used in LuAT.