4/9/2020 3.

Communication and Data Contracts - Building Event-Driven Microservices

 Building Event-Driven Microservices

PREV NEXT
⏮ ⏭
2. Event Driven Microservice Fundamentals 4. Integrating Event-Driven Architectures with Existing Systems
  🔎

Chapter 3. Communication and Data

Contracts
“The fundamental problem of communication is that of reproducing at one point
either exactly or approximately a message selected at another point.” - Claude
Shannon, Father of Information Theory

Introduction
The simplest pattern in communication theory is that of a sender seeking to
provide a message to a receiver via a specific channel. The sender and receiver
must have a common understanding of the message, otherwise the message may
be misunderstood and the communication will be incomplete. In the event-driven
ecosystem, the event is the message, and it is the fundamental unit of
communication. An event must describe as accurately as possible what happened,
and why. It is a statement of fact, and combined with all the other events in a
system provides a complete history of what has happened.

Event Driven Data Contracts

The definition of the data to be communicated is essential for a well-defined
event. The format of the data and the logic under which it is created form the
basis for what is known as the data contract. This contract is followed by both the
producer and the consumer of the event data. This allows the definition of the
event to have meaning and form outside of the context of which it is produced
and extends the usability of the data to consumer applications.

There are two components of a well-defined data contract. The first component is
the data definition, or what will be produced. It is a description of the data that
will be created, namely the fields, types and various data structures. The second
component is the triggering logic, or why it is produced. This is a description of
the fulfillment of specific business logic which triggers the creation of the event.
Changes can be made to both the data definition and the triggering logic,
depending on business requirement changes.

Care must be taken when changing the data definition, so as not to delete or alter
fields that are being used by downstream consumers. Similarly, care must also be
taken when modifying the triggering logic definition. It is far more common to
change the data definition than the triggering mechanism, as altering the
definition of when an event is triggered often breaks the meaning of the original
event definition.

Using Explicit Schemas as Contracts

The best way to enforce data contracts and provide consistency is to define a
schema for each event. The producer defines an explicit schema detailing the
data definition and the triggering logic, with all events of the same type adhering
to this format. In doing so, the producer provides a mechanism for
communicating its event format to all prospective consumers. The consumers, in
turn, can confidently build their microservice business logic against the
schematized data.

Any implementation of event-based communication between a producer and

consumer that lacks an explicit pre-defined schema will inevitably end up relying
on an implicit schema. A consumer must be able to extract the data necessary for
its business processes, and it cannot do that without having a set of expectations
about what data should be available. Implicit data contracts generally result in
much undue hardship for downstream consumers.

With an implicit schema, consumers must often rely on tribal knowledge and
inter-team communication to resolve data issues. This is not scalable as the
number of event streams and teams increase. There is also substantial risk in
requiring each consumer to independently interpret the data. A consumer may
interpret it in a way that is different than its peers, which leads to inconsistent
views of the single source of truth.

It may be tempting to build a common library that interprets any given event for
all services, but this runs into problems with multiple language formats and
constrains independent release cycles. The duplication of efforts across services
to ensure a consistent view of implicitly defined data is non-trivial, and is best
avoided completely.

Producers are also at a disadvantage with implicit schemas. Even with the best of
intentions, a producer may not notice (or perhaps their unit tests don’t reveal) that
they have altered their event data definition. Without an explicit check of their
service’s event format, it may go unnoticed until it causes downstream consumers
to fail. Explicit schemas give security and protection to both consumers and
producers.

Schema Definition Comments

Support for integrated comments and arbitrary metadata in the schema definition
is essential for communicating the meaning of an event. The knowledge
surrounding the production and consumption of events should be kept as close as
possible to the event definition. Schema comments support serves two very
important issues surrounding the use of schemas.

The first major benefit is being able to define the triggering logic of when an
event using that schema is produced, typically done in a block header at the top
of the schema definition. The second major benefit is the annotation of the fields
within the schema, where additional context and clarity can be provided by the
producer of the data about the particular field. These schema definition
comments help remove ambiguity as to meaning of data and reduces the chance
of misinterpretation by consumers.

Full Featured Schema Evolution

The schema format must support a full range of schema evolution rules. Schema
evolution provides a means for producers to update their service’s output format
while allowing consumers to continue consuming the events uninterrupted.
Business changes may require new fields be added, old fields be deprecated, or
You have 2 days le in
the your
scope of atrial, Reddychan.
field be expanded. Subscribe
A schema evolution today.
framework ensures that See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch03.html 1/5
4/9/2020 3. Communication and Data Contracts - Building Event-Driven Microservices

these changes can occur safely, and that producers and consumers can be updated
independently of one another.

Updates to services become prohibitively expensive without schema evolution

support. Producers and consumers are forced to coordinate closely, and old,
previously-compatible data may no longer be compatible with current systems. It
is unreasonable to expect that consumers to update their services whenever a
producer changes the data schema. In fact, a core value proposition of
microservices is that they should be independent of the release cycles of other
services except in exceptional cases.

An explicit set of schema evolution rules goes a long way in providing both
consumers and producers the ability to update their applications in their own
time. These rules are known as compatibility types.

Forwards Compatibility Allows for data produced with a newer schema to

be read as though it were produced with an older schema. This is a
particular useful evolutionary requirement in an event-driven architecture,
as the most common pattern of system change begins with the producer
updating its data definition and producing data with the newer schema. The
consumer is only required to update its copy of the schema and code should
it need access to the new fields.

Backwards Compatibility Allows for data produced with an older schema

to be read as though it were produced with a newer schema. This allows a
consumer of data to use a newer schema to read older data. There are
several scenarios where this is particularly useful.

1. The consumer is expecting a new feature to be delivered by the

upstream team. If the new schema is already defined, the consumer
can release their own update prior to the producer release.

2. Schema-encoded data sent by a product deployed on customer

hardware, such as cell phone application that reports on user metrics.
Updates can be made to the schema format for new producer releases,
while maintaining the compatibility with previous releases.

3. The consumer application may need to reprocess data in the event

stream that was produced with an older schema version. Schema
evolution ensures that the consumer can translate it to a version that it
is familiar with. If backwards compatibility is not followed, it means
that the consumer will only be able to read messages with the latest
format.

Full Compatibility The union of both forwards compatibility and backwards

compatibility. It is the strongest guarantee and the one most recommended for use
whenever possible.

Code Generator Support

Code Generator Support allows for an event schema to be turned into a class
definition or equivalent structure for a given programming language. This class
definition is used by the producer to create and populate new event objects. The
producer is required by the compiler or serializer (depending on the
implementation) to respect data types and populate all non-nullable fields that are
specified in the original schema. The objects created by the producer are then
converted into their serialized format and sent to the event broker, as shown in
Figure 1.

Figure 3-1. Producer event production workflow

using code generator
The consumer of the event data maintains its own version of the schema, which
is often the same version of the producer but may also be an older or newer
schema, depending on the usage of schema evolution.If full compatibility is
being observed, the service can use any version of the schema to generate its
definitions. The consumer reads the event and deserializes it using the schema
version that it was encoded with. The event format is either stored alongside the
message, which can be prohibitively expensive at scale, or it is stored in a
schema registry and accessed on demand (see Chapter TODO for schema registry
information). Once deserialized into its original format, the event is then be
converted to the version of the schema supported by the consumer. Evolution
rules come into play at this point, with defaults being applied to missing fields,
and unused fields dropped completely. Finally, the data is converted into an
object based on the schema-generated class. At this point, the consumer’s
business logic may begin its operations. This process is shown below in Figure 1.

Figure 3-2. Consumer event consumption and

conversion workflow using code generator
The biggest benefit of this feature is being able to write your application against a
class definition in your own language of choice. If you are using a compiled
language it provides compiler checks to ensure that you aren’t mishandling event
types or missing the population of any given non-null data field. Your code will
not compile unless it adheres to the schema, and therefore your application will
not be shipped with adhering to the schema data definition. Both compiled and
non-compiled languages also benefit from having a class implementation to code
against. A modern IDE will also notify you when you’re trying to pass the wrong
types into a constructor or setter, whereas you would receive no notification if
you’re instead using a generic format such as a map of object key-values.
Reducing the risk of mishandling data provides for a much higher consistency of
data quality across the ecosystem.

Selecting an Event Format

While there are many options available when deciding how to format and
serialize event data, data contracts are best fulfilled with strongly defined formats
such as Avro, Thrift or Protobuf. It is worth noting that some of the most popular
event broker frameworks have support for serializing and deserializing events
encoded with these formats. Apache Kafka, for example, has a readily available
Avro schema support, while Apache Pulsar can easily support JSON, Protobuf
and Avro. Though a detailed evaluation and comparison of these serialization
options is beyond the scope of this book, there are a number of online resources
available that can help you make a decision between these particular options.

You may feel inclined to choose a more flexible option in the form of plain text
events using simple key-value pairs, which still offers some structure but
provides no explicit schema or schema evolution frameworks. Caution is advised
with this approach as the ability of microservices to remain isolated from one
another via a strong data contract become compromised, requiring far more inter-
team communication.

TIP

It is often the case that the flexibility of the format proves to

be a burden and not a benefit, and as such, I recommend
choosing a strongly-defined, explicit schema format such as
Apache Avro or Protobuf.

You have 2 days le in your trial, Reddychan. Subscribe today. See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch03.html 2/5
4/9/2020 3. Communication and Data Contracts - Building Event-Driven Microservices

Designing Events
There are a number of best practices to follow when creating event definitions, as
well as several anti-patterns to avoid. Keep in mind that as the architectures
powered by event-driven microservices expand, the number of event definitions
expands. Well-designed events will minimize the otherwise repetitive pain points
for both consumers and producers. With that being said, none of these are hard-
and-fast rules. You can break them as you see fit, though I recommend that you
think very carefully about the full scope of implications and the trade-offs for
your problem space before proceeding.

The truth, the whole truth, and nothing but the truth

A good event definition is not a message just indicating that something happened,
but is the complete description of everything that happened during that event. In
business terms, this is the resultant data that is produced when input data is
ingested and the business logic is applied. This output event must be treated as
the single source of truth and must be recorded as an immutable fact for
consumption by downstream consumers. It is the full and total authority on what
actually occurred, and no other source of data should need to be consulted to
know that such an order took place.

Use a singular event definition per stream

An event stream should contain events representing a single logical event. It is

not advisable to mix different types of events within an event stream, for the
inference of what the event is, and what the stream represents, begins to get
muddled. It is difficult to validate the schemas being produced, as new schemas
may be added dynamically in such a scenario. Though there are special
circumstances where you may wish to ignore this principle, the vast majority of
event streams produced and consumed within your architectural workflow should
each have a strict, single definition.

Use the narrowest data types

Use the narrowest types for your event data. This lets you rely on the code
generators, language type checking (if supported) and serialization unit tests to
check the boundaries of your data. It sounds simple, but there are many cases
where ambiguity can creep in without using the proper types. Here are a few
easily avoidable examples of what I have seen.

Using string to store a numeric value. This requires the consumer to parse
and convert the string to a numeric value, and often comes up with GPS
coordinates. This is error prone and subject to failures, especially when a
value of “null” or an empty string are sent.

Using integer as a boolean. While 0 and 1 can be used to denote false and
true respectively, what does 2 mean? How about -1?

Using string as an enum. This is problematic for producers as they must

ensure that their published values match an accepted pseudo-enum list.
Typos and incorrect values will inevitably introduced. A consumer
interested in this field will need to know the range of values that can be,
and this will require talking to the producing team, unless they put the
range of values in the comments of the schema. In either case, this is an
implicit definition, since the producers are not guarded against any changes
to the range of values in the string. This whole approach is simply bad
practice.

Enums are often avoided because producers fear creating a new enum token that
isn’t present in the consumer’s schema. However, the consumer has the
responsibility to consider enum tokens that it does not recognize, and determine
if it should process them using a default value, or simply throw a fatal exception
and halt processing until a human can work out what needs to be done. Both
Protobuf and Avro have elegant ways of handling unknown enum tokens, and
these should be used whenever possible.

Keep Events Singular Purpose

One common anti-pattern is the addition of a “type” field to an event definition,

where different type values indicate specific sub-features of the event. This is
generally done for data which is “similar-yet-different”, and is often a result of
the implementer incorrectly identifying that the events are singular purpose.
Though it may seem like a time-saving measure or a simplification of a data
access pattern, overloading events with type parameters is rarely a good idea.

There are several main problems with this approach. Each of these type
parameter values usually has a fundamentally different business meaning, even if
their technical representation is nearly identical. It is also possible for these
meanings to change over time and the scope that an event covers can creep.
Some of these types may require the addition of new parameters to track specific
type-centric information, whereas other types require other separate parameters.
Eventually it is possible to reach a situation where there are several very distinct
events all inhabiting the same event schema, making it difficult to reason about
what the event truly represents.

This complexity not only affects the developers who must maintain and populate
these events, but it also proves difficult for users to consume this data.
Consumers of this data need to have a consistent understanding about what data
is published and why it is published. If the data contract changes, it is expected
that they be able to isolate themselves from those changes. Adding additional
field types requires that they must therefor filter-in only data that they care
directly about. There is a risk that the consumer of the data fails to fully
understand the various meanings behind the types, leading to incorrect
consumption and logical wrong code. Additional processing must also be done to
discard messages that the consumer does not care about, with the implementation
replicated between each consumer.

It is very important to note that the underlying complexity inherent in the data
being produced is not reduced or solved by adding type fields. In fact, this
complexity is merely shifted from multiple distinct event streams with distinct
schemas, to a union of all the schemas merged into one event stream. In fact, the
complexity can even be said to have increased significantly due to this
entanglement. Future evolution of the schema is made more difficult as is
maintaining the code that produces the events.

Remember the principles of the data contract definition. Events should be related
to a single business action, not a generic type of event that records large
assortments of different kinds of data. If it seems like you need to have a generic
type of event with various different type parameters, consider if your business
problem is well-defined. This is usually a tell-tale sign that your problem space
and bounded context is not well defined.

E X A M P L E - OV E R LOA D I N G E V E N T D E F I N I T I O N S

There exists a simple website where a user can read a book or watch a movie.
When the user proceeds to engage with the website, say by opening the book up
or by starting the movie, a backend service publishes an event of this
engagement, named ProductEngagement, into an event stream. The data structure
of this cautionary-tale event may look something like this.

TypeEnum: Book, Movie

ActionEnum: Click

ProductEngagement {
productId: Long,

You have 2 days le in your trial, Reddychan. Subscribe today. See pricing options.
productType: TypeEnum,

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch03.html 3/5
4/9/2020 3. Communication and Data Contracts - Building Event-Driven Microservices

actionType: ActionEnum
}

A new business requirement comes in - we want to track who watched the movie
trailer before watching the movie. There are no previews for the books we serve,
and though a boolean would suit the movie watching case, we need to allow it to
be nullable for book engagements.

ProductEngagement {
productId: Long,
productType: TypeEnum,
actionType: ActionEnum,
//Only applies to type=Movie
watchedPreview: {null, Boolean}
}

At this point, it can be seen that watchedPreview has nothing to do with Books,
but it’s added into the event definition anyways since we’re already capturing
product engagements this way. If we are feeling particularly helpful to our
downstream consumers, we can add a comment in the schema to tell them that
this field is only related to type=Movie.

One more new business requirement comes in - we want to track users that place
a bookmark in their book, and what page it is on. Again, because we have a
single defined structure of events for product engagements, our course of action
is constrained to adding a new action entity (Bookmark) and adding a nullable
PageId field.

TypeEnum: Book, Movie

ActionEnum: Click, Bookmark

ProductEngagement {
productId: Long,
productType: TypeEnum,
actionType: ActionEnum,
//Only applies to type=Movie
watchedPreview: {null, Boolean},
//Only applies to type=Book,Action=Bookmark
pageNumber: {null, Int}
}

As you can see by now, just a few changes in business requirements can greatly
complicate a schema that is trying to serve multiple business purposes. This puts
added complexity on both the producer and consumer of the data, as they both
must check for the validity of the data logic. The complexity of the data to be
collected and represented will always exist, but by following single responsibility
principles you can decompose the schema into something more manageable.
Let’s see what this would look like if we split each schema up according to single
responsibilities:

MovieClick {
movieId: Long,
watchedPreview: Boolean
}

BookClick {
bookId: Long
}

BookBookmark {
bookId: Long,
pageNumber: Int
}

The productType and actionType enumerations are now gone, and the schemas
have been flattened out accordingly. There are now three event definitions
instead of just a single one, and while the schema definition count has increased,
the internal complexity of each schema is greatly reduced. Following the
recommendation of one event definition per stream would see the creation of a
new stream for each event type. Event definitions would not drift over time, the
triggering logic would not change, and consumers could be secure in the stability
of the singular purpose event definition.

The takeaway from this example is not that the original creator of the event
definition made a mistake. In fact, at the time, the business only cared about any
product engagements but no specific product engagement, and so the original
definition is quite reasonable. As soon as the business requirements changed to
be able to track movie-specific behaviour, the owner of service needed to re-
evaluate if the event definition was still serving a single purpose, or if it was
instead now covering multiple purposes. Due to the business change of needing
finer resolution to lower-level details of an event, it became clear that, while the
event could serve multiple purposes, it soon would become complex and
unwieldy to do so.

It’s worth spending time to determine how your schemas may evolve over time.
Identify the main business purpose of the data being produced, the scope, the
domain, and if you are building it in such a way that has a singular purpose.
Validate that the schemas accurately reflect business concerns, especially for
systems that cover a broad scope of business function responsibility. It could be
that the there is misalignment between the business scope and the technical
implementation. Finally, evolving business requirements may require that the
event definitions be revisited and changed beyond just incremental definitions of
a single schema. Events may need to be split up and redefined completely should
sufficient business changes occur.

Minimize the size of events

Events work well when they’re small, well-defined and easily processed. Large
events can and do happen though. Generally these larger events are representing
a lot of contextual information. Perhaps they have collected many data points that
are related to the given event, and are simply just a very large measurement of
something which occurred.

There are several considerations when looking at a design that produces a very
large event. Ensure that the data is directly related to the event and relevant to the
event that happened. Additional data may have been added to an event “just in
case”, but may not be of any real use to the downstream consumers. Ensure that
the data within is directly related and relevant to the business needs at hand. If
you find that all the event data is indeed directly related, take a step back and
look at your problem space. Does your microservice require access to the data?
You may wish to evaluate the bounded context to see if the service is performing
a reasonably sized amount of work. Perhaps the service could be reduced in
scope with additional functionality split off into its own service.

This scenario is not always avoidable though - some event processors produce
very large output files (perhaps a large image) that are much too big to fit into a
single message of an event stream. In these scenarios a pointer to the actual data
may be used, but this approach should be used sparingly. This adds risk in the
form of multiple sources of truth and payload mutability, as an immutable ledger
cannot ensure the preservation of data outside of its system. This is a design
tradeoff though, and it can be used when the need arises.

Involve Prospective Consumers in the Event Design

When designing a new event it is important to involve any anticipated consumers

of this data. A consumer will understand their own needs and anticipated
business functions better than the producers and may help in clarifying
requirements. Consumers will also get a better understanding of the data coming
their way. A joint meeting or discussion can shake out any issues around the data
contract between the two systems.

You have 2 days le in your

Avoid trial,
events Reddychan.
as semaphores or signals Subscribe today. See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch03.html 4/5
4/9/2020 3. Communication and Data Contracts - Building Event-Driven Microservices

It is best to avoid using events as a semaphore or a signal. These events simply

indicate that something has occurred without being the single source of truth for
the results.

Consider a very simple example where a system outputs an event indicating that
“work is completed” for an arbitrary job. Although the event itself indicates the
work is done, the actual result of the work is not included in the event. This
means that to consume this event properly, you must find where the completed
work actually resides. Once there are two sources of truth for a piece of data,
consistency problems become very real.

Summary
In this chapter we looked at the need for a data contract between event-driven
microservices, and the tradeoffs between explicit and implicit schemas. I
provided a number of guidelines to show the need for focused, singular-purpose
events, and illustrated some of the pitfalls that tend to happen over time with
poorly-designed event schemas.

av

Settings / Support / Sign Out

© 2020 O'Reilly Media, Inc. Terms of Service / Privacy Policy
PREV NEXT
⏮ ⏭
2. Event Driven Microservice Fundamentals 4. Integrating Event-Driven Architectures with Existing Systems

You have 2 days le in your trial, Reddychan. Subscribe today. See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch03.html 5/5