MSC3005: Streaming Federation Events

[ShadowJonathan]

Rendered

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>

[ShadowJonathan]

Wrote some sleep-deprived comments that i know are absolute issues that need to be addressed here, this MSC requires more comments.

[ShadowJonathan]

Small PSA: I might refactor this MSC into a "generic streaming" spec instead, due to some insightful discussion i had.

[ShadowJonathan]

This can also be ext_ops, though i wasn't sure.

[ShadowJonathan]

The original spec is incredibly lax with how the "processing results" are defined, so I don't have a choice to make it just as ambiguous.

[ShadowJonathan]

Concerns from @Half-Shot re: this, that the spec should avoid bloat, this is a new protocol which'd be implemented in the spec, and matrix has so far been wanting to avoid reinventing the wheel, generally being in favour of using something else that's "out there" before using this protocol.

[ShadowJonathan]

Some extra discussion on #matrix-spec:matrix.org has yielded the following feedback i'll be implementing;

• Versioning (Still thinking about it, I need to think of a proper way that doesn't inhabit the entire front-end of the stream session)
• Changing the assumption of the underlying connection to be a framed one.
  • This is done to deduplicate efforts of framing and make it possible to take advantage of underlying framing in many situations.
  • Pure-streaming connection types will have to implement a "shim" of this, e.g. plastering opcode+payload length in front to segment each chunk into a frame, this is up to those individual proposals to implement and specify (if needed)

  • "the underlying connection MUST be framed, either through native capabilities (WebSocket frames), or emulated (TCP with payload length header)"

• Make the heartbeat opcodes completely optional, and put these into a dedicated extensions (with their extension frames being freely mappable)

[ShadowJonathan]

I believe that this MSC does no longer have much going for it at this stage.

While I believe that a binary&streaming-based solution can speed up raw transfers and decrease raw latency of federation traffic, it will increase implementation complexity, and possibly heighten the barrier to entry for a matrix homeserver.

I do ask the SCT to consider ways to speed up raw transfers, but I don't think I have confidence that this MSC, at this time, and possibly with me as the author, can achieve something like that.

Thus, thanks for your help with this, I am abandoning this MSC.

For anyone reading this at any point in the future, I invite them to take note from the proposal contents to make a better solution with a better context or perspective on this issue.