Doc quassel protocols » History » Version 6
Version 5 (Sputnick, 02/27/2013 01:36 AM) → Version 6/20 (Sputnick, 02/27/2013 02:28 AM)
h1. The Quassel Protocol
h2. Overview
When we talk about the "Quassel protocol", we mean the format of data sent between a Quassel core and connected Quassel clients. At the moment (i.e., as of version 0.8), only one protocol - the "legacy protocol" - is in use. It has evolved from Quassel's early days and hasn't really changed all that much over the years. However, back then we didn't really expect Quassel to ever become popular, much less other developers writing alternate clients such as QuasselDroid or iQuassel. Accordingly, instead of designing (and documenting) a well-defined and easy-to-use format, we chose a rather pragmatic approach for sending data over the wire: Because Qt already had a facility to (de)serialize arbitrary data types over a binary stream - using QDataStream - we simply went with that.
While being both straightforward and easy to implement in Quassel, this choice turned out to be rather unlucky in retrospect:
* QDataStream's serialization format is not the most efficient one. In particular, strings are serialized as UTF-16, which means that almost half of the data exchanged between client and core is nullbytes. However, this is partially compensated by Quassel using compression if possible.
* Speaking of which, we don't use streaming compression, which means that lots of potential for benefitting from recurring strings is not used. And since many of the objects we send are key/value maps which tend to have the same set of keys every time, this does matter in practice.
* And to add insult to injury, we waste even more space all over the place because we simply didn't think about optimizing the protocol. Mobile use of Quassel was just not on our radar in 2005.
* The serialization format is nowhere documented in a concise and complete way. Yes, there's documentation somewhere in Qt for built-in types; for Quassel's own types however, one would have to hunt through the source. And without reading (and understanding) some rather icky parts of Quassel code, it's close to impossible to understand what's going on even if one manages to deserialize the binary data into proper objects. Bad news for people wanting to write alternate clients. Amazingly, some smart people still managed to reverse-engineer the protocol...
To fix these and more issues, we're now planning to replace the legacy protocol by something more sensible. As the first (and most complicated) step, work is now underway to implement a protocol abstraction that will then allow us to much more easily support alternative formats. As a neat side effect, the resulting refactoring also will make some core parts of the code (e.g. SignalProxy) much nicer to understand.
h2. Protocol Abstraction - The Master Plan
h3. Requirements for new protocols
h3. Keeping compatibility
TBD: for how long?
h2. Abstract View [DRAFT]
h3. Handshake
h4. Probing
Because we might want to support more than one protocol, we cannot start to send messages right away. First, both client and core need to agree on which protocol to use and if to enable things like compression or SSL. Therefore, right after the connection has been established, a few well-defined bytes are exchanged to probe for the capabilities on both ends and to determine in which way the real data is going to be exchanged.
# The client sends a 32 bit value to the core. The lowest byte contains the supported protocols as defined in the Protocol::Type enum. The upper 24 bit contain the magic number [TBD] to indicate that a post-0.9 client is connecting. Since the resulting value is larger than 0x00400000, legacy (pre-0.9) cores will immediately close the connection. The client can detect this and reconnect in compatibility mode.
# The core replies with a 32 bit value. The lowest byte specifies the protocol it has selected based on what the client supports. Bits 8-15 contain the protocol version. The upper 16 bit contain a protocol-specific set of features that the core supports - things like compression and SSL support.
# The client sends another 32 bit value containing its own version of the protocol in bits 8-15, and the protocol-specific features to be used in bits 16-31.
# At this point, both client and core enable the agreed feature set (e.g. turn on compression and encryption). The remaining communication happens in the form of protocol messages.
_Note: The legacy protocol determines the supported and enabled feature set, as well as the protocol version, only during the handshake phase. Therefore, both compression and encryption are turned on later in the process. Also, a client reconnecting in compatibility mode will skip the probing phase and proceed directly with the handshake._
h4. Init and Authentication
h3. SigProxy Mode
h2. On-Wire Format
h3. Legacy Protocol
h2. Overview
When we talk about the "Quassel protocol", we mean the format of data sent between a Quassel core and connected Quassel clients. At the moment (i.e., as of version 0.8), only one protocol - the "legacy protocol" - is in use. It has evolved from Quassel's early days and hasn't really changed all that much over the years. However, back then we didn't really expect Quassel to ever become popular, much less other developers writing alternate clients such as QuasselDroid or iQuassel. Accordingly, instead of designing (and documenting) a well-defined and easy-to-use format, we chose a rather pragmatic approach for sending data over the wire: Because Qt already had a facility to (de)serialize arbitrary data types over a binary stream - using QDataStream - we simply went with that.
While being both straightforward and easy to implement in Quassel, this choice turned out to be rather unlucky in retrospect:
* QDataStream's serialization format is not the most efficient one. In particular, strings are serialized as UTF-16, which means that almost half of the data exchanged between client and core is nullbytes. However, this is partially compensated by Quassel using compression if possible.
* Speaking of which, we don't use streaming compression, which means that lots of potential for benefitting from recurring strings is not used. And since many of the objects we send are key/value maps which tend to have the same set of keys every time, this does matter in practice.
* And to add insult to injury, we waste even more space all over the place because we simply didn't think about optimizing the protocol. Mobile use of Quassel was just not on our radar in 2005.
* The serialization format is nowhere documented in a concise and complete way. Yes, there's documentation somewhere in Qt for built-in types; for Quassel's own types however, one would have to hunt through the source. And without reading (and understanding) some rather icky parts of Quassel code, it's close to impossible to understand what's going on even if one manages to deserialize the binary data into proper objects. Bad news for people wanting to write alternate clients. Amazingly, some smart people still managed to reverse-engineer the protocol...
To fix these and more issues, we're now planning to replace the legacy protocol by something more sensible. As the first (and most complicated) step, work is now underway to implement a protocol abstraction that will then allow us to much more easily support alternative formats. As a neat side effect, the resulting refactoring also will make some core parts of the code (e.g. SignalProxy) much nicer to understand.
h2. Protocol Abstraction - The Master Plan
h3. Requirements for new protocols
h3. Keeping compatibility
TBD: for how long?
h2. Abstract View [DRAFT]
h3. Handshake
h4. Probing
Because we might want to support more than one protocol, we cannot start to send messages right away. First, both client and core need to agree on which protocol to use and if to enable things like compression or SSL. Therefore, right after the connection has been established, a few well-defined bytes are exchanged to probe for the capabilities on both ends and to determine in which way the real data is going to be exchanged.
# The client sends a 32 bit value to the core. The lowest byte contains the supported protocols as defined in the Protocol::Type enum. The upper 24 bit contain the magic number [TBD] to indicate that a post-0.9 client is connecting. Since the resulting value is larger than 0x00400000, legacy (pre-0.9) cores will immediately close the connection. The client can detect this and reconnect in compatibility mode.
# The core replies with a 32 bit value. The lowest byte specifies the protocol it has selected based on what the client supports. Bits 8-15 contain the protocol version. The upper 16 bit contain a protocol-specific set of features that the core supports - things like compression and SSL support.
# The client sends another 32 bit value containing its own version of the protocol in bits 8-15, and the protocol-specific features to be used in bits 16-31.
# At this point, both client and core enable the agreed feature set (e.g. turn on compression and encryption). The remaining communication happens in the form of protocol messages.
_Note: The legacy protocol determines the supported and enabled feature set, as well as the protocol version, only during the handshake phase. Therefore, both compression and encryption are turned on later in the process. Also, a client reconnecting in compatibility mode will skip the probing phase and proceed directly with the handshake._
h4. Init and Authentication
h3. SigProxy Mode
h2. On-Wire Format
h3. Legacy Protocol