Digital Systems Complexity Reduction by studying & challenging Protocol notion

stman

Dear friends,

Welcome here, it's gonna be way better for our work. I was becoming mad with mastodon.

In my next post, I am going to try to resume all what was said so far on the topic.

This work is almost finished, and it is a key one, and we can all be very proud of ourselves because we had no garantee at all we would find solutions, and we're about to : It's at our fingertips.

As a small introduction, I'd remind everybody that we could not find solutions to this specific issue in the past because we were not looking under the right angle of view, and we were not looking at the right things either, for a few reasons that we will have to write down too : It is always very important to learn from our mistakes. So writing down too WHY we could not tackle the issue under the right angle of view before is indeed part of the job, and it will help all of us in the future to solve other issues : It's maturity and experience gained !

stman

Okay so let's move forward a little bit here.

History of protocols :

70 years ago... Hardware protocols appeared.

Protocols appeared first in hardware, with analog and then digital logic electronics, a 80 years ago, they are the successors of the simple multiplexers both in analog and digital logic electronics.

Their primary original function was to reduce wire counts on PCBs, and to allow a better modularization of the circuits designed, enabling a true huge complexity reduction of the overall design thanks to more efficient and simple interfaces for whole hierarchy of modules composing any design (mostly standalone systems, by that time).

They implicitely or explicitely introduced the notion of address and somehow of registers : The concept of memory space was born. When addresses lines are not present, they are indeed always present implicitly.

Then came the microprocessor, the computer & the computer network concepts in the 1970...

On of the best example of these hardware protocols is what is called a classical "microprocessor bus" on a motherboard implementing a design with a CPU.

When the first microprocessors and analog (PSTN) and digital telecommunication networks appeared around 1970, and as those standalone digital system started either being remotely inter-connected in the form of networks with each others, or that those first standalone computers started including expansion connectors, allowing several kinds of "computer peripherals" to connect to them, the protocols were used to simplify their design, at the cost of being asked to handle more functionalities as "distances were increasing" between the different modules or devices or computers to inter-connect.

This is were the "emulation of many physical wire lines" interconnecting digital modules with hardware protocols evolved, and error detection and correction mechanisms were needed, and for "digital computer networks", the need for routing functionality within a network also appeared : These new needs were sometimes too complex to be fully implemented easily into digital logic, and they were sometime also needing more frequent update, and the protocols started going from pure simple hardware only protocols (Typical examples : RS232C serial ports and Centronics printers parallel port de-facto standard), to hardware and software combined protocols (Typically, a hardware protocol controller, its optional software driver (Typicaly software interrupt subroutines written in assembly language, handling the low level information exchange with the concerned hardware controller, and a software protocol or software protocol stack, the whole thing forming an hybrid hardware and software made complex protocol stack). This also came with the introduction of the protocol stack notion.

Then shortly after that came the purely software implemented protocols...

Here we are situated around the 1980' in terms of history.

These purely "software protocols" or "software protocol stacks" implementations were mostly used for object oriented programming languages, and computer digital communication networks. And this is from here that things started going nuts, because as more and more high level abstractions were introduced with protocols, engineers started forgetting their primary goal that had motivated their initial invention.

The protocol concept drifted from the first roles it was handling with the first hardware protocols in standalone boards.

Digital systems engineering community has started paying more and more attention to the role of the different protocols abstractions composing a complex multi-level protocol stack (The tipycal example being the famous ISO model), and the original first role of the first historical hardware protocols (Wire count reduction on a PCB, enabling better modularity within different architectured modules within a standalone PCB) slowly faded away, so did the complexity reduction they were bringing to those old standalone boards.

THIS POST IS NOT FINISHED AT ALL - TO BE CONTINUED VERY SOON
This is just the first part, dealing with history of protocols.

theruran

It would be interesting to actually measure the amount of code dedicated to protocols on a typical Linux system. I do not see hardly any research of this kind. No one seems to be interested in measuring the complexity of a modern computer system, including all the software. Given that all the source code is available, at least for the software side, it is just a matter of collecting the data and running the calculations. To limit the measure to protocols would probably require manual classification. I believe it is also important to note that protocols also include ad-hoc protocols (that are not standardized), and here you may find a ton of complexity. Because of the limitations of *nix operating systems, everything must be interpreted from raw bytestreams. Sure, "text is the universal interface" until you get into Unicode, itself a kind of protocol, and then everything goes to shit. Now we have a proliferation of protocols to deal with these limitations, such as Protocol Buffers, Thrift, ad-hoc JSON, XML-based, and so on. Parsing, serializing and deserializing are considering some of the most boring and despicable tasks for a programmer yet they are necessary in nearly every non-trivial program.

As @natecull had pointed out, we often see screenshots of tweets on Mastodon. This is an admission of failure and ought to be an embarrassment to all software engineers. Mastodon implements its own API over REST+JSON, and it implements ActivityPub which is defined in JSON which also incorporates Linked Data in JSON. It also implements WebFinger, another protocol. All that JSON which includes escaped HTML, is transported over HTTP, which of course runs over TCP/IP, Ethernet, and myriad other Internet technologies and protocols. But all we wanted to see was some attributed text!

A major contribution to all this complexity is that we are stuck with protocols that have not evolved to afford modern needs. And I believe that many computing technologies were designed explicitly to create markets for companies to implement the standards are charge for it. Old protocols made few assumptions about what information would be transferred, and didn't even bother implementing encryption. On the one hand it can be helpful not to be prescriptive in a standard, on the other hand it created tons of work for engineers and contributed to so much complexity.

stman

theruran Hello James,

Yes, it would be very interesting to do so, to record these statistics about the amount of code dedicated to protocols.
It is clear that the percentage of code we would find would be an underestimation of the reality.

I have found a way to finish the new drawing I am preparing, displaying protocol usage for :

Strictly hardware implemented protocols or protocols stacks (With no software driver or software protocol or software protocol stack) needed to access a remote memory space (Example : Typically, the main CPU bus on a motherboard)
Mixed hardware and software implemented protocols or protocols stack (Part of the protocol or protocol stack is implemented into hardware, and the rest of the protocol or protocol stack implemented in software) needed to access a remote memory space (Example : Typically, TCP/IP, Wifi, Bluetooth, USB, SATA or PCIe standards, as they form a protocol stack, and their implementation is both made out of a dedicated hardware controller handling the low level parts of the protocol stack, while higher levels of the protocol stack are implemented in software).
Strictly software implemented protocols protocol stacks (With no dedicated hardware controller of any kind) needed to access a remote memory space. (Example : Typically, high level desktop applications protocols, usually built on top of protocols like TCP/IP, Bluetooth, or USB).

I am going to use 3D representation in this drawing, to be able to show hardware implemented and software implemented protocols on two different planes, the third plane being used to represent time passing by.

With this new drawing and its specific angle of view focused on protocols and remote memory spaces of all kinds, the understanding of what the CPU does is a bit different from the "usual common understanding" we have. In order to ensure a correct visualization into people's mind, it will be given new definition of the CPU, slightly different from what we usually consider, forcing you to see the whole system as a complex system of digital electronic modules. This is the right level of abstraction to really see where and why things went nuts, therefore allowing more easily to imagine alternatives.

What I find interesting with this format I am having for this drawing is that it is going to be possible to show how asynchronous it is mostly.

Then there are other aspects that are important, it's that most protocols are asymmetrical ones : There is a master and a slave (Typically USB or SATA or PCIe), and most of the time, those roles cannot be reverted. It is interesting to note that some data that cross those numerous protocol layers in stack(s) cross protocols that are all different in terms of symmetry. Some protocols are half duplex, while others are full duplex, this is another thing to note too.

But what can be already said here, is that all this mess represent a very shitty (Latencies are rather random and unpredictable) asynchronous paradigm : Barely forcing any system design working within such paradigm, either at hardware or software levels, or both, to always create almost fully asynchronous applications : The few applications that pretend having a true synchronous mode of operation are cheating, they just used trick to reduce latencies enough to lure end-users being using synchronous or real time applications).

By the way, I hope @natecull will be back here, as the way to communicate with him for me on mastodon has been suspended against our will.

alcinnz

I think it might be useful for me (a developer who works extensively with standards, & who’s actively studying most of the software powering my computer) to share impressions. Even if I haven’t quantified it.

Most of the GNU/etc suite exist to roughly implement some software (POSIX) standard or other in the process of offering vital features, as that helped reduce coordination during their initial development. Also it is not uncommon for components of a FOSS operating system to implement multiple standards, since once you’ve implemented a couple support for a new standard becomes an easy & desirable thing to contribute. I find these projects do increase their lines of code but (with the importers/exporters all being mutually independent) their conceptual complexity remains stable.

I haven’t gotten to restudying Linux yet, but I get the clear impression that the hardware/software protocols it implements takes by far the most complexity to implement. Though USB takes relatively little code, there’s some good ideas there. That consumer hardware is nowhere near standardized enough to be supported by a simple OS. As a volunteer repairer I find Linux invaluable for giving new life to old machines, but a simple OS requires its own hardware.

I cannot comment on pure hardware protocols.

alcinnz

In working on software standards I do see a few issues in the space.

If we categorize standards under the labels “messaging”, “data”, & “process” (distinctions are fuzzy) I see software standards focusing far too much on process standards, & perhaps hardware standards are focused on messaging. As a developer I am much happier dealing with data standards! These give me the most flexibility in delivering the UX & features clients want. While its not ideal for data standards to be redundant, I don’t find it that much of an issue either.

However modern overly-sandboxed computing often seems to make data standards feel unthinkable.

Also I find my fellow standards professionals care far too much about encoding, & they waste a lot of time doing so. There’s an overwhelming insistence that “developers prefer JSON” & as a developer I roll my eyes. RDF advocates repeatedly shoot themselves in the foot here.

This is perhaps fed into by webapps becoming the only legit platform worth considering, & by a drive to appease Google Search.

stman

alcinnz

The state of our researches has changed significantly since the first messages were written here. Many research goals were reached. We found all the answers we were looking for to know if it was possible to get rid of the protocol concept as we know it, and the good news is that not only it is, but we also found at least one way to do it, we found one set of simple minimal conceptual and architectural conditions to design a new digital system paradigm making the protocol concept as we know it completely useless.

This is the result of two years of hard work on this fundamental research abstract topic.

I am going to launch a new thread detailing how we find, and what we find. Protocols and OSI model are now obsolete concepts and models. And this is a very good news, because protocol really do represent most of the complexity and security breaches of current digital systems.

stman

[unknown]

This is the result of two years of hard work on this fundamental research abstract topic.