System Complexity - The Engineer's Nightmare

A few years ago I read a book called "A Philosophy of Software Design" by John Ousterhout. It is a thin but very dense book. I made lots of notes and then forgot about it. However, without remembering properly, I stumbled upon many concepts mentioned in this text again and again.

"A Philosophy of Software Design" focuses on the complexity in a system from the perspective of code. What I think though is, that its points can be applied to the complexity of overall system architectures as well.

Recently, I stumbled upon a PhD thesis called "System Design and the Cost of Architectural Complexity". It underlines my gut feeling that system complexity most likely is very costly for a software project.

With the rise and sometimes overuse of microservices - as if it would be the only way to write software - I stumble upon complex systems again and again. This can be due to the actual code which is complex. Though, the more I focus on system design and "the system" as a whole I see complexity in system architectures as well.

Let's first have a look at how complexity is defined in the context of a software system.

What is complexity?

"Anything related to the structure of a software system that makes it hard to understand and modify the system" ^[1]

Others have been even more concrete. On the one hand, there is "program complexity" ^[2] which is related to control, size, modularity, information content and data structures. On the other hand, there is "psychological complexity" ^[3] which is related to problem comprehension, translation and system design. Additionally, it is "a measure of the resources expended by another system while interacting with a piece of software. If the interacting system is people, the measures are concerned with human efforts to comprehend, to maintain, to change, to test, etc., that software." ^[4]

We can also distinguish complexity by the scope that we apply. We can look at the complexity of the source code (e.g., size of and number of branches in classes or functions), which I will call code complexity going forward. We can also look at the system as a whole (e.g., interfaces between modules or services), which I will call architectural complexity.

What do we perceive as complex and what not when we work on a system? In my experience, this doesn't necessarily depend on the size of the system. A big system can be easy to work on and not perceived as complex whereas a small system with complex parts is usually perceived as so.

When it comes to the results of having overly complex systems, Sturtevant conveys the following:

Complexity in a design always has the potential to be problematic. Even when it remains manageable, it causes project delays, defects and other forms of waste. When we lose control of complexity, project failures can sink firms and accidents can cause property destruction and loss of life. ^[5]

Over time, the software engineering community identified that growing systems need a separate role that manages complexity. Smaller systems can be digested by single individuals even when they write it. At some point, when hundreds of developers work on those systems, they cannot be understood solely by the people who write it. Software developers need to focus on the details a lot. It becomes increasingly difficult to keep an eye on the system as a whole. That's why the role of the Software Architect exists.

A major goal of the Software Architect is to manage complexity:

[She manages the] structural complexity in a design so as to keep the dynamic and emergent complexity of a system in operation well understood and controlled. This is because accidents are often caused by unanticipated interactions between parts. [...] These types of failures are pernicious because they result from the structure of the system as a whole that results in insufficiently constrained behavior. When emergent properties of the system as a whole are unanticipated, they often cause emergencies.

Conclusion

In my experience code is read more often than written. Code is also extended more often than re-written. The same applies to a system architecture. The closer code comes to production, the harder it becomes to make changes. Once code is in production and your system is set up to accept requests of users that pay your bills and salary, it is much harder and expensive to change direction.

Having a look at system complexity is always worth it if you are in the game for the long-run. Working on a complex system that is not complex because of the domain but because of how it has been set up, will make generations of developers unhappy and frustrated. It can even be a reason why your engineers do not want to stay for longer.

In following posts I will go into how to keep complexity under control. I will also take a look at what happens when you don't manage complexity.

For now I want to leave you with a quote by Fred Brooks that resonated well with me:

Software entities are more complex for their size than perhaps any other human construct because no two parts are alike [...] In this respect, software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound [...]

Likewise, a scaling-up of a software entity is necessarily an increase in the number of different elements. In most cases, the elements interact with each other in some nonlinear fashion, and the complexity of the whole increases much more than linearly [...]

Many of the classic problems of developing software products derive from this essential complexity and its nonlinear increases with size. From the complexity comes the difficulty of communication among team members, which leads to product flaws, cost overruns, schedule delays. From the complexity comes the difficulty of enumerating, much less understanding, all the possible states of the program, and from that comes the unreliability. From complexity of function comes the difficulty of invoking function, which makes programs hard to use. From complexity of structure comes the difficulty of extending programs to new functions without creating side effects. From complexity of structure come the unvisualized states that constitute security trapdoors.

Not only technical problems, but management problems as well come from the complexity. I makes overview hard, thus impeding conceptual integrity. It makes it hard to find and control all the loose ends. It creates the tremendous learning and understanding burden that makes personnel turnover a disaster. ^[6]

What experiences have you made with complex systems? How was it to work on such a system? What could you do to reduce complexity?