Software Engineering’s Challenges: Change and Complexity

Every engineering discipline starts off in response to some pressing problem. Software engineering confronts the problems of change and complexity.

The very nature of software—its plasticity—makes it amenable to a continuous cycle of change. It seems rather easy to accomplish. After all, tweaking one statement in a software program can radically alter the program’s behaviour. But such tweaking—little by itself, but considerable in conjunction—can end up
changing the intent of the program’s design in fundamental ways.

It is absurd to expect a car to fly or float. But very often a software system built for one context is expected to function in as drastically different contexts, with the same grace and efficiency. These expectations can be traced to our wide cognitive gap with the use of software. Decades and centuries of using cars and bridges respectively, and millennia of using houses has ingrained in our minds what cars, houses, and bridges can and cannot do. Accordingly, we tune our expectations as well as environmental factors to set the context for these systems to function. In comparison, the use of software amongst a large community of lay users has just begun. Our understanding of how and to what extent software can serve our needs is yet not complete. As a result, the problem of change for software comes primarily from changing user expectations, and also from changes in the environment—technological and social.

Complexity is a complex word and there is no one definition to cover its ken; even reaching a definition is fraught with difficulties. But we need to care about it in life as well as in software engineering as complexity arises out of simplicity, at times suddenly and surreptitiously. Think of a simple computer program of five lines of code. It is straightforward; by carefully reviewing each line, we can hope to have complete knowledge of the program’s structure and behaviour. Now what if, a loop is introduced in the program—a simple construct that executes a set of statements repetitively, until a condition holds. The number of execution paths through the program has significantly increased now, and it has become far more difficult to know for sure what happens in each step when the program runs. This example is just a watered down instance of the combinatorial complexity software systems customarily face. Then there are even more involved issues such as complexity of the problem domain, complexity in the interaction of the various forces—technological, commercial, political—that a software system has to balance to be successful.

A common feature of complex systems is that they are greater than the sum of their parts. Anyone who has worked on a course project to build a piece of software stretching across several files can appreciate the sense of this statement: A piece of software is made of individual files, but it delivers something that merely bunching the files together will not achieve. Now scale-up to a real world system—with hundreds, if not thousands of files; and thousands, if not millions of interfaces between them; perhaps simple by themselves, but certainly complex when functioning together. And this is just one, relatively less significant, facet of software complexity.

A central theme of software engineering is about framing an adequate response to the problems of change and complexity.