Software Engineering’s Challenges: Change and Complexity

Every engineering discipline starts off in response to some pressing problem. Software engineering confronts the problems of change and complexity.

The very nature of software—its plasticity—makes it amenable to a continuous cycle of change. It seems rather easy to accomplish. After all, tweaking one statement in a software program can radically alter the program’s behaviour. But such tweaking—little by itself, but considerable in conjunction—can end up
changing the intent of the program’s design in fundamental ways.

It is absurd to expect a car to fly or float. But very often a software system built for one context is expected to function in as drastically different contexts, with the same grace and efficiency. These expectations can be traced to our wide cognitive gap with the use of software. Decades and centuries of using cars and bridges respectively, and millennia of using houses has ingrained in our minds what cars, houses, and bridges can and cannot do. Accordingly, we tune our expectations as well as environmental factors to set the context for these systems to function. In comparison, the use of software amongst a large community of lay users has just begun. Our understanding of how and to what extent software can serve our needs is yet not complete. As a result, the problem of change for software comes primarily from changing user expectations, and also from changes in the environment—technological and social.

Complexity is a complex word and there is no one definition to cover its ken; even reaching a definition is fraught with difficulties. But we need to care about it in life as well as in software engineering as complexity arises out of simplicity, at times suddenly and surreptitiously. Think of a simple computer program of five lines of code. It is straightforward; by carefully reviewing each line, we can hope to have complete knowledge of the program’s structure and behaviour. Now what if, a loop is introduced in the program—a simple construct that executes a set of statements repetitively, until a condition holds. The number of execution paths through the program has significantly increased now, and it has become far more difficult to know for sure what happens in each step when the program runs. This example is just a watered down instance of the combinatorial complexity software systems customarily face. Then there are even more involved issues such as complexity of the problem domain, complexity in the interaction of the various forces—technological, commercial, political—that a software system has to balance to be successful.

A common feature of complex systems is that they are greater than the sum of their parts. Anyone who has worked on a course project to build a piece of software stretching across several files can appreciate the sense of this statement: A piece of software is made of individual files, but it delivers something that merely bunching the files together will not achieve. Now scale-up to a real world system—with hundreds, if not thousands of files; and thousands, if not millions of interfaces between them; perhaps simple by themselves, but certainly complex when functioning together. And this is just one, relatively less significant, facet of software complexity.

A central theme of software engineering is about framing an adequate response to the problems of change and complexity.

Software Engineering: The First Forty Years

The phrase ‘software engineering’ was first used in a public discourse at a NATO Science Committee sponsored conference, held at Garmisch, Germany, from 7th to 11th October, 1968. More than forty years hence, there is still not much unanimity on a definition of software engineering.

According to Boehm, software engineering involves the application of science and mathematics through which the facilities of computer equipment are made useful to human beings via computer programs, procedures, and associated documentation. Pfleeger identifies software engineering with the utilization of tools, techniques, procedures, and paradigms toward quality improvement of the software product. Naur and Randall see software engineering in terms of establishing and using sound engineering principles to obtain economically effective and reliable software that can work efficiently on real machines. According to Freeman and Von Staa, software engineering involves the organized application of methods, tools, and knowledge towards fulfilling stated technical, economic, and human goals for a software-intensive system. Kacmar says simply applying engineering principles to designing and constructing computer software can be termed software engineering. To Schach, software engineering is the discipline that aims at producing fault-free software, to be delivered on time and within budget, which satisfies the user’s needs. Whitmire describes software engineering as a ‘slippery’ term, and says for some it is something that can only be applied to a large project, while to others it is just a ‘figment of collective imaginations’. He gives a working definition as, ‘Software engineering is the science and art of designing and building, with economy and elegance, software systems and applications so they can fill the uses to which they may be subjected’

Many views and much variety!

However, there seems to be little room for debate on Brooks’ characterization of software - software is inherently complex; software must be made to conform to existing interfaces; software is constantly subject to change; and software is invisible and unvisualizable.

Software engineering can be viewed as an organized response to the challenges these very characteristics of software engenders. Software engineering is concerned with the problem of change and the problem of complexity.

Between change and complexity, does one necessarily insinuate the other?

What do you think?

The Lives of a Cell

The Lives of a Cell by Lewis Thomas is one of those rare books; small, sweet, and sublime. When I picked it up at a second hand book store, I was impressed by its size, or the lack of it: 6.9 x 4.2 x 0.7 inches. I was not intimidated by the author.  I was not acquainted with Lewis Thomas’ name at that time, not even at the Wikipediac level. I was later to learn – from Wikipedia – that Thomas was a physician, poet, etymologist, essayist, administrator, educator, policy advisor, and researcher; he attended Princeton and Harvard as a student, and went on to become the Dean of Yale Medical School and New York University School of Medicine, and President of Memorial Sloan-Kettering Institute. As evident, he had much preparation and many positions.

His writing has no affectation of any of them.

I am glad I was drawn to the book by its dimensions. It is of a size which creates space for itself in any luggage, however full. Thus it was very easy reading while travelling or waiting to travel. While reading, I discovered the books many dimensions.

Thomas qualifies the title “The Lives of a Cell” by the subtitle “Notes of a Biology Watcher”. That a person of his eminence calls himself a mere “watcher” indicates some humility. A glance at the headings of the 29 essays also points to considerable versatility. Thomas has written specifically about biology, society, language, music, information, computers, the nature of scientific research and the planning of science et al.; and generally about life and death -- the mortality of Man as well as the annihilation of man. Even after more than three and half decades since the book came out, many of Thomas’ ideas are fresh, astonishingly few are dated, and some are eerily prescient. For example, some of his projections seem to prognosticate an increasingly wired world, the burgeoning online social media of the current decade, even Second Life avatars (“software selves”, he calls them.)

Thomas probably did not want to weave a common thread through the collection, but connection seems to be a unifying theme of the book. Thomas talks about the wonder of how cells, organisms, species, ideas, are connected across the terrestrial and celestial worlds and he takes the wonder to a new level. He talks with a scientist’s precision, and an artist’s poetry.

The Lives of a Cell is a masterclass for appreciating the complexity of our living world, while awaking to its romance. 

Ontogeny recapitulates phylogeny

Well, that’s quite a mouthful, I agree – ontogeny recapitulates phylogeny. What it reflects is rather interesting. The phases in the development of an individual organism mimic to an extent its ancestral form. For example, the human embryo develops gills bars which are then modified for other purposes. But, “to an extent”. Biologist point out that the so called recapitulation occurs at a very coarse level, bereft of details [The Sciences of the Artificial, Third Edition by Herbert A. Simon, MIT Press, 1996].

But this serves as an exciting analogy for something similar that happens while we are learning a subject. The student’s progression seems to resemble the progression of the discipline from childhood to maturity.

Take for example, software engineering. First there was waterfall, then the iterative and incremental methods, and then agile. The streamlined regularity of waterfall is most useful for the tyro software engineer; there are copious milestones and checkpoints so that (s)he has enough guidance to get along. The essence of iterative and incremental development is appreciated with experience, when the power of feedback becomes manifest. Finally, agile in all its facility marks the transition from instruction to interaction. The software engineer now has sense enough not to need micro-management, (s)he can leverage the typical loose-coupling of an agile team. (Indeed, a criticism of agile has been its over-dependence on “premium people” to be successful). So the progression of a software engineer from a neophyte to a seasoned pro is marked by a progressing embrace of the more mature development methodologies.

What is the implication of this trend on how we teach software engineering to students? Do we introduce waterfall, then iterative and incremental development, then agile, in sequence? Or should we indoctrinate students in the most mature methodology of the day and touch upon others as historical footnotes? In conventional sciences the dogma of the day is “taught”, while its ancestors may (or may not) be mentioned – for example ether in physics, or phlogiston in chemistry. But unlike ether and phlogiston, waterfall is not merely of historical import, systems of relevant scope are successfully built using waterfall, very much in the present.

So must software engineers learn and unlearn (or at least learn to supersede) a methodology before moving on to the next? Or is there a more organic advancement?

Blueprint versus Recipe

Simon presents two different but complementary perspectives of complex systems: state descriptions versus process descriptions, or simply, blueprint versus recipe. For a simple system, the congruence of these two views is almost intuitive. A circle is locus of points all of which are at an equal distance from another point (the center). To draw circle, anchor one arm of a compass at a fixed point (the center), and rotate the other arm until it returns to the starting point. The first statement is a blueprint for a circle, the second, a recipe for drawing a circle. A blueprint characterizes the world as sensed, while a recipe reflects upon the way it as acted upon. [The Sciences of the Artificial, Third Edition by Herbert A. Simon, MIT Press, 1996].

For software systems, endowed as they are with Brooksean “invisibility” and “unvisualizability”, neither of the blueprint or recipe views are easy to arrive at. Describing the structure of code may not necessarily be a complete or even accurate description of the software subsuming the code. And although directions for constructing a unit of code can be specified with precision, there is no guarantee the desired behavior will be derived from that code.

Then how should we go about learning how to build software systems? Do we focus on absorbing knowledge about the blueprint or acquiring skills around the process? This confusion is often reflected in software engineering textbooks. Some authors take the descriptive route, trying to narrate software's characteristics – in all its quirks – using natural language. Others assume away all the messiness and indulge in the purity of mathematical models and their transformations. The former allows for easier reflection of the blueprint, the latter for the recipe. Both approaches are useful, but not one in the exclusion of the other.

That elusive balance is the key.

Use and renewal

In one of Tagore's dance dramas, the diva asks her cronies to renew her with new jewelry. How much renewal she had by such means, and what she did with such renewal, is what the dance and the drama is all about, and I will not divulge it here. But those of us, like the diva, who are obliged to recognize the decay of flesh, crave at one point or the other for renewal. Trees have at least the ritual of renewal. Once a year old leaves shed, and new ones shine. We alas, have no such facility -- the ritual of renewal, or renewal itself.

Decay comes with age, with use, with wear and tear, and tears. These come with living. In every mechanical system -- or systems with at least some mechanical parts, like the human body -- parts will wear, mostly the moving ones. But in software, there are no moving parts. So what wears a software system down, what wearies it?

This question has been asked and answered many times. Most fascinatingly, each time the question has been a little different, and the answer commensurately so. Thus it is no more one question with one answer. The decay of software comes with use, as new and newer functionality is stacked on an unsuspecting initial design. It is hoped that the underlying plasticity of software will be equal to anything. But what is unique about the use of software that warrants such wanton expectations? What is that which sets apart the use of software from the use of say a car, or bridge? We take a car or a bridge as given, love it or hate it, or become indifferent to it, and move on to another car, or another bridge (if we are in a car). But with software we are ever so tantalizingly close to what is, and what we want it to be. For software, the die is never cast. Frequently software never dies, one system merely reincarnates to another. Like our lives, there is always change. Change like the diva's new jewels, has the power to renew. Or thus the diva budgeted.

Ruminations on renewal apart, a key challenge of software engineering is to know what to do about changing requirements, or at least know what it does to a software system, take it as given, and live life from there.

Software metrics? Oh! You mean lines of code.

With due deference, I beg to differ. Lines of code (LOC) was once the software metric, is now merely a software metric. One among many other software metrics that are being continually conceived, created, and enhanced as software engineering grapples with deepening complexity. There was a time when the biggest challenge with writing code was to make it run, just as the earliest buildings humans built were good if they stood. You could count bricks and measure up such buildings. But then we learnt to build structures like the Taj Mahal. Just as it would be facile to try and measure the Taj by the weight of its marble, software systems of today are intricate interactions of components that are way beyond the ken of code count. Not only are the components larger than their lines of code, the interaction is larger than the component count. Simple intuitive metrics – unrelated to LOC – can give us insights into how design happens, and how responsibilities can be delegated (equitably, one hopes!) to components. I look forward to lively discussions on this and other topics at a forthcoming conference.