Software Engineering: The First Forty Years

The phrase ‘software engineering’ was first used in a public discourse at a NATO Science Committee sponsored conference, held at Garmisch, Germany, from 7th to 11th October, 1968. More than forty years hence, there is still not much unanimity on a definition of software engineering.

According to Boehm, software engineering involves the application of science and mathematics through which the facilities of computer equipment are made useful to human beings via computer programs, procedures, and associated documentation. Pfleeger identifies software engineering with the utilization of tools, techniques, procedures, and paradigms toward quality improvement of the software product. Naur and Randall see software engineering in terms of establishing and using sound engineering principles to obtain economically effective and reliable software that can work efficiently on real machines. According to Freeman and Von Staa, software engineering involves the organized application of methods, tools, and knowledge towards fulfilling stated technical, economic, and human goals for a software-intensive system. Kacmar says simply applying engineering principles to designing and constructing computer software can be termed software engineering. To Schach, software engineering is the discipline that aims at producing fault-free software, to be delivered on time and within budget, which satisfies the user’s needs. Whitmire describes software engineering as a ‘slippery’ term, and says for some it is something that can only be applied to a large project, while to others it is just a ‘figment of collective imaginations’. He gives a working definition as, ‘Software engineering is the science and art of designing and building, with economy and elegance, software systems and applications so they can fill the uses to which they may be subjected’

Many views and much variety!

However, there seems to be little room for debate on Brooks’ characterization of software - software is inherently complex; software must be made to conform to existing interfaces; software is constantly subject to change; and software is invisible and unvisualizable.

Software engineering can be viewed as an organized response to the challenges these very characteristics of software engenders. Software engineering is concerned with the problem of change and the problem of complexity.

Between change and complexity, does one necessarily insinuate the other?

What do you think?

The Lives of a Cell

The Lives of a Cell by Lewis Thomas is one of those rare books; small, sweet, and sublime. When I picked it up at a second hand book store, I was impressed by its size, or the lack of it: 6.9 x 4.2 x 0.7 inches. I was not intimidated by the author.  I was not acquainted with Lewis Thomas’ name at that time, not even at the Wikipediac level. I was later to learn – from Wikipedia – that Thomas was a physician, poet, etymologist, essayist, administrator, educator, policy advisor, and researcher; he attended Princeton and Harvard as a student, and went on to become the Dean of Yale Medical School and New York University School of Medicine, and President of Memorial Sloan-Kettering Institute. As evident, he had much preparation and many positions.

His writing has no affectation of any of them.

I am glad I was drawn to the book by its dimensions. It is of a size which creates space for itself in any luggage, however full. Thus it was very easy reading while travelling or waiting to travel. While reading, I discovered the books many dimensions.

Thomas qualifies the title “The Lives of a Cell” by the subtitle “Notes of a Biology Watcher”. That a person of his eminence calls himself a mere “watcher” indicates some humility. A glance at the headings of the 29 essays also points to considerable versatility. Thomas has written specifically about biology, society, language, music, information, computers, the nature of scientific research and the planning of science et al.; and generally about life and death -- the mortality of Man as well as the annihilation of man. Even after more than three and half decades since the book came out, many of Thomas’ ideas are fresh, astonishingly few are dated, and some are eerily prescient. For example, some of his projections seem to prognosticate an increasingly wired world, the burgeoning online social media of the current decade, even Second Life avatars (“software selves”, he calls them.)

Thomas probably did not want to weave a common thread through the collection, but connection seems to be a unifying theme of the book. Thomas talks about the wonder of how cells, organisms, species, ideas, are connected across the terrestrial and celestial worlds and he takes the wonder to a new level. He talks with a scientist’s precision, and an artist’s poetry.

The Lives of a Cell is a masterclass for appreciating the complexity of our living world, while awaking to its romance. 

Ontogeny recapitulates phylogeny

Well, that’s quite a mouthful, I agree – ontogeny recapitulates phylogeny. What it reflects is rather interesting. The phases in the development of an individual organism mimic to an extent its ancestral form. For example, the human embryo develops gills bars which are then modified for other purposes. But, “to an extent”. Biologist point out that the so called recapitulation occurs at a very coarse level, bereft of details [The Sciences of the Artificial, Third Edition by Herbert A. Simon, MIT Press, 1996].

But this serves as an exciting analogy for something similar that happens while we are learning a subject. The student’s progression seems to resemble the progression of the discipline from childhood to maturity.

Take for example, software engineering. First there was waterfall, then the iterative and incremental methods, and then agile. The streamlined regularity of waterfall is most useful for the tyro software engineer; there are copious milestones and checkpoints so that (s)he has enough guidance to get along. The essence of iterative and incremental development is appreciated with experience, when the power of feedback becomes manifest. Finally, agile in all its facility marks the transition from instruction to interaction. The software engineer now has sense enough not to need micro-management, (s)he can leverage the typical loose-coupling of an agile team. (Indeed, a criticism of agile has been its over-dependence on “premium people” to be successful). So the progression of a software engineer from a neophyte to a seasoned pro is marked by a progressing embrace of the more mature development methodologies.

What is the implication of this trend on how we teach software engineering to students? Do we introduce waterfall, then iterative and incremental development, then agile, in sequence? Or should we indoctrinate students in the most mature methodology of the day and touch upon others as historical footnotes? In conventional sciences the dogma of the day is “taught”, while its ancestors may (or may not) be mentioned – for example ether in physics, or phlogiston in chemistry. But unlike ether and phlogiston, waterfall is not merely of historical import, systems of relevant scope are successfully built using waterfall, very much in the present.

So must software engineers learn and unlearn (or at least learn to supersede) a methodology before moving on to the next? Or is there a more organic advancement?

Blueprint versus Recipe

Simon presents two different but complementary perspectives of complex systems: state descriptions versus process descriptions, or simply, blueprint versus recipe. For a simple system, the congruence of these two views is almost intuitive. A circle is locus of points all of which are at an equal distance from another point (the center). To draw circle, anchor one arm of a compass at a fixed point (the center), and rotate the other arm until it returns to the starting point. The first statement is a blueprint for a circle, the second, a recipe for drawing a circle. A blueprint characterizes the world as sensed, while a recipe reflects upon the way it as acted upon. [The Sciences of the Artificial, Third Edition by Herbert A. Simon, MIT Press, 1996].

For software systems, endowed as they are with Brooksean “invisibility” and “unvisualizability”, neither of the blueprint or recipe views are easy to arrive at. Describing the structure of code may not necessarily be a complete or even accurate description of the software subsuming the code. And although directions for constructing a unit of code can be specified with precision, there is no guarantee the desired behavior will be derived from that code.

Then how should we go about learning how to build software systems? Do we focus on absorbing knowledge about the blueprint or acquiring skills around the process? This confusion is often reflected in software engineering textbooks. Some authors take the descriptive route, trying to narrate software's characteristics – in all its quirks – using natural language. Others assume away all the messiness and indulge in the purity of mathematical models and their transformations. The former allows for easier reflection of the blueprint, the latter for the recipe. Both approaches are useful, but not one in the exclusion of the other.

That elusive balance is the key.

Use and renewal

In one of Tagore's dance dramas, the diva asks her cronies to renew her with new jewelry. How much renewal she had by such means, and what she did with such renewal, is what the dance and the drama is all about, and I will not divulge it here. But those of us, like the diva, who are obliged to recognize the decay of flesh, crave at one point or the other for renewal. Trees have at least the ritual of renewal. Once a year old leaves shed, and new ones shine. We alas, have no such facility -- the ritual of renewal, or renewal itself.

Decay comes with age, with use, with wear and tear, and tears. These come with living. In every mechanical system -- or systems with at least some mechanical parts, like the human body -- parts will wear, mostly the moving ones. But in software, there are no moving parts. So what wears a software system down, what wearies it?

This question has been asked and answered many times. Most fascinatingly, each time the question has been a little different, and the answer commensurately so. Thus it is no more one question with one answer. The decay of software comes with use, as new and newer functionality is stacked on an unsuspecting initial design. It is hoped that the underlying plasticity of software will be equal to anything. But what is unique about the use of software that warrants such wanton expectations? What is that which sets apart the use of software from the use of say a car, or bridge? We take a car or a bridge as given, love it or hate it, or become indifferent to it, and move on to another car, or another bridge (if we are in a car). But with software we are ever so tantalizingly close to what is, and what we want it to be. For software, the die is never cast. Frequently software never dies, one system merely reincarnates to another. Like our lives, there is always change. Change like the diva's new jewels, has the power to renew. Or thus the diva budgeted.

Ruminations on renewal apart, a key challenge of software engineering is to know what to do about changing requirements, or at least know what it does to a software system, take it as given, and live life from there.

Software metrics? Oh! You mean lines of code.

With due deference, I beg to differ. Lines of code (LOC) was once the software metric, is now merely a software metric. One among many other software metrics that are being continually conceived, created, and enhanced as software engineering grapples with deepening complexity. There was a time when the biggest challenge with writing code was to make it run, just as the earliest buildings humans built were good if they stood. You could count bricks and measure up such buildings. But then we learnt to build structures like the Taj Mahal. Just as it would be facile to try and measure the Taj by the weight of its marble, software systems of today are intricate interactions of components that are way beyond the ken of code count. Not only are the components larger than their lines of code, the interaction is larger than the component count. Simple intuitive metrics – unrelated to LOC – can give us insights into how design happens, and how responsibilities can be delegated (equitably, one hopes!) to components. I look forward to lively discussions on this and other topics at a forthcoming conference.

Enterprise software is about people, more often than not, enterprising people

I recently traveled by trains across the great Indian plains, from Pune to Kolkata (erstwhile Calcutta), a journey of nearly 1400 miles. I had procured the train ticket sitting in my bedroom in Tallahassee FL, printed it out, and on producing it to the ticket examiner on board the train along with a picture ID, all was hunky-dory.

This was not always like this.

I mean, if you produced the right ticket at the right time things are bound to be hunky-dory; but earlier the procuring part had more challenges than sitting in a bedroom and printing things out. All through my childhood and early youth, we had regular family vacations traveling by train to distant parts of a very diverse India. The excitement of the vacation hit us early; two months prior to the travel date we had to queue up at the railway reservation counters around four in the morning. The counters opened at ten. Arriving at quarter past four could have cost us the vacation.

In the late 1980s and early 1990s, my brother was involved in the project that built the online passenger reservation system for the Indian Railways. Indian Railways mimics India in its size and diversity, carrying more than a million ton of freight traffic and about 14 million passengers daily across 6,856 stations (http://www.indianrail.gov.in/). No computerized system for reservation existed before, all bookings were done across the counter, manually. This was a greenfield projects in a very true sense of the term – projects do not get that green any more. This was also an enterprise software project of the most gigantic proportions: involving not just one enterprise (Indian Railways is quite an enterprise in its own right, a separate government department presenting its own annual budget to Parliament), but touching the lives of millions of people in a direct way. The software framework built by this project was later enhanced with a Web interface, which allowed me the facility of getting Pune-Kolkata train tickets from my bedroom in Tallahassee.

But the online passenger reservation system did something more than providing an online passenger reservation system. (Interestingly, when the system was commissioned, online did not necessarily mean Web based -- it merely referred to something that was available in a way time and place were of no constraint to the user.) It brought in a radical change in legacy thinking -- the Indian Railways was no longer a bumbling behemoth of public transport system, it was seen more as a modern and flexible mass transit infrastructure. This is the kind of change great enterprise software systems usher in. They not merely change the way of doing, they also change the way of thinking.

To change ways of doing and thinking, we need to understand what needs to be changed, and be able to measure the change when it is taking place. Metrics play a vital role here; people who build successful enterprise software know this for sure. The book Metrics-Driven Enterprise Software Development: Effectively Meeting Evolving Business Needs shows us how we can build and play with this kind of metrics, and get something useful out of them on the way.

Enterprise software is about people, more often than not, enterprising people.