For as young a field as software engineering, history is not just something to do with the past. It is also very much what is unfolding in the present. Computing is becoming faster, cheaper, and more ubiquitous. This speed often leaves us cold and confused: How do I know whether what I learn today will have any value tomorrow? Addressing this question lies at the crux of learning a little bit of software engineering’s history. A basic historical awareness can make us ready for change and what it means for individual careers, as well as the collective profession.
Some of the trends that have influenced software engineering and continue to do so can be summarized as:
Programming to software engineering – Programming plays a significant role in software development. This role has been redefined continuously with the evolution of software engineering. From a pre-eminently central position, programming has moved to be one of the concerns for the software developer.
Hardware-software: from coupling to congress – In the early days of computing, software was hardly recognized as something different from hardware. Then, much of the instruction needed to run a computer was hard-wired into the hardware. Additional instructions that came via software were specific to the hardware. When hardware was sold, software came with it; almost no one sold or bought software by itself. Compare that with the present; probably the operating system is the only piece of software now that comes with the hardware we buy. Almost every other application software for general use —those that let us do all those fun and useful things—is acquired and installed separately, often downloaded from the Web for free. The erstwhile coupling between hardware and software can be said to have been replaced by something of a congress.
Advent of High-Level Languages – What has changed profoundly over the past few decades is how we communicate with computers. What has brought about this change is the vehicle of communication: language. Very simply, the so-called high-level languages are programming languages whose syntax and idiom are closer to that of English. High-level programming languages let one be a programmer without being able to communicate in machine or assembly language. This caused an explosion in the number of programmers, which widened the scope of the software engineering profession.
Advent of the Personal Computer – It is estimated that more than one billion personal computers (PC) are in use in the world now. Certainly a far cry from that mere handful they were expected to sell in the early 1980s. Computers have come from being isolated, redoubtable behemoths, to portable, friendly devices sitting on our desks or being carried around in hand bags. Concomitantly, there has been exponential growth in the amount, variety, scope, and power of the software that needs to support what we do with those ‘personal’ computing machines today. With the coming of the PC and its many subsequent variants, software engineers have become ever busier.
Global Software Development – As the Web’s presence increased throughout the 1990s and then into the new millennium, software engineering became a truly global enterprise. It is customary now for the customers, users, developers and managers of a software project to be dispersed across continents, time zones, and cultures. This makes software development something of a unique enterprise. In today’s typical software projects, development proceeds round-the-clock, through the collaboration of offshore and onsite team members. Global software development has deep economic as well as social implications within and without the software engineering profession.
Return of Open Source – Very simply, open source is about sharing the source code (as well as executables) of software developed by a particular group or individual for free, so that others can use or modify the code without paying any royalties. Open source was nothing special in the initial era of computing. Companies sold hardware, and the software came with it, for free. Academics shared code just as they shared ideas. When software became a commercial commodity in its own right, large corporations made every effort to stifle the free flow of code. The coming of the Web in the early 1990s gave open source aficionados the fast, reliable, and cheap media they needed to connect and build exciting and useful software. And then give it away for free. This is how open source returned, after being in exile for many years. Other than being an affirmation of intellectual freedom, the open source paradigm led to the re-examination of many fundamental principles of software development. Corporations large and small have come to recognize the power of open source and are keen to leverage it. Open source has fundamentally changed software development, and the change is here to stay.
The interplay of these trends – many of which overlap and influence one another – have shaped software engineering’s evolving journey. These are living trends, and they beget newer trends.
Saturday, February 5, 2011
Saturday, January 22, 2011
Change and Complexity: The Software Engineering Response
The central challenges for software engineering are complexity and change. Software engineering’s response to these challenges comes in two parts: breaking down the problem of building a software system into smaller, more manageable ‘chunks’ to confront complexity; and setting regular checkpoints during the process of building a software system to address the effects of change.
The breaking down results in workflows and the checkpointing leads to phases; together they constitute the software development life cycle (SDLC). The SDLC lies at the heart of software engineering.
Workflows represent sets of activities starting from understanding what users want from a software system (Requirements), to translating the language of the problem into the language of the solution (Analysis), to expressing the solution constructs in the language of development (Design), to building the system using programming resources (Implementation), and finally, verifying whether the system matches the stated requirements (Testing).
Phases, on the other hand, are focused towards monitoring and managing change. During Inception we ask, what do the users want from the system? During Elaboration, we are interested in knowing if the system is feasible. Next comes the question: How do we build the system? This is the concern of Construction. Finally, during Transition, we enquire, how do we transfer the system from the developer domain to the user domain? In a particular development life cycle, we may not know the answers to these questions when we ask them. But based on our experience and understanding, we have an expectation of what the answers are likely to be. When expectations are not met, it serves as a reality check: A change, not budgeted for, must have occurred. This makes us aware of the need to find out what changed and what all that change might affect.
On the face of it, software engineering’s response to the problems of change and complexity seems adequate. But it admits certain challenges.
An element of linearity is implicit in the way software engineering seeks to address change and complexity. Customers come with requirements, which are analysed, followed by the design of the system, its implementation and testing. If the right questions are asked at each point, Inception, Elaboration, Construction, and Transition seemingly follow one another in harmony.
But reality is much messier. Answers are seldom ready when questions are asked, customers and users change their minds all the time, and technology and business environments change. In the real-world of software development, it becomes imperative to go back and forth across workflows and phases several times, driven by a variety of reasons. Life is inherently non-linear, and software engineering is no exception. But just as in life, in software engineering too, we build our case on assumptions of linearity. And then hope to tackle non-linearity on a case-to-case basis.
The key challenge with software engineering’s response to change and complexity boils down to being able to monitor, control, and leverage the myriad feedback paths that exist in the real-world software development life cycle.
Feedback is one of the most fundamental techniques of engineering. In simple terms, feedback involves controlling an activity by regulating the input based on the output. Feedback exists at many levels, practical as well as perceptual. For example, an exception handler is a simple feedback loop. It monitors the execution of a piece of code and takes appropriate action if the outcome is not as expected. On the other hand, modifying a system’s design based on user response is also an instance of feedback. In software engineering, the difficulty usually lies in integrating feedback of different forms and at various levels into a consistent and repeatable development model.
This is the central challenge with software engineering’s responses to the problems of change and complexity.
The breaking down results in workflows and the checkpointing leads to phases; together they constitute the software development life cycle (SDLC). The SDLC lies at the heart of software engineering.
Workflows represent sets of activities starting from understanding what users want from a software system (Requirements), to translating the language of the problem into the language of the solution (Analysis), to expressing the solution constructs in the language of development (Design), to building the system using programming resources (Implementation), and finally, verifying whether the system matches the stated requirements (Testing).
Phases, on the other hand, are focused towards monitoring and managing change. During Inception we ask, what do the users want from the system? During Elaboration, we are interested in knowing if the system is feasible. Next comes the question: How do we build the system? This is the concern of Construction. Finally, during Transition, we enquire, how do we transfer the system from the developer domain to the user domain? In a particular development life cycle, we may not know the answers to these questions when we ask them. But based on our experience and understanding, we have an expectation of what the answers are likely to be. When expectations are not met, it serves as a reality check: A change, not budgeted for, must have occurred. This makes us aware of the need to find out what changed and what all that change might affect.
On the face of it, software engineering’s response to the problems of change and complexity seems adequate. But it admits certain challenges.
An element of linearity is implicit in the way software engineering seeks to address change and complexity. Customers come with requirements, which are analysed, followed by the design of the system, its implementation and testing. If the right questions are asked at each point, Inception, Elaboration, Construction, and Transition seemingly follow one another in harmony.
But reality is much messier. Answers are seldom ready when questions are asked, customers and users change their minds all the time, and technology and business environments change. In the real-world of software development, it becomes imperative to go back and forth across workflows and phases several times, driven by a variety of reasons. Life is inherently non-linear, and software engineering is no exception. But just as in life, in software engineering too, we build our case on assumptions of linearity. And then hope to tackle non-linearity on a case-to-case basis.
The key challenge with software engineering’s response to change and complexity boils down to being able to monitor, control, and leverage the myriad feedback paths that exist in the real-world software development life cycle.
Feedback is one of the most fundamental techniques of engineering. In simple terms, feedback involves controlling an activity by regulating the input based on the output. Feedback exists at many levels, practical as well as perceptual. For example, an exception handler is a simple feedback loop. It monitors the execution of a piece of code and takes appropriate action if the outcome is not as expected. On the other hand, modifying a system’s design based on user response is also an instance of feedback. In software engineering, the difficulty usually lies in integrating feedback of different forms and at various levels into a consistent and repeatable development model.
This is the central challenge with software engineering’s responses to the problems of change and complexity.
Sunday, January 16, 2011
Software Engineering’s Challenges: Change and Complexity
Every engineering discipline starts off in response to some pressing problem. Software engineering confronts the problems of change and complexity.
The very nature of software—its plasticity—makes it amenable to a continuous cycle of change. It seems rather easy to accomplish. After all, tweaking one statement in a software program can radically alter the program’s behaviour. But such tweaking—little by itself, but considerable in conjunction—can end up
changing the intent of the program’s design in fundamental ways.
It is absurd to expect a car to fly or float. But very often a software system built for one context is expected to function in as drastically different contexts, with the same grace and efficiency. These expectations can be traced to our wide cognitive gap with the use of software. Decades and centuries of using cars and bridges respectively, and millennia of using houses has ingrained in our minds what cars, houses, and bridges can and cannot do. Accordingly, we tune our expectations as well as environmental factors to set the context for these systems to function. In comparison, the use of software amongst a large community of lay users has just begun. Our understanding of how and to what extent software can serve our needs is yet not complete. As a result, the problem of change for software comes primarily from changing user expectations, and also from changes in the environment—technological and social.
Complexity is a complex word and there is no one definition to cover its ken; even reaching a definition is fraught with difficulties. But we need to care about it in life as well as in software engineering as complexity arises out of simplicity, at times suddenly and surreptitiously. Think of a simple computer program of five lines of code. It is straightforward; by carefully reviewing each line, we can hope to have complete knowledge of the program’s structure and behaviour. Now what if, a loop is introduced in the program—a simple construct that executes a set of statements repetitively, until a condition holds. The number of execution paths through the program has significantly increased now, and it has become far more difficult to know for sure what happens in each step when the program runs. This example is just a watered down instance of the combinatorial complexity software systems customarily face. Then there are even more involved issues such as complexity of the problem domain, complexity in the interaction of the various forces—technological, commercial, political—that a software system has to balance to be successful.
A common feature of complex systems is that they are greater than the sum of their parts. Anyone who has worked on a course project to build a piece of software stretching across several files can appreciate the sense of this statement: A piece of software is made of individual files, but it delivers something that merely bunching the files together will not achieve. Now scale-up to a real world system—with hundreds, if not thousands of files; and thousands, if not millions of interfaces between them; perhaps simple by themselves, but certainly complex when functioning together. And this is just one, relatively less significant, facet of software complexity.
A central theme of software engineering is about framing an adequate response to the problems of change and complexity.
The very nature of software—its plasticity—makes it amenable to a continuous cycle of change. It seems rather easy to accomplish. After all, tweaking one statement in a software program can radically alter the program’s behaviour. But such tweaking—little by itself, but considerable in conjunction—can end up
changing the intent of the program’s design in fundamental ways.
It is absurd to expect a car to fly or float. But very often a software system built for one context is expected to function in as drastically different contexts, with the same grace and efficiency. These expectations can be traced to our wide cognitive gap with the use of software. Decades and centuries of using cars and bridges respectively, and millennia of using houses has ingrained in our minds what cars, houses, and bridges can and cannot do. Accordingly, we tune our expectations as well as environmental factors to set the context for these systems to function. In comparison, the use of software amongst a large community of lay users has just begun. Our understanding of how and to what extent software can serve our needs is yet not complete. As a result, the problem of change for software comes primarily from changing user expectations, and also from changes in the environment—technological and social.
Complexity is a complex word and there is no one definition to cover its ken; even reaching a definition is fraught with difficulties. But we need to care about it in life as well as in software engineering as complexity arises out of simplicity, at times suddenly and surreptitiously. Think of a simple computer program of five lines of code. It is straightforward; by carefully reviewing each line, we can hope to have complete knowledge of the program’s structure and behaviour. Now what if, a loop is introduced in the program—a simple construct that executes a set of statements repetitively, until a condition holds. The number of execution paths through the program has significantly increased now, and it has become far more difficult to know for sure what happens in each step when the program runs. This example is just a watered down instance of the combinatorial complexity software systems customarily face. Then there are even more involved issues such as complexity of the problem domain, complexity in the interaction of the various forces—technological, commercial, political—that a software system has to balance to be successful.
A common feature of complex systems is that they are greater than the sum of their parts. Anyone who has worked on a course project to build a piece of software stretching across several files can appreciate the sense of this statement: A piece of software is made of individual files, but it delivers something that merely bunching the files together will not achieve. Now scale-up to a real world system—with hundreds, if not thousands of files; and thousands, if not millions of interfaces between them; perhaps simple by themselves, but certainly complex when functioning together. And this is just one, relatively less significant, facet of software complexity.
A central theme of software engineering is about framing an adequate response to the problems of change and complexity.
Saturday, January 15, 2011
Software Engineering: The First Forty Years
The phrase ‘software engineering’ was first used in a public discourse at a NATO Science Committee sponsored conference, held at Garmisch, Germany, from 7th to 11th October, 1968. More than forty years hence, there is still not much unanimity on a definition of software engineering.
According to Boehm, software engineering involves the application of science and mathematics through which the facilities of computer equipment are made useful to human beings via computer programs, procedures, and associated documentation. Pfleeger identifies software engineering with the utilization of tools, techniques, procedures, and paradigms toward quality improvement of the software product. Naur and Randall see software engineering in terms of establishing and using sound engineering principles to obtain economically effective and reliable software that can work efficiently on real machines. According to Freeman and Von Staa, software engineering involves the organized application of methods, tools, and knowledge towards fulfilling stated technical, economic, and human goals for a software-intensive system. Kacmar says simply applying engineering principles to designing and constructing computer software can be termed software engineering. To Schach, software engineering is the discipline that aims at producing fault-free software, to be delivered on time and within budget, which satisfies the user’s needs. Whitmire describes software engineering as a ‘slippery’ term, and says for some it is something that can only be applied to a large project, while to others it is just a ‘figment of collective imaginations’. He gives a working definition as, ‘Software engineering is the science and art of designing and building, with economy and elegance, software systems and applications so they can fill the uses to which they may be subjected’
Many views and much variety!
However, there seems to be little room for debate on Brooks’ characterization of software - software is inherently complex; software must be made to conform to existing interfaces; software is constantly subject to change; and software is invisible and unvisualizable.
Software engineering can be viewed as an organized response to the challenges these very characteristics of software engenders. Software engineering is concerned with the problem of change and the problem of complexity.
Between change and complexity, does one necessarily insinuate the other?
What do you think?
According to Boehm, software engineering involves the application of science and mathematics through which the facilities of computer equipment are made useful to human beings via computer programs, procedures, and associated documentation. Pfleeger identifies software engineering with the utilization of tools, techniques, procedures, and paradigms toward quality improvement of the software product. Naur and Randall see software engineering in terms of establishing and using sound engineering principles to obtain economically effective and reliable software that can work efficiently on real machines. According to Freeman and Von Staa, software engineering involves the organized application of methods, tools, and knowledge towards fulfilling stated technical, economic, and human goals for a software-intensive system. Kacmar says simply applying engineering principles to designing and constructing computer software can be termed software engineering. To Schach, software engineering is the discipline that aims at producing fault-free software, to be delivered on time and within budget, which satisfies the user’s needs. Whitmire describes software engineering as a ‘slippery’ term, and says for some it is something that can only be applied to a large project, while to others it is just a ‘figment of collective imaginations’. He gives a working definition as, ‘Software engineering is the science and art of designing and building, with economy and elegance, software systems and applications so they can fill the uses to which they may be subjected’
Many views and much variety!
However, there seems to be little room for debate on Brooks’ characterization of software - software is inherently complex; software must be made to conform to existing interfaces; software is constantly subject to change; and software is invisible and unvisualizable.
Software engineering can be viewed as an organized response to the challenges these very characteristics of software engenders. Software engineering is concerned with the problem of change and the problem of complexity.
Between change and complexity, does one necessarily insinuate the other?
What do you think?
Saturday, November 20, 2010
The Lives of a Cell
The Lives of a Cell by Lewis Thomas is one of those rare books; small, sweet, and sublime. When I picked it up at a second hand book store, I was impressed by its size, or the lack of it: 6.9 x 4.2 x 0.7 inches. I was not intimidated by the author. I was not acquainted with Lewis Thomas’ name at that time, not even at the Wikipediac level. I was later to learn – from Wikipedia – that Thomas was a physician, poet, etymologist, essayist, administrator, educator, policy advisor, and researcher; he attended Princeton and Harvard as a student, and went on to become the Dean of Yale Medical School and New York University School of Medicine, and President of Memorial Sloan-Kettering Institute. As evident, he had much preparation and many positions.
His writing has no affectation of any of them.
I am glad I was drawn to the book by its dimensions. It is of a size which creates space for itself in any luggage, however full. Thus it was very easy reading while travelling or waiting to travel. While reading, I discovered the books many dimensions.
Thomas qualifies the title “The Lives of a Cell” by the subtitle “Notes of a Biology Watcher”. That a person of his eminence calls himself a mere “watcher” indicates some humility. A glance at the headings of the 29 essays also points to considerable versatility. Thomas has written specifically about biology, society, language, music, information, computers, the nature of scientific research and the planning of science et al.; and generally about life and death -- the mortality of Man as well as the annihilation of man. Even after more than three and half decades since the book came out, many of Thomas’ ideas are fresh, astonishingly few are dated, and some are eerily prescient. For example, some of his projections seem to prognosticate an increasingly wired world, the burgeoning online social media of the current decade, even Second Life avatars (“software selves”, he calls them.)
Thomas probably did not want to weave a common thread through the collection, but connection seems to be a unifying theme of the book. Thomas talks about the wonder of how cells, organisms, species, ideas, are connected across the terrestrial and celestial worlds and he takes the wonder to a new level. He talks with a scientist’s precision, and an artist’s poetry.
The Lives of a Cell is a masterclass for appreciating the complexity of our living world, while awaking to its romance.
Sunday, October 10, 2010
Ontogeny recapitulates phylogeny
Well, that’s quite a mouthful, I agree – ontogeny recapitulates phylogeny. What it reflects is rather interesting. The phases in the development of an individual organism mimic to an extent its ancestral form. For example, the human embryo develops gills bars which are then modified for other purposes. But, “to an extent”. Biologist point out that the so called recapitulation occurs at a very coarse level, bereft of details [The Sciences of the Artificial, Third Edition by Herbert A. Simon, MIT Press, 1996].
But this serves as an exciting analogy for something similar that happens while we are learning a subject. The student’s progression seems to resemble the progression of the discipline from childhood to maturity.
Take for example, software engineering. First there was waterfall, then the iterative and incremental methods, and then agile. The streamlined regularity of waterfall is most useful for the tyro software engineer; there are copious milestones and checkpoints so that (s)he has enough guidance to get along. The essence of iterative and incremental development is appreciated with experience, when the power of feedback becomes manifest. Finally, agile in all its facility marks the transition from instruction to interaction. The software engineer now has sense enough not to need micro-management, (s)he can leverage the typical loose-coupling of an agile team. (Indeed, a criticism of agile has been its over-dependence on “premium people” to be successful). So the progression of a software engineer from a neophyte to a seasoned pro is marked by a progressing embrace of the more mature development methodologies.
What is the implication of this trend on how we teach software engineering to students? Do we introduce waterfall, then iterative and incremental development, then agile, in sequence? Or should we indoctrinate students in the most mature methodology of the day and touch upon others as historical footnotes? In conventional sciences the dogma of the day is “taught”, while its ancestors may (or may not) be mentioned – for example ether in physics, or phlogiston in chemistry. But unlike ether and phlogiston, waterfall is not merely of historical import, systems of relevant scope are successfully built using waterfall, very much in the present.
So must software engineers learn and unlearn (or at least learn to supersede) a methodology before moving on to the next? Or is there a more organic advancement?
Blueprint versus Recipe
Simon presents two different but complementary perspectives of complex systems: state descriptions versus process descriptions, or simply, blueprint versus recipe. For a simple system, the congruence of these two views is almost intuitive. A circle is locus of points all of which are at an equal distance from another point (the center). To draw circle, anchor one arm of a compass at a fixed point (the center), and rotate the other arm until it returns to the starting point. The first statement is a blueprint for a circle, the second, a recipe for drawing a circle. A blueprint characterizes the world as sensed, while a recipe reflects upon the way it as acted upon. [The Sciences of the Artificial, Third Edition by Herbert A. Simon, MIT Press, 1996].
For software systems, endowed as they are with Brooksean “invisibility” and “unvisualizability”, neither of the blueprint or recipe views are easy to arrive at. Describing the structure of code may not necessarily be a complete or even accurate description of the software subsuming the code. And although directions for constructing a unit of code can be specified with precision, there is no guarantee the desired behavior will be derived from that code.
Then how should we go about learning how to build software systems? Do we focus on absorbing knowledge about the blueprint or acquiring skills around the process? This confusion is often reflected in software engineering textbooks. Some authors take the descriptive route, trying to narrate software's characteristics – in all its quirks – using natural language. Others assume away all the messiness and indulge in the purity of mathematical models and their transformations. The former allows for easier reflection of the blueprint, the latter for the recipe. Both approaches are useful, but not one in the exclusion of the other.
That elusive balance is the key.
Subscribe to:
Posts (Atom)