Change and Complexity: The Software Engineering Response

The central challenges for software engineering are complexity and change. Software engineering’s response to these challenges comes in two parts: breaking down the problem of building a software system into smaller, more manageable ‘chunks’ to confront complexity; and setting regular checkpoints during the process of building a software system to address the effects of change.

The breaking down results in workflows and the checkpointing leads to phases; together they constitute the software development life cycle (SDLC). The SDLC lies at the heart of software engineering.

Workflows represent sets of activities starting from understanding what users want from a software system (Requirements), to translating the language of the problem into the language of the solution (Analysis), to expressing the solution constructs in the language of development (Design), to building the system using programming resources (Implementation), and finally, verifying whether the system matches the stated requirements (Testing).

Phases, on the other hand, are focused towards monitoring and managing change. During Inception we ask, what do the users want from the system? During Elaboration, we are interested in knowing if the system is feasible. Next comes the question: How do we build the system? This is the concern of Construction. Finally, during Transition, we enquire, how do we transfer the system from the developer domain to the user domain? In a particular development life cycle, we may not know the answers to these questions when we ask them. But based on our experience and understanding, we have an expectation of what the answers are likely to be. When expectations are not met, it serves as a reality check: A change, not budgeted for, must have occurred. This makes us aware of the need to find out what changed and what all that change might affect.

On the face of it, software engineering’s response to the problems of change and complexity seems adequate. But it admits certain challenges.

An element of linearity is implicit in the way software engineering seeks to address change and complexity. Customers come with requirements, which are analysed, followed by the design of the system, its implementation and testing. If the right questions are asked at each point, Inception, Elaboration, Construction, and Transition seemingly follow one another in harmony.

But reality is much messier. Answers are seldom ready when questions are asked, customers and users change their minds all the time, and technology and business environments change. In the real-world of software development, it becomes imperative to go back and forth across workflows and phases several times, driven by a variety of reasons. Life is inherently non-linear, and software engineering is no exception. But just as in life, in software engineering too, we build our case on assumptions of linearity. And then hope to tackle non-linearity on a case-to-case basis.

The key challenge with software engineering’s response to change and complexity boils down to being able to monitor, control, and leverage the myriad feedback paths that exist in the real-world software development life cycle.

Feedback is one of the most fundamental techniques of engineering. In simple terms, feedback involves controlling an activity by regulating the input based on the output. Feedback exists at many levels, practical as well as perceptual. For example, an exception handler is a simple feedback loop. It monitors the execution of a piece of code and takes appropriate action if the outcome is not as expected. On the other hand, modifying a system’s design based on user response is also an instance of feedback. In software engineering, the difficulty usually lies in integrating feedback of different forms and at various levels into a consistent and repeatable development model.

This is the central challenge with software engineering’s responses to the problems of change and complexity.