There is a discussion currently in progress among the OpenUP/Basic team about how best to emphasize the fact that we definitely do not advocate BDUF. This seems to be something that the Unified Process (particularly the Rational Unified Process) has been been labeled with over the years - mistakenly, IMHO.
In the OpenUP team, we want to leave no room for doubt...OpenUP is not a BDUF process!
One strand of this discussion concerns how to incorporate some ideas from other agile methods into the Unified Process. Specifically in this case Refactoring and Emergent Design. The discussion was getting pretty messy and intense, which prompted one of the team to say "we need a clean answer on the difference between emergent design and refactoring." That doesn't mean that we didn't know the difference; we just needed to be clear as a team about what we meant by these different terms.
One comment on emergent design suggested that it "..occurs when the working software is refactored from fine-grained and coarse-grained perspectives.”
That prompted me to write a very long email, which I decided to turn into a blog post.
As many of who will doubtless already know, the term Refactoring is pretty much well defined. Heaps of books have been written on the subject. I recall that the sub-title of Fowler's definitive book is "Improving the design of existing code." I also recall that Fowler has written extensively on the subject of Design Patterns, Software Architecture and UML, all of which indicate a recognition of the value of thinking about the solution in the abstract before, during and after coding it.
Fowler gives us a neat, pithy definition for Refactoring on www.refactoring.com
"Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior."
So I'm going to be bold and suggest that Refactoring and Design (including emergent design) are related but distinct activities :-)
What about a pithy definition of emergent design?There are some interesting writings on the subject. My personal favorite coming from David Cavallo at IBM who first coined the phrase "emergent design" in the context of "applied epistemological anthropology" (see http://www.research.ibm.com/journal/sj/393/part2/cavallo.html).
Moving on, another article by Matt Stephens in 2003 discusses emergent design (and emergent architecture) in the context of prototyping. It also discusses XP's emergent design concepts. Stephens explores "early prototyping" as an approach to emergent architecture and design - i.e. a separate code-base, where ideas explored in the sand-pit can be moved into the real-thing as they are proven. This is different to the XP approach, where the prototype evolves into the production code. He concludes that a combination of the two techniques may be the best approach – early prototyping to define the architecture; followed by evolutionary prototyping (XP's "emergent design")
This struck a chord with me, as this is the approach to architecture and design applied in DSDM. DSDM approaches software architecture, design and development as a closely related chunk of work. DSDM emphasizes evolutionary prototyping as a technique for emergent architecture, design, development and requirements elicitation and refinement. There is a form of "up front architecture" in DSDM - but it is *system* architecture, which addresses a broader set of concerns that the UP software architecture domain. In DSDM, creating the software architecture centres around developing a feasibility prototype and documenting the overall technical approach in the Solution Architecture Definition.
The prototype in DSDM is refined through a series of time-boxes into the releasable, production software (the "tested system"). More features are added over time, which means incrementally adding more design and implementation to meet the agreed requirements for each time-box. We can also agree objectives around improving aspects of the architecture and design during a time-box. And of course, micro-level techniques such as Refactoring can also be applied. The DSDM "prototype" practice is manifested at different stages, with different objectives:
- Feasibility prototype - proves the architecture early. This is often discarded.
- Functional Prototype - (similar to UI prototype) enables the users to comprehend and feedback on the features the system will exhibit. It evolves into...
- Design prototype - (the functional prototype with the back-end code developed) essentially the pretested system. This evolves into...
- Tested System - ready to be delivered to users. This becomes...
- Delivered System - the system in operational use.
(As an aside, you should note that this is the position in DSDM 4.2 - a major new release for DSDM is planned for April 2007 - see www.dsdm.org for details - which will build on and modify these terms. The point is still valid for this discussion though).
However, DSDM is not explicit about what specific techniques and practices should be followed during design and development in the way that the Unified Process has been. DSDM's focus is intentionally on the product, not the tasks required to achieve them.
In OpenUP/Basic, it seems to me that our approach to emergent architecture and design is closer to XP's. We focus on addressing risk early through architecture effort and proving it in code (from Inception through to Elaboration) with a build that evolves into a production ready product. We are not prescriptive about throwing away the Inception build (aka Executable Architecture - or Architectural Proof of Concept in RUP) but of course, real-world teams can make their own judgment calls. We then continue to develop the architecture through Elaboration (with documentation/models and code) until we are satisfied that the architecture is baselined.
Once we have a stable architecture, we can work in the Construction phase reasonably confident that, as we have addressed the majority of architecturally significant requirements, we should experience relatively little technical risk. We should be into a "flesh on the bones" situation.
So we should be happy that we are able to speak comfortably about the idea of emergent architecture within OpenUP/Basic. We are outlining and developing the architecture iteratively and proving it with code from start of the project.
So what about emergent design?
In this article from 2001, Ron Jeffries summarizes emergent design in the context of XP as being represented by the application of a set of principles, citing Alan Shalloway in saying that well-designed software:
- quoted text -
- Runs all the tests
- Follows [the] once and only once rule (which I believe means "contains no duplicate code")
- Has high cohesion (clarity)
- Has loose coupling
- end quote -
That's obviously good advice. I say obviously but of course, lots of people don't follow it, so I don't want to diminish it's value.
The only slight criticism I have with this as guidance is that it describes the characteristics of well-designed software but it doesn't offer advice on how to get there. It seems more of a test to apply after the fact. There's plenty of advice in the XP books, of course.
The question for us in OpenUP/Basic is (IMHO) *how* does an OpenUP/Basic practitioner produce software that passes the Shalloway Test? What work should he/she do? I am not sure that we stand to add a lot of value to the community by simply aping the XP advice. I'd like to see us translate that into the UP world by being more explicit about what work we need to do and when. This is based on the idea that the people who will read our content need to know this because they don't yet know how to do it - and don't want to follow XP (otherwise, they would be picking up the XP materials and working with that).
I reacquainted myself with Shalloway and Trott's book (albeit the first edition - the second edition is on order) and it looks to me like their approach is based around the sound application of analysis and design principles - factoring out commonality through analysis and abstraction; followed by the considered use of design patterns in the code. I don't know these guys so I reserve the right to have misunderstood their views :-).Shalloway & Trott offer plenty of good, practical "how-to" advice in their book. Larman also has some very good material too (and his book uses UP as a process model).
So that's some of the views of some of the gurus. What about what I think?
I'll take a moment to share some personal experience on the subject...
I have used an approach on projects that I have referred to as "design discovery" with in conjunction with RUP, although this has typically been in a larger team context than the one we have in mind for OpenUP/Basic. The nutshell of the approach runs like this.
We can apply Use Case Analysis to factor out common behavior and sketch out the design and implementation. We use Use Case Design techniques (old Objectory shorthand) to provide a concrete subsystem design - basically agreeing where in the subsystem/component model to place required behavior and agreeing the interfaces. This is done by spending some time (maybe an hour or so) around a whiteboard with the customer in the room and the results are captured with a digital camera. Once this is done, people can design/implement the scoped requirements in the agreed subsystems against the defined interfaces in the same iteration.
This gives us some confidence that the software will be cohesive and loosely coupled. Sure, stuff changes but we have a starting point to discuss and manage those changes. My experience is that RUP Subsystem Design and (especially) Class Design generally happens in the code. That's also where design patterns tend to get applied. Some subsystems may be more complex and require further decomposition through collaborative design but it's a judgment call.
How is this similar to design discovery / emergent design? Well, to begin with, I don't believe that it is Big Design Up Front. In each iteration we consider the implementation of the requirements that we agreed to deliver in that iteration. At the end of the iteration, we have working software. When we do use case analysis and design, we are explicitly looking to support the implementation of *only* the requirements described by the scenarios in scope. We apply the YAGNI principle and don't implement requirements that *might* be needed later - basically, we implement what we *know* we need (because it's in the scenario description) not what we think we *might* need (or even, what we *know* we will need) in the future.
Future needs are accommodated in the future and the design/code can be refactored to accommodate. This helps stop us from missing a milestone deadline because we implemented features that nobody wanted yet. That really helps to reduce procrastination.
Not only that, but a well-written use case scenario will describe the steps required to realize the behavior we need in a way that translates very quickly into operations on an interface. That's not something that I have found is so well supported by loosely-written user stories. There is very little procrastination in analysis realizations off a use-case scenario, especially if we have a reasonable set of key abstractions.
Early design can take a little longer as it is more exploratory (especially because there is not much design/implementation in the bank). After a few Elaboration iterations, we find that design level realizations can be produced very quickly - and we start to skip analysis realizations because the design work is becoming more intuitive.
Eventually, we reach a point when we can even dispense with almost all design-realizations, as that process itself is intuitive. We start to feel confident that the architecture is stabilizing and is well-understood by the team. We decide that we have reached the end of Elaboration (at a point in time that is, hopefully, broadly in line with expectations in the project plan - but not always).
The net result is what we might expect if we step away from the detail for a moment.
First, people collaborate.
Next, the use of modeling and "methodical" analysis and design techniques is more intensive during Inception and Elaboration. Gradually, as the architecture matures and the knowledge of the team grows, formal modeling techniques diminish. This is because tacit knowledge has reached a point where modeling adds little value - save for the occasional complex or architecturally significant scenario, where these techniques are pulled out of the kit-bag again.
On a small project, with a simple scope and well-understood solution approach we can reach this point pretty quickly. Maybe we only need Inception & Elaboration to last for a few days (or even hours). Maybe we only need to produce one analysis-realization to "get the picture". On larger, more complex projects, it may take longer - weeks or even months. But the principles are the same. And we get working software throughout.
I would like OpenUP/Basic to support the same ideas. The vocabulary will be different (we don't have Use Case Design as a Task, for example); and we may be less prescriptive about UML techniques. I think that the content we currently have (in terms of raw tasks and products in the Method library) support this idea. We just need to straighten out the capability pattern that defines how the tasks combine to form a chunk of process.
To join in the discussion on this and other EPF-related issues, why not subscribe to epf-dev@eclipse.org? You can find out how to register at www.eclipse.org/epf
