Senior GIS Software Architec
Data Transfer Solutions
[email protected]
Chris Spagnuolo
Agile Evangelist
Data Transfer Solutions
[email protected]
What is "Agile"?
Agile refers a set of project management and software engineering practices which focus on reducing project risk by incrementally developing software through a series of iterations.In order to understand many of the Agile concepts, it's worth quickly reviewing how software has historically been developed.
TRADITIONAL SOFTWARE DEVELOPMENT METHODOLOGIES
Since about 1970, traditional software development has been approached from a “waterfall” perspective. This set of methodologies, which were inspired by civil and mechanical engineering, approach software development as a series of steps which proceed linearly to completion – not unlike the building of a bridge. These steps are typically broken out into Requirements, Specifications, Design, Coding, Testing, and Delivery. In the ideal waterfall world, a team approaches a project and immediately gathers all the known requirements from the stakeholders at once. The team then retreats for months or years during development and the end of which they have produced working software that meets the initial requirements of the users. While this sounds good in theory, in practice it has been less than successful. This is illustrated by the following set of statistics about software projects:
In 1994:
- Only 16% of software projects are "Successful" – meaning on time, on budget, and met user requirements
- 52% of projects were "Challenged" – meaning they slipped on time, budget or requirements
- 31% were outright failures
By 2006 things improved:
- 35% of software projects are "Successful"
- 46% of projects were "Challenged"
- 19% are outright failures
Jim Johnson, President of the Standish Group, notes that Iterative Development has helped improve the more recent numbers.
From Software Development Times report on the 2006 Standish CHAOS report
There are several reasons why these waterfall methodologies tend to be challenged or fail outright. One of the primary reasons is that waterfall methodologies do not embrace changing requirements. The waterfall methodology relies far too heavily on up-front requirements gathering and doesn’t address the users changing needs as the development project evolves. Is assumes that the requirements gathered at the outset of the project will remain constant over the lifespan of the development process. In reality, 35% of software requirements change during a typical software development project. Additionally, nearly 65% of features developed in waterfall projects end up being used rarely or not being used at all. Given that up to 50% of a project budget and schedule is dedicated to requirements gathering in waterfall-type projects, this appears to represent a substantial waste of resources.
Agile Software Development Practices
In the early and mid-1990’s, several different factions within the software development industry began to consider the problems with traditional waterfall methodologies. They began examining the patterns and practices of successful software development teams to understand what made them different from their peers. What they found was that the successful teams were not following the traditional waterfall methodologies. They were employing iterative development practices that emphasized constant user-developer interaction, self-organizing and self-managing development teams, and that truly embraced the ever evolving nature of software requirements. These ideas these factions uncovered eventually congealed into what we know today as agile software development practices.
In contrast to the rigid "big design up-front" waterfall models, Agile practices focus on lean requirements and design up front followed by iterative cycles of developing and releasing software to the end users. The end user stakeholders are actively involved in the prioritization of the requirements to be developed in each iteration. Detailed requirements, design, coding and testing are all done in the iteration itself (two to four weeks in length), and the output of an iteration should be "shippable" functionality. These short iterations ensure that the team is always developing what the end user really needs. Should the team get off course, they will be brought back in line with expectations at the review meeting with the stakeholders at the end of the iteration. This is in stark contrast to waterfall methodologies where this divergence is typically found after the software is deployed. In addition, the end user stakeholders may change, add, or remove requirements at the conclusion of each iteration to match their evolving business needs. This helps ensure that the end users receive the highest value for their money. Additionally, since Agile minimizes up-front design, there is no “lost” work should a requirement be dropped and another one added.
Agile Requirements: User Stories
From a requirements perspective, a team can never really collect "all the details" up front (if they did, they would have written the software "up front"). Agile embraces this reality, and only requires high-level requirements at the beginning of a project. These are typically collected and stored as a set of User Stories. A user story is a plain English explanation of what a user of the system wants to achieve. They typically take the form:
"As a I want to so that I can " For example…
"As a Utility Manager I want to create a map of active work locations so that I can allocate resources effectively."
The nice thing about user stories is that everyone on the project can understand them. As the team nears the time of working on a particular user story, additional details are collected from the pertinent end users. As user stories are collected, they become part of the Project Backlog shown in Figure 1
Managing the Process: Scrum
Scrum is the most widely used of the various Agile practices. It has been adopted by large numbers of teams at Yahoo, Google, Microsoft, Oracle and IBM, therefore it comes very highly recommended, with a lot of success stories (think Google Maps).
In Scrum, all of the user stories for a project are added into a Project Backlog (figure 1). The order of the items in backlog is determined by the priority assigned by the "product owner" (aka end user stakeholders). In this way the highest value items are developed first.
At the beginning of an iteration the team picks a set of items off the backlog which they determine they can complete in the iteration. Typical iterations are two to four weeks in length. The team then commits to completing the items they have selected by the end of the iteration. It should be noted that the specific items are not dictated to the team, and work items are not assigned to developers by a manager. An essential element of Scum is that the team is self-organizing and self-managing. The internal team commitment is what drives the team towards excellence. Once the items are selected, the team then starts working on the selected items. At the conclusion of the iteration, the team will have completed the backlog items they committed to. Completion is defined at designed, coded, tested, and documented functionality. This is known as a “potentially shippable product increment”. At this time the team reports back to the stakeholders and demonstrates the functionality they have created at an informal review meeting. They solicit feedback from the stakeholders, and create new backlog items for things which need to be changed or fixed. Figure 2 illustrates the mechanics of the Scrum process.
No Interruptions
During the course of an iteration, the team is not to be interrupted by other requests for their time – they are 100% dedicated to their current backlog. At the end of the iteration new items can be added to the backlog or team members can be pulled into other projects, but nothing can change during an iteration.
Daily Standup
In any process where people are self-managed and self-organizing, there needs to be a lot of transparency. In Scrum, the team meets every day for a 15 minute time-boxed meeting – called the “Daily Scrum” or “Daily Stand-Up”. At this meeting each team member answers three questions:
- What did you do yesterday?
- What are you doing today?
- Do you have any impediments?
By answering these three questions, the entire team is aware of the current project status, and issues are brought to the forefront very quickly. It is often said that while agile will not solve all of your problems, it will certainly surface them, because there is nowhere for them to hide. It is the responsibility of the “scrum master” (a servant leader charged with enabling team members to perform at their best) to remove any and all impediments to a team’s progress during an iteration.
Constant Improvement
At the conclusion of each iteration, the team conducts and internal team meeting – a retrospective. This meeting is a time for the team to reflect on and improve how they work. They discuss what worked during the iteration, what did not, and anything they want to change in the next iteration. During my team's first scrum project, much of our retrospectives focused on getting continuous integration systems setup.
Applying Agile to GIS Projects
Scrum can and has been applied to much more than just software projects. It can be used to manage anything that involves a team and some sort of deliverables. Thus, it could readily be applied to any GIS project. That said, we believe that GIS development will see the biggest benefit, simply because there are so many tools and techniques developed for mainstream software projects which support agile methods.
Tools for the Agile Developer
Alongside those working on creating agile project management practices, there were groups of developers refining their development tools and techniques to embrace changing requirements, while at the same time increase efficiency and reduce defects. These techniques are collectively referred to as “extreme programming” (XP). The following is a listing of the common tools and techniques that empower the developer. The list is arranged in the order that teams typically adopt the tools/techniques.
- Use of a Source Control system
- Automated Source Code Documentation
- Use of refactoring tools to ensure clean software design
- Writing Unit Tests
- Creating an Automated Build process
- Utilizing a Continuous Integration system
- Test Driven Development
Source Control Systems
A source control system simply manages the various versions of files used in a development project. These files can be source code, documents, or graphics. I would tend not to put data in a source control system unless it was used in some unit tests. The source control server allows the team to store successive versions of their source code, as well as orchestrate the movement of changes between the various team members such that no changes are lost or overwritten. Given that Subversion is free, easy to install and easy to use, there is no reason that any developer should not be using source control. Assembla.com takes it one step further – anyone can signup for a free account, and create as many free public or private Subversion repositories as they want. You can then invite other members to join your project. Additionally, they have tools built into the site that Scrum. Assembla is so compelling that for the time being our team is actually using this service instead of setting up our own local Subversion server.
Source Code Documentation
Also called automatic documentation, this refers to the inclusion of special types of comments within the code base, which are then automatically extracted into a set of documentation of the code. In Visual Studio, this is supported for both Visual Basic and C#. When these comments are present, the compiler automatically extracts them. Additional free tools (NDoc and Sandcastle) can then be used to generate windows or web help documents. Another tool worth mentioning is GhostDoc. This tool examines the name of the class/method/property and "ghost-writes" the documentation for you. What's particularly cool is that it will automatically reference the parent classes documentation from a child class. Just get it.
Refactoring
Refactoring is the process of reworking the internals of a piece of code, while keeping its external behavior the same. In ArcGIS development, there is a tendency to write long blocks of processing code – select a feature, buffer it, select other features, get related records… etc. A common type of refactoring in this case would be to extract blocks of the long function into separate functions. The reason for doing this is that you want to move towards having unit tests for your code – and when creating tests you want to test small parts independently. Doing this sort of refactoring manually can be difficult, but refactoring tools like ReSharper can greatly simplify things.
