Back on May 10th, I started a blog titled: Software Development: More Than Code and Ship. My subsequent blog postings have covered my perspective on what goes into software development here at AMS Services. This blog will serve to close that loop by covering some other topics that relate to delivering working software to you, the customer.
A key aspect to delivering software is the build. At AMS Services in Portland, Maine, our objective is to build each product with as part of a single, automated process. This includes packaging of the build output into a product install. In fact, we strive to deliver all of our internal builds as installs so that Quality Assurance can validate both the installation and new functionality or defect fixes. Occasionally we run into problems – generally when we are in the process of upgrading our build machine software or installation software – but this impacts internal delivery, not the ultimate delivery to you. Even in these situations we have the option of testing updated code outside of the formal install, just to keep the ball rolling.
When I first joined the company as a developer (over a decade ago), we performed builds in the Portland, Maine office on developers’ machines. Each time a developer checked in code, that developer was responsible for building the affected EXEs and DLLs and moving them out to a shared area for others to pick up. Can you think of any pitfalls to this?
For a start, what if the developer forgets – or misses – checking just one file into the source control system? That developer could still build the affected programs, but someone else would not be able to. In fact, unless the developer was thorough enough to run through a build by refreshing his or her machine from the source control system, the problem wouldn’t be detected until the next developer in line tried to build using the latest source code.
As I mentioned previously, these days we use a separate build machine, one that can run through a full build of our product with a simple click of a button. One advantage to this is that in order to accomplish a build, the latest source must be retrieved as the initial build step. A successful build ensures that all of the appropriate files and changes have been checked into the source control system – and that no files have been inadvertently left on a developer’s machine.
I’ve personally felt the pain of this exercise in years past. It got to the point where I always ran a build first – before I incorporated any of my own changes. I can attest to the fact that I’ve had to get on the phone with developers late into the evening to work through identifying what was missing – or worse, I was discovering problems with building programs that someone else didn’t think to build because they didn’t realize that a change to a common library impacted some other program. The developer had built only what they thought they had affected.
Correcting problems where other applications would not compile was a real time-waster, since this process typically required additional changes to what others thought was already “done.” And if someone in QA had already tested the functionality, well, guess what? It had to be tested again. So, another advantage that a separate build machine brings to the table is that it ensures that the full product can be built all the time, incorporating any and all changes that have been checked in.
Can you think of another potential problem with developers doing the builds on their machines? Here’s another real-world scenario that I ran into here at AMS Services in the early days: I was assigned to look into a problem where one of our applications was “hanging,” meaning that it wouldn’t stop running when it was supposed to. It remained loaded in memory all the time, and could only be stopped by someone using Task Manager to terminate it.
When I started working on the problem, I naturally used our product install to duplicate the problem – which I did with ease. The next step was to create a debug version of the program by compiling the latest code in debug mode. As soon as I did that, I could not duplicate the problem! Did adding debug information alter the signature of the executable somehow to prevent me from easily discovering the issue? Was I going to have to resort to other means?
I immediately tested this theory by re-building the application in release mode on my machine and running it. And I could not duplicate the problem… I went back to the original executable shipped in our release and found that yes, the problem remained. Any guesses as to why?
The answer in this case was in a Microsoft service pack for the compiler. I had the service pack applied on my development machine, whereas someone else didn’t. My “fix” for this problem was to ensure that we built our software using a complier that had the correct service packs applied. A central build machine can be easily maintained and mitigates problems like this.
As I noted, we also create our installs from our build output, as a separate step in the build process. Over the years we have used Gupta SQLWindows, Microsoft Visual Basic and C++, and now Microsoft .NET in the Phoenix application suite. We also have used various third-party components (grid controls and the like) and we progressed with technology, building out COM components, .NET components, and lately, Web Services. This alone makes for a complex installation.
Installation packages must also track the versions of installed programs so that they can be repaired, or fixes applied at a later date, without having one fix inadvertently overlay an older program on top of a new program. Finally, the user must be able to uninstall the software if they so desire – and the product should “clean up” after itself without leaving things around on a user’s hard drive. We use a commercial package (InstallShield ) to handle these complexities.
In addition, our database schemas for both Phoenix and PremiumBill need to evolve as we alter functionality in our products. We have custom tools developed that are part of our upgrade process to allow our customers to reliably and easily upgrade their databases to support the new versions of our software. Our developers work in conjunction with DBAs to develop the release scripts, and testing database upgrade scripts is also part of the work that our Quality Assurance group performs during a release cycle.
And of course, there is the ever-present need for product documentation. This takes a variety of forms – from on-line help to release notes to data dictionaries to installation guides. Generally, documentation is being developed throughout the course of a project, and these days we make reviewing documentation a part of our release process.
Once a product is deemed “complete” from a functionality standpoint, it is still not quite ready to go out the door (this is true for major product releases, PTFs to address specific problems quickly are handled differently). System testing is conducted as a final check prior to releasing the product. Because there are so many changes occurring (both fixes and product enhancements) during a major release, we strive to perform system tests to ensure that as a delivery, our software will operate as expected for our user base, with all of the included fixes and enhancements.
In order to accomplish this, we “code freeze” at a given point and focus strictly on testing the product, with no new builds or development going into the product unless it is deemed critical and essential. If we introduce too many variables, we violate the intent of the code freeze.
This brings me to the final part of the “code and ship” tour: releasing the product. We have a “lockdown” process where we move the code to a secure staging area, and ultimately the FTP site for distribution where you, the user, can download the software. We also prepare an escrow CD with the code, source, and tools used to generate the build.
This concludes a series of blogs discussing the high points of software development at AMS Services, from my perspective. Hopefully I’ve been able to paint a picture about all the moving parts relative to software development, and why it is more the just code and ship. I welcome any thoughts and feedback!