new features -v- reliability
Its an age old question, a question that has been around since man first decided to to make things better. Do people prefer new and exciting advancements in technology, or would they rather have rapid change at the expense of robustness and reliability?
From a software developers point of view, we hate the idea that our competition may have a feature better than ours, even if we know that our overall product is better. We also take revenue from upgrade contracts, so we know that we need to provide value for money through enhancements.
The question then is this:- Is it possible to deliver software on time, that is both feature rich AND robust?
We worked out the other day – mainly because we were curious – that to test our software on every permutation of operating systems, Microsoft office, SolidWorks and SQL that we currently support would require 1620 testing environments. Testing on DriveWorks takes around 5 days.
Clearly 40.5 years would be a long time to get out a new release.
Obviously we don’t test DriveWorks completely on every permutation that we support. That would be commercial suicide. Releasing flaky software would also be commercial suicide.
So – what can be done about it?
Testing automation provides the main key to unlock the problem. We use TestComplete from AutomatedQA. Using TestComplete we have automated the testing of our new version of DriveWorksXpress (The one that is being shipped inside SolidWorks2008) which now takes just 5 minutes to run a complete test on a single environment.
The initial use of TestComplete has also shown significant benefits for the testing of our main products, reducing the testing of DriveWorks Administrator down to around 40 minutes on a single environment. More work needs to be done to include some of the richer features of DriveWorks that still require manual testing, but its a great start.