As part of my job, I help customers to select the appropriate software to either fulfill a need or as a component of a larger solution.  Fulfilling this role means comparing similar software offerings and selecting the best fit.  The challenge in this goal is to map the vendor offering into a subjective requirement, such as “best fit”.

Throughout my career I have acted in the role of consultant, architect and analyst.  In each role, I have had the same challenge of identifying the best in particular product category.  As a consultant/architect, the selection criteria was based on either an enterprise need or a single solution need.  As an analyst, the challenge was a bit more difficult because the criteria was based on being a solution to a problem domain.

Ultimately, comparing and scoring different products can take two approaches: qualitative and quantitative.  Both can be useful depending upon what you are looking to achieve.  I have found that I prefer to be a qualitative analyst than a quantitative one because in most cases I find ratings useless and arbitrary, whereas a qualitative answer illustrates depth of reasoning for a particular outcome.

I once walked away from an assignment because the research firm wanted me to quantitatively compare CORBA, DCOM and RMI as distributed object computing frameworks.   As much as I tried to explain, that they all have equal merit, the ultimate decision if one is better than the other is the environmental factors, not the individual technology.  Unfortunately, that fell on deaf ears in a firm grounded in scaling each feature 1 – 10.

With regard to qualitative and quantitative analysis of particular products, it gets even more interesting when you start to get the vendors involved.  A quantitative matrix usually entails requesting if a product supports a particular piece of functionality.  For example, do you support LDAP.  Having worked for multiple software vendors, I can tell you filling out these questionnaires by analyst groups can be an entertaining activity.  While certain features are straightforward, there were plenty where the answers got as interesting as, “yeah, if we hang Jim out the 3rd story window on a windy day, while he’s scratching his a$$, I think it we could do X.”  Hence, the resulting set of answers tells you if a vendor’s product has a certain level of completeness, but it doesn’t go into enough qualitative depth to decipher if the implementation is any good.  Moreover, even if the implementation is good, does it fit the purpose for which it will be applied?

An additional benefit of doing qualitative analysis over quantitative, and one that is critical to product vendors, is that you can truly identify where products are differentiated.  For example, many of the SOA platforms out there today, on the surface, all seem to provide similar functionality.  What really differentiates them is intangible’s, such as how much training is required for a developer to be productive or how easily can you build and deploy a service.  Quantitatively, IBM’s WebSphere’s platform clearly wins time and time again when comparing SOA platforms on completeness, but if one explores how much training is required and how many steps are required to build and deploy a simple Hello World service, the simplicity of a Spring Web Services container becomes more attractive.

Ultimately, I believe that this divide is central in IT failures.  Products are selected based on the wrong criteria.  Products that are selected quantitatively, may not be the right product for a given environment.  IT groups that go outside the “popular” choice and leader quadrant on the Gartner reports and look at how a particular product supports their specific needs for a given solution are often more successful.  Additionally, while not popular due to how it increases product proliferation, businesses will be more successful if they stop overloading their product selections with Enterprise requirements and select their product more tactically.  While this seemingly hurts the pocketbook, the losses in productivity and inefficiency–which is often not managed and, hence, difficult to compute–will outweigh the licensing and product management issues.