Why Build? Why Buy? The Proverbial IT Conundrum
Why Build, Why Buy?
The Proverbial IT Conundrum
Over the past 50 years or so, IT (a.k.a. Data Processing and Information Systems) has been spending hundreds of billions of dollars each year. After all of this time and money, IT still seems to be stuck in a rut that continues to perpetuate the problems that have plagued the industry from its beginning. These problems have been so persistent that the IT industry continues to perpetuate them even today.
These problems are frequently described by the following criticisms.
- Takes too long
- Costs too much
- Doesn’t meet requirements
- Difficult to change
- Maintenance costs and time are too high
- Required data not available or accessible
These problems have been translated into one-word issues that confront the IT industry. These issues are:
Alignment and integration have consistently been the top issues facing IT (as identified by IT executives) for the last 30 years or so.
The basic thrust of IT has been to take advantage of advances in technology and use that technology to deliver value to its customers, usually an enterprise, in the form of information systems (i.e. an application and its databases). This translates into a very strong dependency between the business process, its supporting application, and the associated database(s).
The IT industry has embraced an extensive list of purported solutions to these persistent issues without much success. This list includes:
- Structured Analysis
- Structured Design
- Structured Programming
- HIPO (Hierarchical Input Output) Technique
- Method 1
- Enterprise Architecture
- RAD (Rapid Application Development)
- Object Oriented Analysis and Design
- Business Transformation Enabling Program
- Use cases
- Agile methods
- SOA (Service Oriented Architecture)
- Data Warehouses
- Canonical Data Models
- Data Marts
- Federated Data Models
- Component Based Development
- Enterprise Architecture
- MDM (Master Data Management)
- CRM (Customer Relationship Management)
- Process Re-engineering
The above list does not contain the names of hardware and software technologies that have been marketed as providing solutions to persistent IT problems. That list is also quite lengthy.
Hindsight, of course, has a clearer vision of what is happening or has happened and why. Looking back over the last 30-40 years, many of the so-called advancements have been solutions to the symptoms of the problem(s), not solutions to the root cause(s) of the problem(s). There is always a great temptation to “leap to a technical solution”; to go with the apparent quick-and-easy solution even if it just glosses over the causes of the problems. It is kind of like living an unhealthy lifestyle – we become ill, go the doctor to get immediate relief from the symptoms (a.k.a. please make the pain go away!), but take no action to treat the root causes of our illness. Treating the symptoms of illness is a lot easier than changing your lifestyle and personal behaviors in order to deal with the real, underlying causes of our problems.
There is a fundamental cause of these problems and it lies in the basic analytical construct that was adopted at the beginning of the development of information systems. This underlying analytical construct was the “Input-Process-Output” structure for specifying the business process and its target application. This analytical construct is, in my opinion, what got us off on the wrong track. Worse yet, the construct was applied backwards. First we defined the output, then you defined the input, and then we defined the process (transformations) necessary to take the input and produce the output. At first blush, this approach did have the appearance of a fast path to develop and deliver functionality as fast as you can. To some extent that was achieved. However, the cost and the consequences of this construct have been significant. A natural extension of the Input-Process-Output approach is the design and development of application-specific databases which, in turn, created their own set of data-cost and data-quality problems.
Add to the mix are a variety of strategies that have been put forth as a means to deal with many of the persistent IT problems. This list includes:
- Build versus Buy
- Outsourcing (various)
- Shared Services
- Timesharing Services (various)
- SaaS (Software as a service)
- Web Applications
- Cloud Computing
The “Build Versus Buy” strategy is a very instructive situation. As with many of the other purported solutions, this strategy does not provide a very strong, convincing argument for either Build or Buy. Why develop ? Why buy ? Why does this debate even exist? The reason is that neither approach has produced stellar results as many enterprises have already found from experience. Much too often both approaches have resulted in excessive costs far above the initial estimates, schedules that have doubled and tripled, and the resulting systems have been disappointing in terms of alignment with business requirements and promoting enterprise integration.
Very often efforts to develop or to buy and implement packages end up as salvage operations trying to get some value for the cost and time expended. Sometimes these projects are just stopped and the enterprise just takes its losses. And sometimes the enterprise then tries the “other” approach, too often with the same results.
There is no profession or industry with a more spotty record that continues to thrive and prosper at others’ expense than the IT industry. This is not to say that there are no information systems efforts that have achieved some level of reasonable success and delivered value; however, the shining examples of glory are few and far between. To complicate matters, it is pretty hard to predict which information systems efforts will be successful and which ones won’t. Why does this situation seem to go on and on?
The problem with both the “Build” and/or “Buy” options is that the fundamental approach to the development of information systems (applications and databases) is defective. Whether you develop the information system in-house (i.e. the “build” option) or buy a package from a vendor (i.e. the “buy” option), the development approach used in both cases is basically the same. Therefore, the information systems that are acquired using either approach suffer the same fundamental flaws.
These flaws are not inherent in the information technology itself. The basic problems lie in the concepts, principles, methods, and techniques that IS professionals use to conceive, specify, design, and build the information systems applications and databases. (Did you notice that I emphasized applications and databases? Hopefully, you will understand why shortly.) This is not a technology problem. This is a professional knowledge and skill problem, and therefore very much a management problem.
The community of IS professionals has not made any real fundamental changes in their concepts, principles, methods, and techniques in the last 30 years or so. Most of the efforts to improve the information systems development process have been attempts to improve the building of the application component of the information systems. These efforts have been focused primarily on increasing programmer productivity, slicing and dicing the work differently, and/or using different tools to do pretty much the same things in the same way as we have done them in the past. Very little has been done to improve conception and specification of the business processes (notice that I didn’t say applications) and even less in the area of specifying and designing enterprise data (notice that I didn’t say databases).
Some might argue that the advent of the personal computer (PC) and the Internet with Web applications are two very dramatic changes in the way the IS community delivers information systems. Yes, these did change the infrastructure and the computing platforms; however, IS developers continued conceiving, designing, and developing information systems pretty much as they had done for decades. In fact, some attempts at changing how they built the information systems actually made the underlying problems worse. For instance, the data problems have gotten worse, much worse. And the cost and effort to compensate for these problems is growing with usually, at best, marginal results.
The IS industry has also been seeking the promised land of “reusability.” For over 20 years the IS industry has been trying to figure out how to leverage existing functionality by reusing that which they think they know to already exist. This crusade has been known by such names as component-based development (CBD), model-based development (MBD), and Model Driven Architecture (MDA). Even the more current popular approaches such as Object Oriented Analysis and Design, Service Oriented Architecture (SOA) and Agile methods are efforts to discover the “reusability” promised land, however, they are not solving the basic root causes and problems. In some cases, these efforts are creating more and more redundant code that will need to be maintained for decades to come; and are actually creating more and more redundant data which is further compromising data quality.
There are several reasons why these problems persist. A few of the reasons for this persistence are the:
- IT “Job Shop” business model.
- Obsession with Budgets and Schedules.
- Search for the “Silver Bullet.”
- Process/Application Bigotry, a.k.a. “Data Discrimination.”
- IT and User standoff.
- Cost Versus Value Disconnect
- Focus on Who does What, and the Lack of Focus on How and Why.
The reason I list these in this order is that it generally reflects the premises and priorities that IT management uses in the way they conduct their business. The first problem is the sequence of this list; if a lot more attention and effort went into the “HOW” and “WHY” in Number 7, most of the other problems would be greatly diminished.
It is highly unlikely that we will ever significantly improve the capability, quality, and maintainability; the alignment, integration, flexibility, and responsiveness; or reduce the time and cost to deploy information systems, as long the “Job Shop” mentality persists.
The Job Shop approach is the Orville and Wilbur Wright business model. Orville and Wilbur are recognized as the inventors of the airplane; however, they did very little to advance the process of building airplanes. As of 2011, we know who has excelled at conceiving, designing, and building airplanes—Boeing, Airbus, Bombardier, and Embraer. A lot of airplane manufacturers have fallen by the wayside in the meantime.
A key feature of the Job Shop business model is that you wait for a customer to appear and request a product before you do any design work. The Job Shop approach is very common when an industry is in its infancy. However, it is not a business model that is effective as an industry matures. Why? Let me give you a few reasons.
In the Job Shop approach little, if any, attention is given to designing for reusability, because it is a given that every product is a “custom product”. Therefore, the attitude is: “How can we design a part for reusability since we don’t know what the next product will need? Besides, if we attempt to do that, it will take a lot more time and cost a lot more. And besides that, who knows if there is going to be another product”. That, of course, will cause budget and schedule paranoia to surface which will kill the whole idea of designing for reusability. The result is we are trapped in the Job Shop mentality. Further, there is very little, if any, parts interchangeability or reusability; there are very high lead times, very high product costs, and very high maintenance costs. Don’t those sound like many of the pervasive and recurring information systems problems?
There are two very fundamental reasons for the obsession with project budgets and schedules. One is that information systems projects have a really bad record of meeting planned budgets and schedules. As the costs have escalated, the perception of unacceptable risk has driven many people to move more and more to a “buy” bias, because they perceive that it will provide the lower-cost, least-risk option. Evidence is pretty strong that this perception is incorrect.
The other fundamental reason is more subtle. IT management, as well as the enterprise management, tends not to be actively engaged in “how” the information systems development process really works in detail. So, lacking that knowledge and understanding, they tend to deal with the more general subject of the project budget and schedule since that is what typically interests management the most. The buyer is always interested in “how much does it cost and when can I get it”. The buyer would rather not deal with the details of “how” it is built.
As an industry, the information systems development process is not very well defined, and there is not much rigor in the development of metrics that can serve as a basis for developing reasonable information systems development project budgets and schedules. Budgeting and scheduling in the information systems development arena pretty much involves proposing a budget and schedule that will be acceptable, not necessarily realistic.
Another aspect of the information systems development budgeting and scheduling process is that it is driven by variations of the following directives or questions: “Do it for ‘X’ amount of money!” or “Do it by ‘X’ date!” or “What can be done for ‘X’ amount of money?” or “What can you get done by ‘X’ date?” The situation really gets nasty when these directives and questions get combined!
The first fundamental problem here is that a business process is what it is. Its size, shape, complexity, etc, does not vary depending on how much money you want to spend on automating it or on how much time it will take. Don’t get me wrong; I am not advocating that we dispense with using budgets and schedules as planning and control mechanisms. However, we need to make some fundamental changes in the way they are created and used.
The “Project Approach” has been used since ancient times—in the IS business that means way back in the 1950s! To complicate matters, the Job Shop and the Project Approach are highly incestuous. If you have a “Job Shop” business model, you are almost guaranteed to be using the “Project Approach”.
The vast majority of the effort to improve the information systems development process is focused heavily on how to design and build business process functionality, meaning the “application”. Building business process functionality is important, but it is not the only end-game, and there are one or two other critical aspects that are just as, if not more, important. How about enterprise alignment, integration, and flexibility? And then there are the ever-present—and becoming much more relevant—issues of data quality, availability, and accessibility.
Our attraction to and fascination with technology tends to blind us from looking at the basic pervasive problems of the Information Systems industry, which has become so preoccupied with the technology, that it now almost universally likes to be known as the “Information Technology (IT)” industry or profession.
We seem to be emphasizing knowledge and skill in the use of technology over the knowledge and skill in analyzing and designing the enterprise data and business processes. Without improved enterprise data and business process analysis and design, the employment of the technology will be expensive, over-sold, under-utilized, and too often disappointing. I have often used a variety of analogies to help people better understand this critical issue. So, here are a few of those analogies:
- Just because I have a tractor and know how to operate it does not make me a competent farmer. There were very good farmers long before there were tractors.
- Just because I have Microsoft Word and know how to use it does not make me a competent poet or a best-selling author. There were very good poets and authors long before Microsoft Word existed.
- Just because I have a fine set of high-tech carpenter tools and know how to use them does not make me a competent carpenter. There have been some mighty fine carpenters who used very basic hand carpentry tools.
- Just because I have Microsoft Project and know how to use it does not make me a competent Project Manager. There were some very good Project Managers long before the existence of Microsoft Project.
- Just because I have a nifty, hand-held calculator that can calculate a square root does not mean I know how to calculate a square root. In this case, the calculator knows how to calculate a square root; I know how to operate the calculator. These are two different skills. Without the calculator, I become a square-root incompetent, hoping the batteries don’t die at an inopportune moment!
Technology may make a person more productive, but it will not necessarily make them competent at their profession. We have all heard the cliché “garbage in, garbage out.” Well, producing more garbage faster just produces a lot of garbage.
The issue isn’t so much whether we should “build or buy”. The real issue is HOW that which we want to build or buy is built.
The primary issue in the current build-or-buy debate is WHO is going to build it, and WHAT functional capability does their product provide? Currently WHO does the building pretty much uses the same old HOW to do the building. The essential, more relevant question should be “WHO does WHAT and HOW?” And, you might even need to know a little about WHY! The current debate mostly focuses on the WHO and the WHAT and very little on the HOW and WHY. The paramount issues are:
- “HOW” was the business process designed?
- “HOW” was the enterprise data designed? and then
- “HOW” were the databases designed and built? and
- “HOW” was the application designed and built?
The answers to these “HOW” questions will better explain how the pervasive, persistent causes of information systems problems can be recognized, understood, and avoided.
By the way, what technology is used to build the information systems is not the answer to the HOW question. Technology is the answer to the “Which tools were used to build the information system?” question!
In most industries the customer spends most of their time and effort knowing about the WHO and WHAT, not a lot about the HOW. Knowing the answer to the HOW questions is the primary responsibility and under the control of the designer, builder, and manufacturer.
HOW is the primary determinate of the cost and quality of that which is produced. The customer’s perception of the value of WHO and WHAT are determined by the cost, quality, and availability of the product and/or service. The critical difference between Toyota and General Motors is not WHO they are or WHAT they produce, but HOW they conceive, specify, design, and build their product and/or service.
The Information Systems profession needs some major improvements in HOW it conceives, specifies, designs, and builds its products and/or services. The real question is “Who will be the Toyota of the Information Systems business in the future?”
The answer to this question is: “It will be whoever figures out HOW to fundamentally fix the information systems HOW process.” Do you suppose that, if the Information Systems profession dramatically improved their HOW, their customers’ perception of WHO they are and WHAT they produce might improve?
Relatively speaking, if I were to make a assessment of what the information systems industry is today, it is probably a Packard … striving to be a Tucker … when it should be striving to be a Lexus! (I know; the younger people among us probably won’t understand this analogy!)
The issue is really not so much a conflict between whether to build or buy a package. It is an issue of HOW the development is done. Whether you build the information system yourself or buy it, the end result—the information system—was built by someone, somehow.
Only the “build” option has the potential to serve the strategic, tactical, and operational interests of an enterprise. Currently available packages cannot serve the strategic, long-term interests of an enterprise; they can only serve short-term operational and tactical interests. It is possible to build packaged applications that can be dramatically better, but it is very difficult for them to excel at alignment, integration, flexibility, and responsiveness, or to deal effectively with the data problem.
Packages, almost by definition, cannot solve their inherent shortcomings of deficient alignment and integration capabilities; and they are a major cause of data redundancy and the attendant data quality problems. The reason is that a package is always developed out of context of any enterprise. At least no package vendor has yet produced a comprehensive suite of information systems that would integrate the enterprise vertically and horizontally. The result is that a major implementation issue with packages is the effort necessary to “interface” the package with the other information systems in the enterprise. If the enterprise ends up with packages from different vendors, the complexity and difficulty of interfacing these heterogeneous information systems gets magnified. The resulting redundancy of business process functionality and data will cost the enterprise dearly and unnecessarily, and make the ability to change and adapt roughly the equivalent of trying to break out of an iceberg.
Not only is it important for an enterprise to be flexible and responsive, but, more importantly, the enterprise must be flexible and responsive in the way it wants be, when it needs to be. Packages, as currently delivered do not, cannot, and will not serve this critical business requirement.
Currently, package vendors thrive best when they have one version of their package and lots of enterprises that buy their package. The ideal situation for any package vendor is for every one of their customers to use the same version of their package. Unfortunately, this situation is not going to result in any one of the enterprises in question having a so-called “competitive advantage” and these enterprises are not going to be able to change and adapt on their own terms. At best, the package user will have the capability to be average in relation to their competitors. Of course, if the enterprise in question is below average, using the package approach would be seen as an improvement.
However, I do think that it is possible to build and deliver “package” solutions that could be considered “starter sets” and then turn the package over to the customer to enhance and maintain as they see fit. Of course the architecture of the application(s) and the database(s) would have to be dramatically different than what currently exists.
There are several current practices that negatively influence the approach to how enterprises acquire (i.e. build or buy) their information systems. The most essential issue that must be addressed in order to affect a meaningful improvement in the process of acquiring information systems is the information systems development process.
Information Systems professionals have been building information systems which typically consist of the “application” (i.e. the processing part of the information system) and a “database” (i.e. the data storage and access facility) using a selection of information systems technology. The basic premise has been that, if you have an application, there must be a database to support that application—which almost universally has led to application-specific databases—which I refer to as “application databases”. The net result has been application databases which contain redundant, inconsistent, and fragmented data with significant variations in the quality of the structure and content of the data among the application databases. This is perhaps the most significant root cause of the proverbial “enterprise data problem!”
This “application-database approach” also produces a lot of redundant application logic which may or may not be consistent—usually not. This situation will most likely persist if we don’t take decisive action to change the current pervasive, conventional systems development methods and project approach to acquiring and implementing information systems.
If we, the community of IS professionals, want to produce information systems that are at least aligned and integrated, then we must stop using over and over and over again the same flawed development process that has produced the same unsatisfactory results for more than 50 years.
I believe that proven advances in Enterprise Data Management and the adoption of an Enterprise Database approach present us with a great opportunity to resolve the current IT conundrum.
Douglas T. Erickson
ENTARCO USA Inc.