Program Manager: David A. Fisher dfisher@micf.nist.gov, 301-975-3649, fax 301-926-9524
This program is a focused attempt to move software development from expensive, error-prone, hand-crafted, laborintensive production of monolithic multifunction point solutions, to automated composition of independently developed fine-grained components that are competitively available through a multilevel value-added software supply chain. It envisions a radically different semantic-based software world in which any functionally compatible software can be automatically identified and safely combined. The need for software interface standards would evaporate. Software representations would be automatically transformed for compatibility at each point of use. Many sources of cost and error today would be reduced by assigning issues of representation, implementation, and composition to an automated domain. Focus in software development would shift from the artifacts of the design and implementation processes to the application problems being solved, including issues of functionality, performance, interoperation, dependability, and security. Risks that now discourage private investment in software would decline. A new market for fine-grained software components and specialized high-performance tools would enable quality improvements through technical specialization, allow new software technologies to move rapidly into widespread use, expand markets, and reduce both time to market and cost of market entry.
The software industry itself would likely evolve in four directions. A software machine-tools industry would supply automated high-performance software tools and a variety of software architectural consulting services. A software components industry, analogous to chip makers in the microelectronics industry, would supply fine-grained components that would eventually end up in enormous numbers of software products, many of them unanticipated. Both of these industries would supply each other, as well as a much larger custom and vertical-market software industry which would assemble end-user software products and packages from largely purchased components. Most software, however, would be produced by end-users of application-specific packages in much the same manner that customized spreadsheets are produced by accountants rather than programmers today. All of these industries would require highly specialized skills and would produce higher value-added output per worker. The importance and role of many traditional programming tasks would be analogous to that of circuit and logic design today. Program Plan and Scope
Two solicitations are planned approximately 18 months apart. The first solicitation will begin to lay the technical foundations and to illustrate the independent benefits of the supporting technologies, but will not attempt to establish the end-user confidence and critical mass necessary for selfsustaining commerce in component software. The second solicitation will be constrained by the foundations and directions defined by ongoing projects, and will be more narrowly focused on technologies needed to complement existing projects and to ensure success of the program as a whole. It will be aimed at filling gaps and imbalances in the program's portfolio of projects and at satisfying requirements generated from knowledge of progress elsewhere in the program. If there is sufficient early progress, the program may be expanded to address issues of critical mass in available components and tools, of adequacy and completeness in establishing a foundation for viable commerce in software components, and of convincing demonstrations in real industrial or commercial settings. In the outyears projects may be conducted jointly with other ATP programs that are focused on complex software developments in specific application domains, but would not themselves develop the advanced software technologies of this program.
The first solicitation will focus on the three generic elements of research and development as defined by the program's technical goals. The business goals of the program represent the anticipated synergistic and cumulative effect from satisfaction of the technical goals. All projects must fall within the technical scope as defined by the program's technical goals. Project proposals should clearly identify which technical goals they are addressing. Especially sought are projects that involve technological innovations that are deep and narrow, that involve semantic approaches, that increase automation within the software development process, or that have potential for order of magnitude gains relative to some measure of software quality or cost. Ideas that are limited to personnel management techniques, manual approaches, data collection, or are incompatible with semantic approaches and fine-grained software components will not be funded.
It is important that participants be willing to cooperate where appropriate to achieve synergistic interoperability and other mutual benefits. Projects that are critical to overall program success require the highest degree of commitment to interproject coordination and cooperation. Joint ventures involving normally competing companies are welcome. Participation by platform providers, systems integrators, and end-user organizations as joint-venture partners is encouraged. It is anticipated, however, that the majority of participants will be small software firms. Abbreviated proposals (preproposals) are allowed for the first solicitation of this program.
Where appropriate, projects should be planned in consideration of the vastly different business, economic and technological contexts that may prevail in the coming years. Anticipated changes might include the effects of increased levels of economic competition, the virtual corporation, rapid technical advances in a variety of physical technologies, the effects of success in this program, ubiquitous dependency on information technology, the emergence of a national information infrastructure, and the eventual replacement of many physical products by information products and substitution of communications for transportation.
Program management will emphasize three points: synergy among projects, effective transfer to widespread use, and relevance to end-user applications. Coordination among the projects and with the impacted industries are especially important. An annual meeting of participants will be held to supplement bilateral interproject interactions, to report progress toward program goals, to increase rapport among participants, to discuss new and continuing technical and business barriers, and to seek advice for updating program plans.
Overcoming nontechnical barriers also requires involvement and visibility by a wider community of informal participants and affected industries. Particularly important are end-users in software-dependent industries, platform providers, potential horizontal product producers, and custom and vertical-market providers. One approach is a system of explicitly coordinated projects that are not ATP funded. Another is a periodic conference addressing technical and business issues, possibly augmented by a newsletter and on-line services. FTP, world-wide web and email services for information on ATP software and information technology activities, related discussions by the industry, and identification of partners will be available.
The ATP budget for this five-year program is estimated to be $150 million. The private-sector participants must meet ATP matching fund and indirect cost payment requirements and are expected to contribute a similar amount of funding. Outyear ATP funding is contingent on future appropriations.
Despite a high degree of shared low-level functionality across broad classes of applications, a viable software parts industry does not exist to exploit it. Applications are often built from scratch as monolithic structures without benefit of purchased components or identifiable reusable parts. Each vertical-market supplier and custom builder must possess every horizontal software technology speciality in-house. Thus, it is not surprising that software is expensive, unreliable, or shows little general improvement over time. Market pressures that have motivated a multilevel supply chain structure in other industries to recover costs, improve quality through specialization, and enable rapid assembly from purchased components, increasingly apply to software as well. The primary barriers are technical: we do not know how to independently produce reusable software components. Currently, a software
component cannot be used at all without prior agreement between its designers and the designers of each component with which it must be composed or interoperate. The primary technical goal of this program is to develop pacing technologies that will surmount this barrier.
Alone, technical success in this program is unlikely to change the entire basis of competition within software production and use. To have lasting impact technical success must lead to products that can sustain significant subsequent use. Four nontechnical barriers offer substantial potential for holding commerce in software components below levels needed to become self-sustaining:
Although it will be difficult to reach critical mass, once reached, acceleration beyond that point should be overwhelming. Automated semantic-based fine-grained composition generates widespread reuse, which enables technical specialization and competition, which in turn encourages quality improvement. Increasing automation and declining duplication of effort push down costs while raising industry capacity. Automation eliminates the error-prone aspects of software implementation and can assist in requirements validation. With widespread commerce in software components, the entrepreneur with a great technical idea could rapidly capture the innovation in a broadly reusable component product. This contrasts with the situation today, in which technical innovation in software can be exploited only by selecting a vertical market niche, raising millions of dollars of investment capital, hiring a wide variety of software talents, and spending years producing a specialized product for that vertical-market. Furthermore, a multilevel value-added supply chain in software would provide a channel for rapid transition of new technologies to widespread use. In this sense, the software components industry would come to resemble the electronic components industry with its rapid infusion of new technology and seemingly endless succession of order of magnitude advances.
Although potential economic benefits from this program derive primarily from interoperation and quality improvements through specialization, benefits from reuse alone are significant and more easily calculated. Conservatively assuming that development of a reusable component would double its cost and that each reuse would cost a quarter as much as redeveloping the component, an average reuse of 10 times would lower overall costs by 60 percent. With even a five percent market share, the savings in software development today from reuse alone would exceed $2 billion annually. Because technologies sought in this program have the potential to change the entire basis of competition in software, estimating long-term benefit from the program would be as difficult as attempting to value the electronics industry of today based on the cost of transistors in 1960.
If the U.S. is first to move to software based on automated semantic processing of fine-grained components, it would gain a substantial advantage that would be difficult for another nation to duplicate. The software expertise currently concentrated in the U.S., together with a first-producer advantage accruing to U.S. developers, and an effective competition among U.S. firms in world markets would represent a significant entry barrier for foreign producers of software component and tool products.
In the absence of rapid expansion in capacity or a major paradigm shift, the U.S. may lose its dominance in software. The package software industry is already running full-out trying to serve foreign demand. Most of the real foreign demand, like that in the U.S., is in custom systems that cannot be addressed commercially today. This combination is creating tremendous pressure for creation of foreign software industries which, once established, will quickly become competitive with U.S. companies.
A growing number of nations, including India and Russia , are now offering programming services at a fraction of their cost in the U.S. With or without this program, the U.S. will not likely retain jobs for the low value added portion of software workers. This program would enable software professionals to become concentrated in larger numbers of more specialized roles in software architecture, design, component and aspect development, automated implementation technologies, tool building, or focused application domains.
There is another more subtle national advantage. In software development, Europeans emphasize rigor and discipline, and the Japanese various forms of pragmatic discipline. Americans pursue a gun-slinging pioneer approach with little discipline and devoid of rigor but with a large measure of risk-taking and survival instinct. Rigor is theoretically ideal, but generally does not scale to real problems. Formal methods have proven effective in Europe, but are generally abhorrent to Americans. European computer scientists regularly ridicule American software methods, never realizing that we succeed not in spite of our lack of discipline, but because of it. This program embraces the benefits of discipline in formal methods but assigns them to the automated domain, while preserving undeveloped open spaces of new higher-level semantic frontiers. This qualitative advantage is unlikely to be understood, let alone exploited, by more disciplined cultures.
Software technology is also strategically important. It is infrastructural and enabling for all industries. It is pervasive in most industries and rapidly becoming ubiquitous in our society. Software is the ultimate green technology and is increasingly used to eliminate environmental ill effects in other industries; fly-by-wire for example reduces weight, fuel consumption, and cost. Software is a critical enabling technology for a national information infrastructure (NII), but, given the current frequent failures in large-scale software developments, is incapable of satisfying the envisioned levels of complexity and interoperability. An NII would also increase demand for software hundreds fold, enable replacement of physical products and services by information products (i.e., software), and create many new, as yet unimaginable, information-based industries.
Deficiencies in software are a reflection of human limitations in understanding and managing complexity during the software development process. These limitations are aggravated by emphasis on issues of syntax, representation and completeness, which relate to the artifacts and processes of software construction. Instead, the focus must be moved to issues of application needs, software architectures, and the semantic aspects of design. This program emphasizes semantic processing, automated composition and synthesis, and fine-grained interoperation. It seeks innovative technologies that will raise the developer-interface level for end-applications to that of specification of requirements, constraints, functionality, and performance; that will automate various aspects of implementation and requirements satisfaction; and that will contribute to a critical mass of reusable components. The program seeks automated tools and tool fragments that can work cooperatively in a context where the typical software developer has minimal direct involvement with issues of syntax, form, representation and implementation.
Neither the pervasive concentration on low-tech solutions of the past decade, the artificial intelligence debacle of the previous decade, nor long-standing nonscalable theoretical solutions have produced demonstrable increases in the reliability, affordability or utility of software. For a given functionality, the perceived costs in labor, frequency of errors, demands on machine resources, and the ratio of response time to hardware performance has been rising rather than falling. Typical applications today have mean-time-between-failures measured in minutes and contain thousands of known errors at the time of production release. A new direction is needed. Innovations that cannot be integrated with other software technologies on an automated basis would be inappropriate.
The pacing technologies for this program will be ones that automate composition of independently developed software, and enable interoperation of fine-grained software in any functionally compatible context. Listed below are a few incomplete examples of ideas in selected areas and aspects of software technology. Larger numbers of better ideas on a broader range of issues are expected in project proposals. The examples below are to illustrate neither the best ideas nor the anticipated focus of actual project proposals; they are offered only as indications of the breadth and depth of ideas that could be appropriate. Some ideas are:
Effective automated solutions are beyond the reach of most industries today because software is unaffordable, inadequately reliable, or insufficiently responsive to real end-user needs. In industries that are already software-intensive or vitally dependent on software, there are few, if any, commercial products; most software is produced in-house by its end-user company or by a custom supplier. The most common exception occurs when a solution developed for one member of an industry is made available throughout the industry. Widespread end-user disillusionment with the extant software condition has shifted software research to application-specific solutions, and has reduced expectations for the potential of automated systems in general.
Since October 19, 1993, ATP has received over 100 white papers suggesting program ideas in the area of software and information technology. Of these, 56 contributed to this program. Although expressed in a variety of ways, common concerns included the need for more systematic reuse, interoperation, automation, and semantic approaches.
In January and February 1994, ATP held a series of regional workshops and public meetings to discuss white papers, alternative program focuses, and industry participation in possible ATP programs within software and information technology. Total attendance was 316 representing 206 organizations. The majority were key decision makers and chief engineers from software and software-intensive companies. In surveys taken at the workshops, after the least interested 10-20 percent had already left, there was nearly unanimous indication of intent to participate should a program be established in this area. Several local non-profit industry associations also participated and indicated their intent to follow through in matchmaking and in encouraging and organizing their member's involvement.
Both during and after the workshops, there was intense interest by the industry in joint-venture partnering. NIST participants observed more than a dozen incidences of partnership discussions initiated at the workshops. There has been a flood of calls requesting assistance in identifying partners since the workshops. The complete list of white papers with company sponsors and contact information for all those received in the areas of software and information technology is included in the workshop proceedings. Those that contributed to this focused program are so indicated. Copies of the list may also be obtained via email by sending a request for "SIT White Paper List" to atp-sit-ws1-request @osi.ncsl.nist.gov.
The majority of participants are expected to be small companies with innovative approaches. Despite their many good ideas and their generally enthusiastic reaction to the possibility of a focused ATP program, most software companies remain unlikely to have heard of ATP. There are 40,000 software firms in the U.S. with an average of fewer than 20 employees. There are no broad-based industry associations, most companies do not belong to any such organization, and there are few traditions of cooperation. The five Regional Workshops on Software Technology held in January 1994, were effective in broadening interest and awareness and in encouraging cooperation within the industry. Additional workshops in software will be held both at a general level and on narrower topics in anticipation of possible future focused programs.