Date: Tue, 12 Jul 1994 15:26:56 -0400 (EDT)
From: Michael Ransom
Subject: ATP Analysis of Abbreviated Proposals for Component-Based Software
To: atp-sit-ws1@nist.gov, atp-sit-ve1@nist.gov
Cc: ransom@wombat.ncsl.nist.gov
Status: RO
X-Status:
As described in the solicitation announcement for the ATP Component-Based
Software program, proposers were invited to submit optional abbreviated
proposals. A generic analysis of the abbreviated proposals is provided
in the included message below.
This message is being widely distributed to those who have shown
interest in any facet of ATP activities related to software and
information technologies. It contains information that almost
certainly will be useful to anyone contemplating future submission
of an ATP project proposal.
Michael Ransom
mransom@nist.gov
----- Begin Included Message -----
Analysis of Abbreviated Proposals
for
Advanced Technology Program (ATP)
Component-Based Software Competition 94-06
Our intent in soliciting abbreviated proposals was to reduce industry's
investment in preparation of unsuccessful full proposals. Consistent with
that intent, this analysis is being sent to all of those who requested the
solicitation package for Advanced Technology Program (ATP) Component-Based
Software Competition 94-06. There are three purposes in providing this
information:
* to encourage submission of full proposals that have innovative
ideas involving high technical risk and potential for significant economic
benefit,
* to discourage submission of full proposals that do not satisfy
the ATP selection criteria, and
* to improve the quality of all full proposals submitted.
General analysis
Ten of thirty abbreviated proposals received under Solicitation 94-06 were
judged sufficiently promising to justify preparation of a full proposal.
All abbreviated proposals lacked sufficient detail and provided inadequate
evidence to justify a superior rating relative to the selection criteria.
This may reflect the combination of limited preparation time and page
restrictions for the abbreviated proposals. Negative recommendations were
given only to abbreviated proposals that did not adequately meet the
criteria for technical or business merit and that failed convince
reviewers that a derivative full proposal would satisfy ATP's minimum
standards for an award.
Because abbreviated proposals were optional, the relevance of the following
information is unknown. Of the abbreviated proposals given positive
recommendations, six were from single applicants and four from joint
ventures. At least one of the single applicants was a start-up. Only one
addressed the first technical goal of a practical technology for automated
composition. Six addressed the second technical goal of developing
automated methods and tools with potential for a factor of two to ten
improvement in some important measure of software quality or of
productivity in software development. Of those six, five concerned
innovations that would push previously known approaches to their practical
limits and one proposed to apply well understood techniques in an
economically important but previously unaddressed context. None of the six
proposed radically new approaches that this program hoped to elicit. Each
of remaining three proposed an innovative technical approach to overcoming
one of the non-technical barriers to commerce in software components
identified in the third technical goal. We are hopeful that additional
high quality full proposals will be submitted in each of these categories.
While the percentage of abbreviated proposals under 94-06 that received
passing scores on both technical and business merit was significantly
higher than that for the General Solicitation, it remains lower than
desired. Many of the abbreviated proposals that received negative
recommendations could have been disqualified by their authors had they paid
attention to the basic requirement and the selection criteria.
Should you submit a full proposal?
There are three simple tests that can be applied to determine whether it is
worthwhile to submit a proposal to ATP:
* Do the proposing organization and project parameters
satisfy the eligibility requirements, limitations on assistance, and other
basic requirements of ATP? These requirements are found in the Proposal
Preparation Kit (dated February 1994), the Commerce Business Daily
announcements of May 6, 1994, and the Component-Based Software White Paper.
All of these documents are included in the Solicitation Package for ATP
Competition 94-06.
* Is the project based on an innovative idea of high
technical risk and potential for significant economic benefit?
* Does the proposer have the strategies, commitment and
ability needed to see the project through to technical and business
success?
If you have a good idea that satisfies the above three tests, we strongly
encourage you to submit a full proposal. The odds remain very high for
funding of quality proposals submitted under this Component-Based Software
solicitation. Proposals that do not satisfy ATP's high standards for
technical and business merit, however, will not be funded regardless of the
number of submissions.
Full proposals
A full proposal to ATP should be a convincing explanation that the above
tests have been satisfied and should be presented in a form that can be
evaluated against the ATP project evaluation criteria. The criteria begin
on page 58 of the Proposal Preparation Kit and are summarized on pages 8
and 9 of the same document. Proposals should address all five criteria and
each subcriteria. Explanations and evidence should be given in support of
each point being made. Your proposal will be read by both technical and
business reviewers. While technical reviewers assess only the technical
merit, technical and business issues are often intertwined. It is
important that your business sections be written in a form that can be
understood by technical experts. While business reviewers assess only the
business and economic merit, they must interpret these in the context of
the technology being developed. It is important that your technical
sections be written in a form that business reviewers can follow. Your
interpretation of the state-of-the-art and the state-of-the-practice will
be helpful to all reviewers. You should not assume that either business or
technical reviewers are familiar with the idiosyncratic characteristics of
your target markets or with the unique business implications of software
and information processing products. Explain your assumptions and
approaches.
The closer your proposal follows the outline provided by all the published
criteria the more likely that reviewers will not miss important points. It
is not necessary to repeat information that is relevant to multiple
criteria, but do not assume that reviewers will recognize the relevance;
instead cross reference those sections. A common mistake made in many
abbreviated proposals was to substitute personnel and corporate
qualifications for substantive discussion of the issues when addressing one
or more of the technical criteria. The remainder of this note identifies
specific weaknesses in abbreviated proposals in the hope they will not be
repeated in full proposals.
Weaknesses in Specific Abbreviated Proposals
Basic requirements. Most abbreviated proposals were within the basic
requirements for eligibility, format and content. None were rejected
because they were outside the technical scope. There were, however, two
proposals that arrived a day late, a single applicant proposal submitted by
a university (Note: only for-profit companies are eligible for direct ATP
funding as single applicants) and a joint-venture that proposed to provide
less than 50 percent funding. A common problem with abbreviated proposals
from joint-ventures was that, because the roles of participants were not
yet assured, proposals could not confidently address all of the criteria.
Such joint-venture "structural" problems may remain to a lesser degree for
full proposals. In such cases, the degree of uncertainty should be
identified, but the proposal should address all criteria as if the desired
agreements were in place. ATP is most interested in proposers ability to
develop credible strategies, plans, and organizational structures that will
ensure long-term success.
Quality and innovation. Many submissions failed to identify their
innovation and characterize the uniqueness of their proposed research.
Some proposals involved routine developments without pretense of
innovation. Others contained marvelous visions, but did not identify
technical barriers to be overcome or the innovations required to enable
that vision. Projects with new-to-context as well as new-to-world
innovations were given positive recommendations. Several proposals failed
to address current industry practices or to compare and contrast their
approaches with competitive approaches.
Technical risk and feasibility. At least one submission proposed to
reimplement their existing product in a new language dialect. ATP supports
only high risk technology developments. If a technology is near
commercialization or involves only routine development it cannot be funded
by ATP. Many proposals failed to identify the technical barriers that
hinder commercialization. It is impossible for proposer or reviewer to
demonstrate or assess risk and feasibility without defining the technical
barriers and how the technical risks will be managed throughout the
research project. Feasibility should be discussed relative to the state of
the art.
Clarity of vision and coherency of technical plan. Vague textbook
outlines of technical plans are not convincing evidence of vision and
coherency. Neither are finely detailed task and schedule structures
without indication of the purpose and interdependencies of various task.
Many abbreviated proposals failed this criteria because the proposed tasks
were not tied to the technical barriers and project goals. Several
appeared to depend on magic solutions in which the critical innovation of
the project would not be addressed anywhere in the technical plan. Plans
which received the highest ratings presented a clear vision of technical
objectives and related each task to those objectives.
Adequacy of systems-integration and multi-disciplinary planning. One
company with highly qualified personnel in other technical areas and in
marketing proposed a technically infeasible approach because they
apparently did not involve anyone with software experience in the proposal
process. Several of the abbreviated proposals given negative
recommendations indicated either by assertion or omission that there would
not be any systems integration issues in their projects, an unlikely
situation in software development where integration and interoperability
are typically both difficult and essential. Software is inherently
inter-disciplinary, has little purpose in isolation from applications and
its influence is frequently indirect involving requirements from a
diversity of domains. Meaningful technical requirements can seldom be
established without consulting potential users of the technology.
Potential broad impact on U.S. technology and knowledge base.
Differentiating factors in ranking proposals relative to this criteria
include whether the proposed technology and innovations are enabling, are
infrastructural, are captured in tacit or codified form , and are
complementary to the proposed technical and business strategies. Many of
the proposals did not adequately address this criteria. If the project is
important to achieving certain technical goals of the Component-Based
Software Program, the connection should be identified and explained.
Potential broad-based economic benefit. ATP measures its success, not on
technical achievement alone, but on contribution to U.S. economic growth
and competitiveness. The potential economic benefit should be estimated
quantitatively in terms of jobs and growth potential in clearly defined
markets addressing identified needs. The size and scope of the target
markets and the rationale for choosing them should also be given.
Proposals should address both the direct benefits of products and services
that the proposing organization will produce and the indirect benefits that
derive from their use or accessibility by others. The latter is important
in establishing the potential magnitude of the benefit, but will lack
credibility if the former is not believably established.
Adequacy of plans for eventual commercialization. One proposer stated
that, if necessary, issues of commercialization would be considered at
later time. There must be a clearly identified path for realization of the
economic potential. While detailed marketing plans are not expected for
products that are unlikely be commercialized for a number of years, every
proposal should have a vision of the most likely products, what markets
would be addressed, in what order, and why. There should be a credible, if
tentative, strategy for funding product development and commercialization
subsequent to completion of the ATP project.
Proposers' level of commitment and organizational structure. Projects
cannot be funded without evidence of follow-on commitment to product
development, commercialization, and business success. Lack of commitment
was evidenced in a large company's abbreviated proposal by failure to
involve the company's business and marketing decision makers in the
proposal process. Companies whose primary customers are government
agencies may find it difficult to satisfy this criteria without a
commercially oriented partner. Matching funds are an early indicator of
commitment with greater amounts and percentages expected from larger
organizations. Even stronger indicators, however, are the priority given
to the project within the proposing organizations, the commitment of
decision makers within those organizations, and the consequences of project
success or failure to the proposers. As with other ATP criteria,
assertions of commitment or intent are necessary but not sufficient. A
strong case requires convincing evidence and rationale for those
commitments and intents.
Experience and qualifications of proposing organization. In addition to
the technical experience and qualifications called for in the published
criteria, proposers should emphasize the business and marketing
qualifications of their proposal team. When confronted with business and
economic claims that are not otherwise adequately substantiated, business
reviewers tend to look to the business and marketing experience of the
proposers in assessing those claims.
Proposal length. The best full proposals are expected to be far smaller
than the number of pages allowed. With the possible exception of
joint-ventures, proposals of more than 25-30 pages seldom add value and
often dilute or obscure other strengths.
Full Proposal Submissions
We encourage you to take the results of this analysis of the abbreviated
proposals into account in preparing your full proposals, regardless of
whether you submitted an abbreviated proposal. Full proposals are due at
ATP before 3:00 p.m., on August 4, 1994. Please do not call or write ATP
seeking advice on specific proposals. We will, of course, respond to
questions regarding our criteria and requirements for full proposals.
------------------------------------------------
For general information about For additional technical information on
ATP or to be placed on the ATP Component-Based Software Competition
mailing list contact: 94-06 contact:
e-mail: atp@micf.nist.gov e-mail: dfisher@nist.gov
voice: 1-800-ATP-FUND voice: 301-975-3649
fax: 301-926-9524 fax: 301-926-9524
----- End Included Message -----