Tool Structure
Step 1: Screen Candidate Projects
- Screen #1: Does this project or technology meet the primary goal?
- Screen #2: Can this project or technology achieve the targeted benefits within the required timeframe?
Step 2: Develop Weighted Scoring Criteria
Identify Key Criteria that will determine the relative ranking of candidate projects or technologies and apply weights to the various criteria. See example criteria below.
Step 3: Score and Rank Candidate Projects
Organize candidate projects and technologies by "Type" and use the Weighted Scoring Criteria to score each candidate project or technology (In this example the projected timeframe to achieve benefits was used to organize porjects).
Step 4: Review Ranked List of Candidate Projects and Technologies
Review list in order of highest to lowest scores.
- If ranked list appears reasonable, proceed to development of Implementation Plan.
- If ranked list does not appear reasonable, adjust Weighted Scoring Criteria and re-score.
(Note: An iterative process is often needed to calibrate a new scoring tool.)
Step 5: Develop Implementation Plan
Develop plan for implementing projects and technologies that appear to provide high potential benefits within the organization's investment and risk management policies.
Customizing Weighted Criteria
The examples provided are illustrations only. The specific criteria used and their relative weights should be customized to each individual or organization’s specific goals and objectives for an activity of this kind.
The sample scoresheet provided on the next page seeks to estimate values for two factors:
- Benefits attempt to evaluate the ability of any particular project or technology to support or advance California’s resource, economic and environmental priorities.
- Adoption Challenges attempt to understand the types of challenges that targeted adopters will need to address in order to successfully implement the proposed project or technology.
There is no limit to the number of criteria that can be included, although the scoring process becomes much more difficult with the number of criteria.
Similarly, weights can be applied at multiple levels; but again, too much complexity may thwart one of the objectives for this type of process, which is to ensure transparency and objectivity.
Further, there may be more than one scoresheet – for example, different types of technologies with very different purposes may very well have different criteria (for example, technologies that address drought resilience vs. electric reliability).
Especially if this is the first time that this score sheet is being used, calibration is usually needed.
After the first round of scoring (a sample can be used to calibrate the criteria and weights), the portfolio team reviews the relative ranking of various projects and technologies to determine whether the goals and objectives of the scoring process were met. If the purpose was to develop a portfolio of technologies that have high potential to advance drought resilience, a scoresheet that ranks AWG higher than F&B should be scrutinized and revised as needed to assure the appropriate result.
The process of collectively identifying and ascribing weights to evaluation criteria helps to levelize participants’ knowledge and understanding of the portfolio’s goals and objectives, and the portfolio developer’s needs and interests. If the tool does not seem to be producing a rational outcome, change the criteria and weight and re-score some of the candidate technologies. This is usually an iterative (and sometimes frustrating) process; but with each iteration, new insights are gained as to the types of criteria that should be included and their relative importance to the evaluation process.
A sample scoresheet designed to rank candidate drought resilient technologies has been provided below, along with illustrative scoring of two technologies with very different characteristics:
Scoring Example 1: Atmospheric Water Geneartor
AWGs condense water from humidity in ambient air; combined with ultrafiltration (reverse osmosis), AWGs produce high quality emergency potable drinking water on-site. Download the scoring example below.
Scoring Example 2: Food & Beverage Water Recycling/Reuse
Several technologies are available today that enable cost-effective on-site recycle/reuse of F&B process water. One technology uses proven physical-chemical processes to efficiently remove solids from F&B process water effluent. Download the scoring example below.