.

What are different stakeholders and their needs and motivations?

Forecasters : improved processes; better reputation; more support from data

Why don’t before/after studies happen now?

• funding that could take away from project budget
• self-interest [ don’t want to look bad ]
• once a project is “done”, there is little interest in re-open it

What can Zephyr do to help? What goals/specific outcomes should this initiative have?

Previous work on before/after studies conducted by FTA showed that almost all models have flaws, are inaccurate, and unreliable. However, this finding must be weighed against the lack of data for validation.

Objectives

• Improve forecast accuracy by advance the state of the art by understanding what is happening today
• Provide accountability
• Estimation of project benefit

Develop “best practices”

Develop a set of best practices for before/after studies and the ongoing programs that support them. There are likely already some good examples here [ i.e. the World Bank? ] Types of practices could include:

• data collection [ ongoing, and a start/end of project ]
• data standards
• sensitivity tests for inputs
• data and analysis tool storage
• performance metrics
• appropriate timeframe for evaluation and period of performance of facility
• features of the evaluator
• costs and funding strategies

Most likely, best practices will differ based on the type of study or needs placed on specific forecasts.

Consideration should also be given to the variety of forecasts that are often produced behind the scenes to develop the project. These often involve multiple sensitivity tests of potential scenarios and are not always published since most required documentation requests a single point forecast or a specified set of inputs that may not be reasonable.

Types of performance measures:

• The original scope
• Predicted and eventual capital cost
• Transit service level
• Project ridership

Conflicts of interest should be appropriately considered. For this reason, academics or other third parties may be the best “evaluators”

Execute specific before/after studies

Could be used to test the best practices.

Evaluate a specific “shovel-ready project” like a managed facility.

What priority/urgency should this initiative have compared to others for Zephyr?

There are probably a lot of potential funders and interested parties that could be willing to pay for such an effort, which should push this up the priority list.

However, these studes are not inexpensive and it could be difficult to motivate an agency that has already “closed its books” on a specific project to pay for something additional.