The following comments were prepared for the June 2010 meeting of the Pacific Fishery Management Council. The comments
concern the way the Council deals with uncertainty and how they decide on allowed catch. They are posted here so interested parties can read and comment.
Ladies and Gentleman of the Council,
We have a problem. It manifests itself in the Supplemental Report of the Science and Statistical Committee, “An approach to Quantifying Scientific Uncertainty in West Coast Stock Assessments.” I asked a distinguished colleague to review this document. He is unfamiliar with the Council but he is well qualified to speak to the issue of “Best Available Science.”
Here is what he said, “The idea that we will ignore some key uncertainties and then adjust the decision for that oversight is silly. It is usually more important to understand (and characterize) what you don’t know than it is to refine an analysis of the things you do know. Statisticians tend to focus on the latter because that is where the data are.”
Actually, I think we have two problems. The first is a problem of inappropriate framing. The second is a misuse of probability theory.
Inappropriate framing is the root cause of most bad decisions in any field. People often refine the answer to the wrong problem. The Council has framed the allowable catch problem as a fishery science problem. It is a management science problem.
The Council is charged with making an important decision on behalf of the American tax payer. The decision is how many fish should we allow fishers to catch? The applicable best available science for making this decision should be management science, supported by the best available biological and physical science. It is not biological and physical science alone.
Management science begins with the decision, not with the data. Decisions require us to look forward. When we make decisions we usually don’t have adequate data about the future. It is not appropriate to base these decisions entirely on statistical data. The right approach is to begin by clearly framing the decisions to be made and clarifying the objectives. Then we can start modeling and collecting information. The modeling and the information collection are guided by what is important for improving the decisions. For example, we expect that marine protected areas and ocean zoning will affect our estimates of fish population dynamics. Decisions about marine protected areas and ocean zoning should not be treated as an afterthought.
Information and modeling are expensive so we need to gather and analyze information efficiently. In the case of the Council, the cost includes not only the cost of the SSC, but also the cost of the time of all the people who are involved. Not to mention the cost of bad decision making.
The second problem is with the SSC’s use of probability theory. Management scientists rarely rely on raw statistics. Instead they rely on informed judgment, guided by all the relevant information that is available. It’s called the Bayesian approach. The Bayesian approach is normative. It is what we should do, not a description of what we usually do. The SSC appears to regard Bayes rule as optional, a choice for them to make. It is not a choice, it is a fundamental law of the calculus of probabilities. We don’t ignore the laws of physics when we don’t understand them or they are difficult to use. It means we have to make the decision based on the best available information, judgment, insights from models, data, experts, etc. Whether we have historical data is not the issue.
Uncertainties are represented by probabilities. According to the laws of probability theory there are strict rules for updating our information as we learn more, i.e., Bayes rule. Management science has developed ways of determining the economic value of new information. They rely on the use of Bayes rule. Fishery science using classical statistics has no way of placing an economic value on information. As near as I can ascertain, none of this science is being applied by the SSC.
The SSC is constrained by what they know, classical statistics. Classical statistics is good for testing scientific theories. It is only marginally useful for making strategic resource decisions. The perspective of the SSC appears to be, If they are uncertain about something (usually because they have no data) then it doesn’t exist. But then when they are finished analyzing they change their philosophy radically and arbitrarily start assigning probabilities. The methodology they use for assessing probabilities is definitely not best available science. The SSC paper is clear evidence of why classical statistics is not best available science in this situation. The big idea is summarized in the question: Is it better to be precisely wrong or approximately right? The SSC and the Council are acting as though it is better to be precisely wrong.
Management science is about process as well as tools. The modeling process should begin with simple, transparent models. The addition of more complexity is guided by sensitivity analysis and the needs of the decision makers. We add complexity if it is going to improve our ability to make decisions. We don’t add more detail purely for the purpose of increasing precision.
The models that the SSC uses were not built using any management science discipline. Consequently we now have models that are metaphorically like white elephants. They are big, unwieldy, and they have big appetites for expensive data. Furthermore they are not transparent. The SSC appears to have lost track of why the models were created: to inform decision makers and stakeholders.
I could go on about what is wrong with the existing system. The root cause is that best available science is not being applied. The best available science is management science, supported by the best available fishery science.
On June 8, 2008, I testified before this body while I was a member of the Groundfish Advisory Panel. My testimony follows:
“I recommend that the Council develop a normative framework for making total allowable catch, stock assessment, and information collection decisions. The framework should include the costs and benefits of raising or lowering catch limits (preferably expressed in dollars. ) The framework should also include the uncertainties in fish stock estimates. In developing this approach NMFS should rely on the extensive literature and experience related to the science and engineering of decisions under uncertainty.
Such a framework would improve the Council’s decisions and would provide more defensible arguments. As a welcome side benefit it would prevent many unproductive discussions about the precautionary approach. A normative quantitative framework would enable us to talk in a constructive way about how much precaution is appropriate in each situation.
Anyone interested in learning more about normative decision making should consult the vast literature on decision analysis or consult Steve Barrager, GAP Conservation Seat.
Development of this normative framework should have a high priority. It is not currently in the Research and Data plan.”
What has the Council done to address this issue in the last 2 years? Where do we go from here? Innovation is not going to come from doing more of the same. As Albert Einstein said, “Insanity is doing the same thing over and over and expecting a different result.” I think we are going to have to take a new approach.
In summary, the objectives of all of all our analytical efforts are understanding, learning and efficiency. Does the current way of doing things help the Council understand the important issues and how they relate to decision making? Are we learning how to do things better or are we stuck in the same old pointless debates and political thumb wrestling? Are we getting value for our money?