Kaustia 2012 readable - Lecture notes 2 PDF

Preview

Summary

From devil’s advocate to crime fighter: confirmation bias and debiasing techniques in prosecutorial decision-making...

Description

The current issue and full text archive of this journal is available at www.emeraldinsight.com/1940-5979.htm

RBF 4,1

Overconfidence and debiasing in the financial industry Markku Kaustia

46

Finance Department, Aalto University, Helsinki, Finland, and

Milla Perttula Accenture, Helsinki, Finland Abstract Purpose – The purpose of this paper is to measure overconfidence amongst finance professionals in domain relevant knowledge, and test for the impact of different debiasing methods. Design/methodology/approach – The approach used was survey field experiments with varying debiasing attempts. Findings – The subjects were overconfident in terms of probability calibration, better-than-average beliefs, and unfounded confidence. Debiasing attempts yielded mixed results. Explicit written warnings reduced better-than-average-type of overconfidence. There was a further strong effect from attending lectures on investor psychology covering relevant examples. In contrast, there was only limited success in reducing miscalibration in probability assessments. Research limitations/implications – Different types of overconfidence are distinct and respond differentially to debiasing. Future research on debiasing professional judgment should concentrate on testing in-depth/personally engaging methods. Practical implications – It is important for bankers to acknowledge the dangers of overconfidence. Correct confidence interval calibration is needed in order to have a sense of the risks involved in different asset allocation policies and trading strategies. Bankers should also be able to help their clients avoid overconfidence. Social implications – Debiasing overconfidence in the finance industry likely carries public benefits. The results imply that this task is not easy, but not impossible either. The authors think further investment in this endeavor is justified. Originality/value – Documenting an important judgment bias among finance professionals and estimating the effects of debiasing. Keywords Overconfidence, Debiasing, Bankers, Financial advisors, Banking, Finance Paper type Research paper

Review of Behavioral Finance Vol. 4 No. 1, 2012 pp. 46-62 r Emerald Group Publishing Limited 1940-5979 DOI 10.1108/19405971211261100

1. Introduction Overconfidence among finance professionals can lead to many kinds of problems. For example, it can contribute to excessive risk-taking through undue reliance on risk management models calibrated with limited data. Dowd et al. (2008) quote the chief financial officer of Goldman Sachs, David Viniar, referring to the losses experienced in one of the bank’s flagship hedge funds as “25-standard deviation moves, several days in a row.” However, as Dowd et al. note, even a single 20-SD move is less likely than accidentally stumbling on one particular atom in the universe. It appears that the experts at Goldman were overconfident, setting their confidence intervals too narrowly. Other potential consequences of overconfidence include too frequent trading (Barber and Odean, 2001), value-destroying corporate transactions (Malmendier and Tate, 2008), and the adoption of harmful financial regulation (Hirshleifer, 2008). The authors thank Veli Kaukomaa for collecting the data for study no. 1 and an anonymous referee for helpful comments.

In this paper we measure the degree of overconfidence among finance professionals in estimation tasks related to assessing own ability and probability calibration. Prior studies have shown that most people are overconfident about their own abilities (Frank, 1935; Taylor and Brown, 1988) and are overoptimistic about their future (Weinstein, 1980). Experts can help people to avoid mistakes, but experts too can be overconfident, overestimating the accuracy of their knowledge (Lichtenstein and Fischhoff, 1977; Lichtenstein et al., 1982; Henrion and Fischhoff, 1986). McKenzie et al. (2008) find that experts do provide more accurate point estimates compared to laymen, but experts also provide tighter confidence bounds. The net effect is a similar level of overconfidence in experts and laymen. However, both experts and laymen expected – mistakenly – that the experts would suffer less from overconfidence. Oskamp (1965) shows that people became increasingly overconfident as they receive more information. Russo and Schoemaker (1992) find that managers are grossly overconfident regarding the accuracy of their knowledge about their own company or field of industry. Lin and Bier (2008) summarize studies on expert overconfidence covering 27 different data sets. Plous (1993, p. 217) goes on to state that “No problem in judgment and decision making is more prevalent and more potentially catastrophic than overconfidence.” Based on these studies we expect to find at least some overconfidence in finance professionals as well. We use expert subjects as well as questions that directly relate to their area of expertise[1]. This is different from most calibration studies that typically ask undergraduate students about values they are unlikely to know much about (e.g. “What is the gestation period of an Asian elephant?”). The set of finance professionals in our experiments consists of bank branch managers (Study 1) and financial advisers (Studies 2 and 3). Our results show that all the groups we study are overconfident on average, and along all measures of overconfidence we use. There is variation over individuals, and not all subjects are overconfident. This variation nevertheless mostly comes from unobserved heterogeneity: subjects’ gender, income, and education are not related to their degree of overconfidence. Evidence regarding experience is mixed: number of years on the job significantly reduces overconfidence among bank branch managers, but not among financial advisers. We then study the effects of attempting to reduce the subjects’ overconfidence prior to the estimation tasks. Our debiasing treatments consist of direct warnings, as well as demonstrations given on a lecture. Earlier research has shown that reducing overconfidence is possible, but not easy (Fischhoff, 1982). Effective debiasing is naturally valuable: Russo and Schoemaker (1992) report that Shell used feedback on past assessments to reduce the overconfidence of its geologists’ forecasts of where to drill for oil, creating significant cost savings. Overconfidence can be reduced, for example, by making the subjects list reasons for why they might be wrong (Koriat et al., 1980; Sanna and Schwarz, 2004). In our debiasing tests we first issue explicit written warnings against setting too wide confidence intervals, but find limited success in reducing miscalibration. We then study the impact of informing subjects about the tendency of people to generally see them as better-than-average. This results in a significant reduction in the perceived ability to pick superior mutual funds. However, we find only a small impact on self-assessed ranking on job-related ability. We then have another group of financial advisers attend a lecture on investor psychology, covering examples of overconfidence. Compared to the written warnings alone, this further reduces the better-than-average effect in both domains (picking superior mutual funds and general job-related ability).

Overconfidence and debiasing

47

RBF 4,1

48

But neither warnings nor lectures reduce miscalibration or unfounded confidence in probability estimates. In addition to research on debiasing expert judgment, this paper connects to prior research in documenting overconfidence among finance professionals. Such results have been found with money managers (Russo and Schoemaker, 1992), in economists’ quarterly forecasts of recession (Braun and Yaniv, 1992), in foreign exchange ¨ nkal et al., 2003), as well as in a stock market predictions by dealers and analysts (O prediction task administered to portfolio managers, analysts, brokers, and investment advisers (To¨rngren and Montgomery, 2004). Using a large data set of stock market predictions, Ben-David et al. (2010) find that top financial executives are severely miscalibrated: realized returns are within the executives’ 80 percent confidence intervals only 33 percent of the time. Kadous et al. (2006) find that professional financial market analysts’ optimism about company earnings is reduced when the analysts are asked to consider reasons why the company managers’ plans could fail. The paper also relates to the literature on judgment and bias among finance professionals more generally. Shefrin (2001) finds that portfolio managers and analysts often expect safe stocks to have better return prospects than risky stocks, and attributes this finding to the representativeness heuristic. Kaustia et al. (2009) show that framing a stock return expectation as required rather than expected returns changes the slope of the risk-return relationship, as seen by financial advisers, from negative to positive. Haigh and List (2005) find that professional traders suffer from myopic loss aversion to a greater degree compared to students. Kaustia et al. (2008) find anchoring effects in financial advisers return expectations. Gilad and Kliger (2008) conduct an experiment showing that financial advisers’ risk preferences are more susceptible to change due to priming compared to those of students. In line with these studies, we document a behavioral bias among finance professionals. This paper is organized as follows. Section 2 shortly describes a classical calibration study with general knowledge questions using student and professional subjects. Section 3 describes the main experiments of the paper and discusses their results. Section 4 briefly concludes. 2. Basic calibration with general knowledge questions We begin with a classical calibration test where the respondents are asked to provide 90 percent confidence intervals. The test consists of five trivia questions with quantitative answers. The questions involve general knowledge items (e.g. “What is the length of the border between Spain and Portugal?”, see Appendix for the complete set) with no particular emphasis on any field. The meaning of confidence intervals is first explained to the subjects. They are advised not to set their limits too narrowly so that the true answer would turn out to be outside of the bounds, but on the other hand not to set them too wide either because the aim is not to be correct with certainty. We collect data from two groups. The first group consists of Bachelor’s students at Aalto University taking their second finance course. All have taken an elementary course in statistics covering the topic of confidence intervals. The second group consists of financial advisers in a large Scandinavian Bank taking part in an internal seminar. We provide no rewards for participating. However, for both groups the experiment takes place during a lecture with everyone participating. The participants know that the results will later be tallied and discussed in class, although individual performance will not be made public. There are 123 students and 23 financial advisers. The subjects

filled in paper questionnaires and we collected the answers. For each subject we then count the cases wherein the true answer lies between the bounds provided by the subject. The mean number of correct answers is 1.54 for students and 1.70 for financial advisers, while the expectation for a perfectly calibrated subject is 4.50. Figure 1 shows the frequency of the number of correct answers. A perfect calibration at the 90 percent level produces a 60 percent chance of having all five answers correct. The chances are o1 percent that a perfectly calibrated subject would have two or fewer correct answers. Yet 78 percent of students and 65 percent of financial advisers obtain such scores. Only two subjects (one student and one financial adviser) are able to get all five correct. These results are comparable to earlier studies on calibration. They add by documenting calibration overconfidence among finance professionals using general knowledge questions (Figure 2).

Overconfidence and debiasing

49

60 % Perfect calibration Bankers (N =23)

50 %

Students (N =123) 40 % 30 % 20 % 10 % 0% 5

80

Baseline

4

3 2 Number of correct answers Warnings only

1

0

Figure 1. Demonstration of 90 percent confidence intervals for five trivia questions

Warnings + lecture

70 60 50 40 30 20 10 0 Better-thanaverage, scale

Better-thanaverage, rank

Interval calibration

Unfounded confidence

Figure 2. Measures of overconfidence in Studies 2 and 3

RBF 4,1

50

3. Debiasing experiments This section presents the results of three experiments designed to detect overconfidence among finance professionals using domain relevant questions, as well as the effect of debiasing in this context. A. Study 1: domain-specific calibration questions for bank branch managers This experiment targets a specific group of finance professionals and tests for the impact of explicit warnings against miscalibration. Subjects are bank branch managers responsible for, among other things, the financial performance and the business development of their branch. They typically have about 20 subordinates. They are also ultimately responsible for corporate credit issued in their branch. Such decisions involve balancing the aspects of risk and return, and prudence in probability judgments is generally valued. Each subject makes dozens of credit decisions for small- and medium-sized enterprises during a typical year. We identified 63 branch managers and asked them to participate in the study. Only three declined, so the final sample consists of 60 subjects. Data collection was done over the phone. Subjects were provided with ten questions with quantitative answers from the domain of their expertise: finance and economics. For example, “What is the turnover of an average Finnish company?,” and “What was the level of the one-month Euribor interest rate on the first trading day of 2007?” (see Appendix for the complete list of questions used). The respondents were randomly allocated into two conditions: baseline and debiasing treatment. The baseline group received just basic instructions about answering confidence interval questions. The debiasing group was warned against overconfidence as follows. They were provided with a hypothetical example involving the assessment of the 90 percent confidence bounds for client profitability. The example mentions that a decision maker who is not too familiar with the client would not want to give too narrow intervals which could easily go wrong. A further warning was issued: it is important to recognize that when making these kinds of estimates, respondents often show overconfidence by setting too narrow confidence intervals. With perfect 90 percent calibration the probability of getting six or fewer out of ten questions to stay within bounds is about 1 percent. The results show that the bankers were grossly miscalibrated: the mean number of correct answers was 3.3, and only 13 percent of the subjects got more than six correct. The mean success rate of 33 percent (3.3/10) with these domain-specific questions is no higher than the corresponding number with general knowledge questions discussed in the previous section where the mean success rate was 34 percent (1.7/5). One must nevertheless bear in mind that these figures come from different groups of subjects. Panel A of Table I shows the results by treatment, i.e. baseline versus debiased group. There are some indications of successful debiasing. The average number of correct answers was 3.03 for the baseline group, and 3.67 for the debiased group. Subjects in the debiasing treatment scored more correct answers in seven out of the ten individual questions. Also all individuals with more than six correct answers come from the debiasing treatment. The increase in the number of correct answers with the debiasing treatment (0.63, t-statistic 1.43) is not statistically significant, however. We also conduct regression analysis to estimate the effect of debiasing. This allows us to use subjects’ work experience, measured in years, as a control variable. Unfortunately, no other information on the subjects was available in this experiment. As before, a mistake is defined as an interval not containing the true answer. Define a

Expected with 90% calibration Panel A True answer within bounds % with six or fewer correct Number of subjects Panel B Dependent variable Debiasing Log experience Constant Question fixed effects Winsoring No. of observations R2/pseudo-R2

9 1.3 (1) Mistake 0.413* 1.83 0.281** 2.12 2.000*** 4.10 Yes No 600 0.14

Baseline

3.03 100 30

Debiasing

3.67 87 30

Difference

Overconfidence and debiasing

0.64(1.43)

(2)

(3) (4) Forecast error 0.373 0.327 0.340 1.61 1.49 1.60 0.308*** 0.322*** 0.314*** 2.87 3.07 3.09 0.968*** 0.999*** 1.034*** 2.91 3.11 3.33 Yes Yes Yes No 99% 95% 600 600 600 0.25 0.28 0.28

51

Notes: Data are based on subjects’ 90 percent confidence intervals for ten domain specific questions. Panel A shows univariate results based on the number of correct answers and a comparison to a perfectly calibrated benchmark. The figure in parentheses represents the t-statistic for the test of the difference in means. Panel B shows regression results. Specification 1 is logit. The dependent variable (mistake) takes the value of 1 when the true answer is outside of the confidence intervals, 0 otherwise. Specifications 2-4 are OLS and the dependent variable represents a forecast error. The forecast error is specified as 0 when the true answer is within the given bounds, and as abs((x – true answer)/true answer) where x is either the lower bound or the higher bound given by the subject, whichever is closer to the true value. We then add a constant of 0.01 to this variable and apply a log transformation. Columns 3 and 4 apply winsoring at the high end of the dependent variable. Debiasing is a dummy variable receiving the value of one for a subject in the debiased group, 0 otherwise. Log Experience is the natural logarithm of banking experience in years. Data from ten questions for each subject are pooled together. Standard errors are clustered at the subject level and are robust to heteroskedasticity. Table I. All specifications include question fixed effects. *,**,***statistical significance at the 10, 5, and 1 Bank branch managers, percent levels, respectively study 1

variable yij as taking the value of 1 when subject i makes a mistake in question j, and 0 otherwise. We run the following logit regression:   pij ln ¼ a þ gDi þ b 0 Xi þ uj þ eij ð1Þ 1  pij where pij is the probability that yij ¼ 1 conditional on the explanatory variables, a is a regression constant, Di is an indicator variable for subject i being in the debiasing treatment, Xi is a vector of subject-specific control variables (in this study just one variable, log of work experience), jurepresents question fixed effects, and eij is the error term. The regression pools all answers (ten per participant) together, and we use clustering-corrected standard errors that are robust to correlation between the answers provided by the same individual. Panel B of Table I, Column 1, shows the results. Debiasing reduces mistakes, and now the effect is statistically significant, albeit only at

RBF 4,1

52

the 10 percent level. The length of work experience also reduces the rate of mistakes, and the effect is statistically significant. McKenzie et al. (2008) find that experts and laymen show similar levels of overconfidence. Our results on the other hand suggest that higher levels of expertise among a group of experts can reduce overconfidence[2]. This is also consistent with earlier studies finding that experience ameliorates behavi...