{"id":317,"date":"2020-03-01T04:43:38","date_gmt":"2020-03-01T04:43:38","guid":{"rendered":"https:\/\/maybury.ca\/the-reformed-physicist\/?p=317"},"modified":"2020-03-01T21:43:04","modified_gmt":"2020-03-01T21:43:04","slug":"a-paper-to-read-by-gelman-and-shalizi","status":"publish","type":"post","link":"https:\/\/maybury.ca\/the-reformed-physicist\/2020\/03\/01\/a-paper-to-read-by-gelman-and-shalizi\/","title":{"rendered":"A paper to read by Gelman and Shalizi: Philosophy and the practice of Bayesian statistics"},"content":{"rendered":"\n

The great 20th<\/sup> century physicist Richard Feynman<\/a> supposedly quipped \u201cPhilosophy of science is about as useful to scientists as ornithology is to birds.\u201d As always, Feynman has a point, but in the fields of statistics, machine learning, and data science, understanding at least some of the philosophy behind techniques can prevent an awful lot of silliness and generate better results.<\/p>\n\n\n\n

\"\"
Feynman: You philosophers!<\/figcaption><\/figure><\/div>\n\n\n\n

In their paper, Philosophy and the practice of Bayesian statistics<\/a>, (British Journal of Mathematical and Statistical Psychology 2013, 66, 8-38) Andrew Gelman and Cosma Shalizi offer a thoughtful piece on what is really going on \u2013 or what really should be going on \u2013 in Bayesian inference. This paper is a short, highly interesting read, and I strongly suggest that all data scientists in the federal government put it on their reading lists.<\/p>\n\n\n\n

For the uninitiated, statistical inference falls into two broad schools. The first, often called \u201cclassical statistics\u201d, follows Neyman-Pearson hypothesis tests, Neyman\u2019s confidence intervals, and Fisher\u2019s p-values. Statistical inference rests on maximizing the likelihood function, leading to parameter estimates with standard errors. This school of statistics is usually the first one people encounter in introductory courses. The second school \u2013 Bayesian statistical inference \u2013 starts with a prior distribution over the parameter space and uses data to transform the prior into a posterior distribution. The philosophies behind each school are often said to be deductive in the classical case, and inductive in the Bayesian one. The classical school follows a method that leads to rejection or falsification of a hypothesis while the Bayesian school follows an inductive \u201clearning\u201d procedure with beliefs that rise and fall with posterior probabilities. Basically, if it\u2019s not in the posterior, the Bayesian says it’s irrelevant. The Bayesian philosophy has always made me feel a bit uncomfortable. Bayesian methods are not the issue, I use them all the time, it\u2019s the interpretation of pure inductive learning that has always bothered me. To me, I\u2019ve felt that in the end the the prior-to-posterior procedure is actually a form of deductive reasoning but with regularization over the model space. <\/p>\n\n\n\n

Gelman and Shalizi\ngo right to the heart of this issue claiming that \u201cthis received\nview [pure inductive learning] of Bayesian inference is wrong.\u201d In\nparticular, the authors address the question: What if the \u201ctrue\u201d\nmodel does not belong to any prior or collection of priors, which is\nalways the case in the social sciences? In operations research and\nanything connected to the social sciences, all models are false; we\nalways start with an approximation that we ultimately know is wrong,\nbut useful. Gelman and Shalizi provide a wonderful discussion about\nwhat happens with Bayesian inference in which the \u201ctrue\u201d model\ndoes not form part of the prior, a situation they label as the\n\u201cBayesian principal-agent problem\u201d. \n<\/p>\n\n\n\n

In the end, Gelman and Shalizi emphasize the need for model testing and checking, through new data or simulations. They demand that practical statisticians interrogate their models, pushing them to the breaking point and discovering what ingredients can make the models stronger. We need to carefully examine how typical or extreme our data are relative to what our models predict. The authors highlight the need for graphical and visual checks in comparisons of the data to simulations. This model checking step applies equally to Bayesian model building and thus in that sense both schools of statistics are hypothetico-deductive in their reasoning. In fact, the real power behind Bayesian inference lies in its deductive ability over lots of inferences. The authors essentially advocate the model building approach of George Box and hold to a largely Popperian philosophy.<\/p>\n\n\n\n

Finally, Gelman and\nShalizi caution us that viewing Bayesian statistics as subjective\ninductive inference can lead us to complacency in picking and\naveraging over models rather than trying to break our models and push\nthem to the limit. \n<\/p>\n\n\n\n

While Feynman might have disparaged the philosopher, he was a bit of a philosopher himself from time to time. In an address to the Caltech YMCA Lunch Forum on May 2, 1956<\/a>, he said:<\/p>\n\n\n\n

\u201cThat is, if we investigate further, we find that the statements of science are not of what is true and what is not true, but statements of what is known to different degrees of certainty: “It is very much more likely that so and so is true than that it is not true;” or “such and such is almost certain but there is still a little bit of doubt;” or \u2013 at the other extreme \u2013 “well, we really don’t know.” Every one of the concepts of science is on a scale graduated somewhere between, but at neither end of, absolute falsity or absolute truth.<\/em><\/p>\n\n\n\n

It is necessary, I believe, to accept this idea, not only for science, but also for other things; it is of great value to acknowledge ignorance. It is a fact that when we make decisions in our life we don’t necessarily know that we are making them correctly; we only think that we are doing the best we can \u2013 and that is what we should do.<\/em>\u201d <\/p>\n\n\n\n

I think Feynman would have been very much in favour of Gelman\u2019s and Shalizi\u2019s approach \u2013 how else can we learn from our mistakes?<\/p>\n","protected":false},"excerpt":{"rendered":"

The great 20th century physicist Richard Feynman supposedly quipped \u201cPhilosophy of science is about as useful to scientists as ornithology is to birds.\u201d As always, Feynman has a point, but in the fields of statistics, machine learning, and data science, understanding at least some of the philosophy behind techniques can prevent an awful lot of … Continue reading “A paper to read by Gelman and Shalizi: Philosophy and the practice of Bayesian statistics”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-317","post","type-post","status-publish","format-standard","hentry","category-commentary"],"_links":{"self":[{"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/posts\/317","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/comments?post=317"}],"version-history":[{"count":10,"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/posts\/317\/revisions"}],"predecessor-version":[{"id":328,"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/posts\/317\/revisions\/328"}],"wp:attachment":[{"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/media?parent=317"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/categories?post=317"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/maybury.ca\/the-reformed-physicist\/wp-json\/wp\/v2\/tags?post=317"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}