Interpreting Success Stories and Studies
Do you ever read in the newspaper or news magazine or hear stories on the TV news about a new study that's been done that "seems to suggest" that a certain nutrient or eating a certain food "may" reduce the risk of a certain kind of disease? I'm sure you have, because they're all over the news. It seems that every week we hear about at least one new study. I remember back about 20 years ago when oat bran was all the rage. It was supposed to reduce cholesterol and protect against heart attack, and everyone was eating oat bran muffins and oatmeal. Then a new "study" came out which suggested that maybe oat bran wasn't the magic potion it had been touted to be....and guess who immediately disputed the new study? Of course it was a major manufacturer of oat products. Well, the oat bran rage died down somewhat, but even today, oats are promoted as "heart-healthy" (you can buy cereal with a picture of a heart-shaped red bowl on the front), and I guess oats actually are healthy when compared to a grease laden sausage biscuit (although my grandmother lived to be 93 and ate sausage and biscuits made with lard...). And even cholesterol itself has sort of faded into the background, and people today are no longer obsessed with cholesterol like they were 15 or 20 years ago. In fact most people would have probably forgotten about cholesterol, if not for the pharmaceutical advertisements on TV encouraging you to "ask your doctor" about a medication that will put your so-called good cholesterol where the pharmaceutical manufacturer tells you it should be. The oat bran and cholesterol thing is just one of many examples--this food is believed to protect against certain kinds of cancers, diets rich in this or that nutrient are believed to protect against certain kinds of heart disease, this vitamin is believed to protect against Alzheimer's, this oil is believed to promote lung health, people who eat diets rich in such-and-such nutrient have less risk of whatever.....there's no end to it.
And it's the same in the horse world. Success stories abound telling about how such and such a method or such and such a product helped a horse with a certain problem. In addition, "studies" come out regularly suggesting that this or that nutrient or this or that method or this or that supplement is effective against a certain disease or problem. People have made careers talking about these things...and careers producing and selling the supplements and methods. Go to any feed and tack store, go to a horse exhibition, or look on the internet, and you'll find a huge array of supplements, vitamins, herbs...anything you can think of. It comes in a variety of forms--supplement powders, pellets, liquids, licking blocks, feed with the stuff already in it. All kinds of claims are made about how those things will help your horse, and sometimes people who do clinics will include some of those "secrets" in their clinics...."Oh, you know, studies suggest that such-and-such a nutrient will help prevent or cure so-and-so." (And then they act like they've just revealed the Secrets of the Universe to you.) If you fed your horse all that stuff, you'd need a warehouse to store it all in, you'd go bankrupt buying it all, and your horse would have to stand around all day eating the stuff as you shoveled it in the trough.
But, how much trust should you put in studies and success stories, and how should you go about interpreting them?
An interesting study was done several years ago about the correlation between AIDS and VCR's. It was found that the number of people diagnosed with the AIDS virus and the number of households with VCR's seemed to be related. Plotted on a graph, the number of households with VCR's seemed to lead the number of people diagnosed with the AIDS virus. What was most interesting, though, was that statistical calculations of causality "proved" from a statistical standpoint that VCR's caused AIDS...that is, statistically speaking, you could show a causal relationship between the number of households with VCR's and the number of people diagnosed with the AIDS virus. If you plotted the number of households with VCR's and the number of people diagnosed with AIDS, as the number of households with VCR's increased, the number of people with the AIDS virus would increase a similar amount later...and this continued for a number of years. Thus, this study "proved" that VCR's caused AIDS, using the same methods that are used every day for studies that "seem to show" this or that.
Of course the authors of the study didn't really think that VCR's caused AIDS. What they were doing is showing how statistics and studies can sometimes lead to wrong conclusions. Obviously, VCR's don't cause AIDS, it just happened by chance that they both came on the scene at about the same time and by chance happened to increase at about the same rate. The flaws of a study correlating VCR's and the AIDS virus are so obvious that no one would take such a study seriously...but think about it...what if the culprit in this study (VCR's) had been something else, something not so obviously eliminated? Might it then have been said that "it," whatever "it" was, caused AIDS? Or at least might it have been said that "studies suggest that so-and-so causes AIDS"?
Probably so. And this is how most all the studies are that get reported. In humans for example, first coffee is bad for you, then it's good for you, then it's bad for you again...all because of what "studies suggest." Such studies rely heavily on statistics. But, remember, statistics are just a tool. They don't represent actual fact. They must be interpreted, and that interpretation is always influenced by the preconceived notions of the interpreter.
For example, a number of years ago a major study was done investigating whether smokeless tobacco causes mouth cancer. This study was done because of the well publicized case of a teenager who developed mouth cancer and just happened to use a well known brand of smokeless tobacco, and his parents blamed the tobacco use for his mouth cancer. A large number of men who used smokeless tobacco were followed for a number of years to see if any developed mouth cancer. None did. However, the study reported that a number of them developed white spots on the inside of their lips that were identified as "pre-cancer." (It's important to realize that while billions of dollars are made by the medical industry in the U. S. each year treating "pre-cancer," in much of Europe and many other countries it is understood for what it actually is--something entirely different from cancer and something that will not turn into cancer.) Anyway, in this study, although not a single man developed mouth cancer, the conclusion, which was widely reported in the media, was that smokeless tobacco causes mouth cancer. Those who interpreted the study had already decided before the study began that they would conclude that smokeless tobacco causes mouth cancer, and they weren't about to let the fact that not a single subject developed cancer hinder them from concluding what they intended to conclude from the beginning...and this in the face of the fact that while approximately 85% of all adult males in Scandanavian countries use smokeless tobacco, the rate of mouth cancer in males in Scandanavian countries is no higher than in countries where smokeless tobacco use is nonexistent. This fact about men in Scandanavian countries, which is so irritating to the anti-tobacco folks, is why a can of smokeless tobacco does not have a warning label which states "This products causes mouth cancer" but instead has a warning which states "This product MAY cause mouth cancer." Well, sure, anything MAY happen. The ceiling in my office MAY fall in on me before I get finished writing this.
There's another thing to consider in addition to improper interpretaton of statistics and the preconceived notions of the study authors, and that is the issue of sample size. I sort of like to gamble, and we've been to Las Vegas a number of times. If you go by the Roulette table, you'll see a lighted sign which posts the results of the last 12 or 15 spins. If you go by the Baccarat table, you'll sometimes see a sign where the results of the last 12 or 15 hands are posted. In Roulette, the chance of hitting red or black is about 50% each, while in Baccarat, the chance of hitting player or bank is about 50% each. So you might expect in 12 spins or 12 hands to see an equal number of both on the board. But that rarely happens. In fact, if you keep records of 20 or 30 spins or hands, you'll see that the results are rarely 50-50. In 20 spins, there might be 5 reds and 15 blacks, or in 30 hands, there might be 20 banks and only 10 players. This doesn't mean the game is fixed or that the wheel is defective or that the cards are somehow stacked, it simply means that the sample size is not great enough for the actual outcome to be equal to the expected outcome. You can see this yourself if you toss a coin. Heads or tails...each is 50% probability. But if you toss that coin ten times, you probably won't get 5 heads and 5 tails...you might get 3 heads and 7 tails. Again, this doesn't mean that the coin is not balanced, it just means the sample size is not great enough. I have dealt myself over 1500 poker hands, just playing around. And although I have dealt myself the hands against 7 other imagined players, which means that my win percentage should be exactly 12.5%, in 1500 hands my actual win percentage is only a little over 11%. Again, it doesn't mean my cards are defective...it only means my sample size is not large enough. The consultation of a statistical text available from any public library will give you an idea of the sample size required relative to the number of different possible outcomes...and it's probably a lot larger than you might imagine. One thing I have learned when dealing with horses, or any subject, is to go OUTSIDE that subject for confirmation of claims made...to a purely independent source. There's just too much personality promotion and too many snake oil salesmen in the horse industry.
Let's talk more about tossing a coin. That's a 50-50 proposition, so you should get an equal number of heads and tails. I just tossed a penny in 4 sets of 10 times each. The first set of 10 tosses yielded 7 tails and 3 heads. The second set yielded 4 tails and 6 heads. The third set yielded 7 tails and 3 heads. The fourth set yielded 3 tails and 7 heads. So, in no case did I get the 50-50 expected result in actuality. In 40 tosses I got 21 tails and 19 heads, which is pretty close to 50-50. So, I'm going to toss 10 more times and see what happens....well, I got 6 tails and 4 heads...still not 50-50. The interesting thing is that my cumulative total now stands at 27 tails and 23 heads, which means that in 50 tosses my cumulative total is now farther away from the expected 50-50 split than it was when I had only done 40 tosses. This COULD indicate that my coin is unbalanced, or it COULD just be normal expected deviation in a sample size of only 50 tosses...the point is, we don't know and can't know with only 50 tosses. 50 tosses seems like a lot, but from a statistical standpoint, it's not.
Think of the polls done before a Presidential election. For all practical purposes, there are only two choices--the Democratic candidate or the Republican candidate. So how many voters do the pollsters survey? 50? 100? No, they usually survey 1000 or more. But even then sometimes they get it wrong, with the actual election results falling outside the calculated margin of error of the polls...and that is having only two possible outcomes and a sample size of 1000 or more.
Let's think a little more about the coin toss. It's a 50-50 chance. There are only two possible outcomes, but even with 50 tosses, which seems like a lot, I didn't get actual results to equal expected results. Now, assume a situation where there are more than just two possible outcomes. Assume there are many, as is the case when "studying" the effectiveness of a supplement. In addition, there will also be a lot of outside variables that can affect the results. I can't control many of these variables, and many of them I am unaware of and can't identify. If 50 tosses of a coin don't yield meaningful results in a simple and straightforward situation, how in the world can 20 or 40 or even 100 "case studies" involve the use of a supplement yield meaningful results? Well, they can't in most cases. In the case of the poker hands mentioned above, there are 8 possible outcomes. I or any of my 7 imagined opponents have an equal chance of winning, with each outcome having a 12.5% probability. Yet in 1500 hands I still don't have actual results that equal expected results! So you can see that in order for these "studies" that "seem to suggest" so and so to have any validity whatsoever, they would have to be done on literally THOUSANDS of horses. Otherwise they are basically meaningless with the results determined by chance.
There's another thing to consider. Some might object and say that in a study on supplement effectiveness, there are really only two outcomes: the horse either gets better or it doesn't. So then they would say that you don't need thousands of trials in order to get meaningful results. Well, let's look at that in conjunction with the poker hands I dealt myself. Using the "horse either gets better or it doesn't" logic, there are really not 8 possible outcomes in the poker example, there are only 2--either I win or I lose. And if you follow that logic, then with only 2 possible outcomes, I should win 50% of the time, not the paltry 11% that I actually did win. Of course in the poker example, it's clear that you can't use the logic of "either you win or you lose," and practically no one would take you seriously if you tried to say there were only two possible outcomes. Unfortunately in the example of a study of a supplement, it's not that clear...and it's easier to lose sight of the fact that there are multiple possible outcomes and multiple outside variables, which means that it's easier to take at face value assertions made from studying an actually insignificant number of horses.
What does all this tell us about horses? Well, suppose you read an article in a magazine which talks about how a certain shoeing method helped a certain horse with a problem. That example is actually meaningless and not only could be but most likely is simply due to chance. It is what I refer to as a "well-chosen example." In such a case, just like the supplement example, there are not just two possible outcomes, there are many, and there is not just one variable, there are many, most of which we can't control or allow for and some of which are unknown to us. This means that in order to actually show that this shoeing procedure has any benefit, one would have to study it on thousands of horses. Study the effects on just 20 or 30 or 40 horses, and the results are meaningless, with chance being the determinant of the results. The sample size is simply not large enough. The results on 30 or 40 or even 100 horses don't tell you a thing. Go to a seminar or read a book about a trimming or shoeing method where a few examples are given about the miracles a personality guru has performed, and the results are actually meaningless, considering that there are 9,000,000 horses in the United States. What would have happened had that method been applied to 100,000 horses? I suspect it would greatly deflate the ego of the personality, not to mention knock him or her off the throne of guru status.
During our last trip to Las Vegas, my wife walked up to a slot machine while I was checking us out of our room, put in a quarter, and won $20.00. Does that mean she is gifted at slot play or that particular machine always pays out? Nope. It means she got lucky. But I guess she could write a book and give seminars about how to play slot machines....I mean, after all, couldn't she make a career out of bragging about how she turned 25 cents into $20.00 in just one spin?....Well, it happens in the horse industry all the time. But chances are that if you walked up to that same slot machine and put in 25 cents, you wouldn't win $20.00, just as chances are that if you use the trimming method or the shoes or the supplement that some guru brags about, the results for your horse won't be as promised....
We need to be very careful when we read about the results of "studies" or when we hear about the well-chosen examples used to "prove" to us that such-and-such method is effective, or that such-and-such a product is effective against something, even when, and maybe especially when, the person who did the study has seemingly impressive credentials. Chances are that the results that are bragged about are purely due to chance, and if applied to a statistically significant number of horses, the outcomes would be nothing to brag about...and on top of that, you have to consider the interpretation put on the statistics, which I guarantee will be favorable to the person selling the product or method in question.
Under some conditions studies can be useful, for example, in the case of white line disease. As discussed in the article on white line disease, a farrier did a study of white line disease to identify exactly what was growing there. He found four common fungus but no bacteria. Now based on the results of that study we can't say that those four common fungus are all that are ever involved in white line disease; that is, we can't say for sure that no other fungus are ever involved or that bacteria are never involved. But what we CAN say is that bacteria and wierd types of fungus are not necessary for white line disease. Look at it another way: If someone tells you, "It snows every day in January," you don't need the weather results of every day in January to prove that statement false...you just need ONE day where it doesn't snow.
Use caution in interpreting studies, and when you read a study, analyze it carefully and analyze the results logically. When you hear a claim made about a supplement or treatment method or shoeing or trimming technique, analyze the whole situation carefully and logically. Never consider a study or a claim in isolation, but always consider it in relation to your own knowledge and experience, and especially keep in mind knowledge and experience from outside the horse world. Also, use common sense. Stop and think.
In the words of the old cliche..."Don't believe everything you hear." I remember years ago when I was first learning German. Someone gave me a high school textbook used in German schools that contained dialogue between a high school student and an adult. In one of the dialogues, the student and the adult were discussing freedom of the press. The student had the idea that freedom of the press was absolute, since the government itself didn't monitor the press. But the adult reminded the student that the press is actually a business, operated for a profit, and freedom of the press has limits in that a particular newspaper, for example, can't just print anything it wants to....because that might offend advertisers. Freedom of the press has limits, in that you have to make your publication attractive to advertisers, because in the final analysis, you aren't publishing a newspaper as a public service, you're running a business and trying to make a profit. I'm reminded of how a major car manufacturer recently pulled millions of dollars of advertising from a particular newspaper because the paper ran a review critical of one of the auto maker's new models. While the paper may have publicly stood by the story, you better believe that they'll be sure something like that doesn't happen again, because a newspaper isn't really in the news business, it's in the advertising business. News is a vehicle to sell advertising. I'm reminded of the nationally syndicated political talk radio host who has said over and over again of his show, "First and foremost, this is a business." But apparently, the real meaning of that is lost on the majority of the devotees.
I think sometimes we look at the world through rose-colored glasses and forget that we live in a free enterprise system where business is operated first and foremost to make a profit. The cardinal rule of living in such a system is "Let the buyer beware." And that is actually an unwritten law which courts in the United States will follow in consumer law.
The horse care world is full of products that are being sold, from actual physical products to methods to things that are little more than promotions of certain personalities, produced and promoted by companies who know that in general horse people have more money than the average person and are willing to spend lots of it on their horses. This is not to say that the horse care world is full of con artists, but it is to say that we need to keep in mind that all these products and methods available for our horses (including the personalities), and which are advertised to be so wonderful, are produced and marketed first of all in order to make a profit.
A few years ago a major horse magazine did a series on whether or not your horse really needs the paste wormers that are so well advertised. If you look in that, or any, horse magazine, you'll see that wormer manufacturers buy a lot of advertising. So...think about it...would you expect such a magazine to conclude that those things are not really necessary...and most of all, would you expect the magazine to conclude that they are actually harmful in the long run?
