Public Health 250a - Lecture 24

Art Reingold: Okay. So, first of all in case people have lost track, again a reminder next Wednesday is the midterm and you'll need a calculator. This morning we're going to have a presentation that often we have a little later in the semester. It's a critically important presentation. Professor Steinmaus has been here before talking about cross sectional studies and historical cohort studies. One of the things that I think we'll be addressing in a little more detail later in the semester is once we have a body of evidence concerning a particular exposure outcome relationship, smoking and lung cancer, whatever it is we need to think about how to gather that evidence together in some sort of systematic way and figure out what we think it all means taken collectively. And generally there are multiple studies about a particular subject and how do we go about bringing that information together. One approach in the old days was what might be called systematic reviews. Another approach that has a lot of traction these days is what's called meta-analysis and Professor Steinmaus is certainly one of our local experts on meta-analysis. In fact he and Professor Smith give a course each fall on meta-analysis and systematic reviews and if you are interested in this topic next fall you may want to investigate taking that course. It's really quite valuable for those of you who are going to be working in public health. In any event Craig is just going to give you a taste this morning about what meta-analysis is all about when we have multiple studies of the same exposure outcome relationship. Craig Steinmaus: Okay. Can we start? Can you hear me in the back? All right. So yeah we'll talk about meta-analysis today. Unless anybody wants to talk about somebody else? No? Meta-analysis. Okay. If you're asking for it we'll talk about it. All right. So yeah, what Art said. Sometimes on a particular topic that you're interested you have a whole bunch of studies and you want to somehow summarize those studies so meta-analysis is one way of summarizing those studies. Taking a whole bunch of studies and coming up with an overall summary. Why do you need to know about meta-analysis? Number one there's a lot of them out there. On any particular topic you'll do your Pub Med search and you'll come across meta-analysis. When I started doing this when I think I went on Pub Med this is ten years ago there weren't that many. Over time it seems to grow and grow and grow. This is 2011. 51000 when you type meta-analysis. I'm sure today it's close to 60 or 70000. Meta-analysis is a whole bunch of different topics. Occupational medicine, environmental medicine, infectious diseases, clinical trials, genetics. Cohort studies, case control studies and you combine studies that present data as relative risk or difference between means or regression coefficients. It incorporates a whole bunch of different types of studies where you can summarize those literature or that literature. Okay. So first question is what are some of the general ways we have summarized literature in the past? I'm going to give you an example. The guy at the office next to me is a pediatrician. His name is Mark miller. His boss came up to him and said Mark, does environmental tobacco smoke or secondhand tobacco smoke does that increase the risk of breast cancer. My office at Cal EPA wanted to know that. We wanted to know if we should push regulations to limit secondhand tobacco smoke. Mark had to take a large body of literature on this topic and summarize it to come up with, yes it does seem to cause it, no it does not seem to cause it. I don't know is not a great answer. Right? Because I don't know means well, you don't do anything about it. Right? In epidemiology it's an imperfect science. There's a lot of times where we can't really prove this causes this. We don't have absolute 100 percent proof. So what we try to do is come up with well, what's the general consensus? Is there enough that we should probably be safe and go ahead and regulate something like environmental tobacco smoke. So Mark was asked that question. He went through the usual steps of summarizing the literature research. Find the research question. Do your literature search to look for all the literature you can find on this particular topic. Evaluate the studies and that's what you are learning in this class to evaluate the studies. Look at bias, confounding, consistency, causal inference, look at all that different stuff. And then take all that stuff that you just did and sort of somehow summarize it. So that's what we're going to talk about today. Now one thing I did want to point out that keeps coming up in the meta-analysis class is the literature search. I always get asked what's the best way to get studies where should we get studies? If I was going to give you a one word answer, Pub Med is the greatest. I think we've published 12, 13 meta-analysis and I hate to admit this but every single study that we've ever found that we've ever put into a meta-analysis are all on Pub Med. Saying that you should always double check any work you do in epidemiology. Always double check it. So we always look at another. Try to look at some other database. So what I did was I actually went on these major environmental journals and looked up meta-analysis and looked how they found their articles they ended up selecting for their meta-analysis. Every single article did use Pub Med. You can see some other ones. Web of science. I'm not an expert in those. That's where people generally seem to do their literature searches in published meta-analysis. Again, we've defined our research question. We've gotten, we've done our literature search. We've gotten all the studies on our particular topic and then again the next step, you want to evaluate each study. Is there major bias is there confounding? What's the probabilities due to chance? We're on semesters right? You are, that's a big giant job right there. Evaluating a study. That's not what I'm talking about today. That's all your other classes. Evaluating each study and deciding what are the good studies and what are the studies that aren't good? What are the studies going to give you information on your question and what aren't? What have fatal flaws. Significant bias, significant confounding. We can push those to the side. And then when we're doing this whole process we do want to set up inclusion and exclusion criteria on what types of studies we're going to include. That's based on knowledge of your topic and what the important biases are that could affect the studies on your topic and the important confounding variables that could affect the studies on your topic. You want to set up really strict and formal inclusion exclusion criteria. I'm going to include these types of studies. Now I can't give you a whole lot of information on this because it varies from topic to topic to topic. But the bottom line is you want to have a good reason for everything that you do if you are going to exclude a study you want to have good reason for it and you want to avoid vague criteria. You don't want to say I'm only going to include good studies. You want to be specific. This is my first meta-analysis where we looked at the effect of beta carotene and whether it decreased bladder cancer. You can see our inclusion criteria. We weren't going to include cross sectional studies or ecological studies but we had a reason for each one of those. In the meta-analysis process you'll do the same thing. And again it's all dependent on what your topic is. Maybe some topics, cross sectional studies are okay, some topic they aren't. Strict formal exclusion inclusion criteria. Again, that's based on your topic and evaluation of bias, looking at important confounders. Eventually you'll reach the point where you'll get what you have decided are the good studies on your topic. The studies that give you information you need or want and there's no fatal flaw in terms of significant or substantial bias. Mark came up with these particular studies. He did environmental tobacco smoke and breast cancer in post and premenopausal Wyoming it's thought that with breast cancer there are two different diseases. The things that cause breast cancer before you have menopause are probably different than after menopause. That's why he split it up. Anyway he found all these particular epidemiology studies on this particular topic. Here's the relative risk and confidence intervals. Some cohort studies, some case control studies, CC. We'll talk about this in a few minutes. So those, and he went through that whole process, evaluating was there major confounding, was there major bias in these particular studies and he had evaluated each one of those and decided there wasn't. But these would give him good information on that particular topic. Okay. Once he's got these studies then how does he come up with an overall yes or no. While I'm talking, while I'm blabbing on go ahead and take a look at these numbers here and these numbers here and take my word for it that these did not most likely have major bias, major confounding, but they were pretty good studies and in your mind kind of come up with an overall conclusion if you had this data here knowing they were good studies would you say yes there's an association or no there's not an association. Okay. And that's one way of summarizing literature. It's just kind of looking at the data, mixing everything up in your brain, right? And then out comes an answer. A yes or no. Yes it does seem like there's an association or no there doesn't. Right? But that's not a very quantitative process. Right? That's a subjective review. Let me emphasize that word subjective. Different people's brains my evaluate things differently. You would like to know how somebody's brain did evaluate something. If they took it all in and spit something out, a yes or no that's not really science. We want to quantify things a little bit more than that than these subjective reviews. That's the goal of meta-analysis. There's also expert panels. You don't rely on one guy and his brain to spit out an answer. Maybe you get a whole bunch of experts including people from UC Berkeley. Get a whole bunch of experts to sit together in a room and they go over the data and all of them eventually at the end of the day they bolt. This was on formaldehyde. Is it a carcinogen? They looked at all the literature and they eventually at the end of the day raise their hand for yes or no. That's a little subjective too. It depends who you get in the committee and what their tendencies seem to be. Again, that's somewhat subjective as well. Meta-analysis we want to quantify things a bit more. Another method is boat tallying. Count the number of number of positive and negative studies. If there's more positive studies then yes there's an association. So you could do something like that. This is a lot of times in the past was done based on statistical significance. For each study that is statistically significant in red, found a statistically significant effect, statistically significant means it excludes one, that would be a positive study. Each study that did not find a statistically significant effect that would be in the black. That would be a negative study. If you went through these studies you would find what is it? 7 and 7. Seven positive studies and seven negative studies. It's a wash. No association. That's vote counting. Counting each study based on its statistical significance. I'm sure most of you can see a whole bunch of problems with that. One is you'll have studies like this, the Delfino study they did find almost a threefold increased risk of breast cancer in people exposed to secondhand smoke. That was negative because the confidence interval included one. But the question is, is that really a negative study? They found almost a threefold association. But just because it was small, probably, the confidence interval included one. So, is it really negative or was it just -- or was it positive and just too small? It didn't have enough statistical power or subjects. Maybe if it was a little bigger that confidence interval would have been a little narrower. I think that's a general principle you'll learn in stats. As the studies get bigger the confidence intervals get narrower. Again, is that really a negative study? That's one of the problems with vote tallying. That's one of the problem that meta-analysis tries to deal with. Another thing you could do is take an average result and see if that average result is greater than one. You could take an average of all these. Again, there's some problem with this. Not all studies are created equal. Some studies, for example, are bigger than others. So wouldn't it make sense if a study, we have one study with 10000 people. We have a is very similar study and it only has three people. Wouldn't you want to give greater weight to that study with 10000 people? Probably. I would think. Everything else being equal. So just taking an average you don't really do that. And maybe it would be nice to do that. All right. Another thing is that well, all studies are not created equal. And we could maybe give studies with greater quality more weight. For example, we could say and never say it in my class because I do case control studies. We could say that cohort studies are better than case control studies. We could say that. Right? And we could say, okay, well every case control study we're only going to give half the weight we would give a cohort study. We could take each result, weigh it by how good a quality we think it is and then do a weighted average. A weighted average. But the problem is that, okay, I just said that cohort studies are better than case control studies. Sometimes they are, sometimes they're not. Right? It all depends on the study. You could have the absolutely worst terrible cohort study that was ever done. Tons of bias. Only one subject. Absolutely terrible and you're going to still give it two times as much weight as the world's greatest case control study. Right? So, the sort of blanket statements about study quality, it's difficult to support those statements. So it's difficult to weight things on study quality. Based on factors like this. Case control versus a cohort. Then maybe you really do believe case control studies are better than cohort studies? Why not 1.8? Why two? That's why most of us that do meta-analysis we don't use these quality scores like this based on these factors. Where do you come up with the number? A lot of times there's so many exceptions to these sorts of ratings that they are not usable. What we do in most meta-analysis is we weight of study size. And what I mean by study size is precision. I'll talk about precision in a second here. And we don't really weight on study quality except for what we usually do, what most meta-analysis do. We've gone through each of our criteria to rate whether a study gives us the information we want or doesn't. We've assessed is there major bias? Is there major confounding? If there is we do weighting. That weight is 0. If there isn't a fatal flaw we give it a weight of one. We exclude our bad studies. We are weighting but it's 1 and 0. There's other ways of incorporating information or looking at issues of study quality and that's subgroup analysis. I'll talk about those in a few minutes. The bottom line is this is how most meta-analysis is done. We take the studies that are good enough, combine them and weight them on study size or precision. All right. So, all right. We're going to go through the mathematical portion of a meta-analysis. This is all your other classes. Let's go through this right here, this mathematical portion. We have our good studies. And we've selected them. We have our relative risk. We have our confidence intervals. When you do meta-analysis you put everything on the log scale. I don't want to get too into this. It's basically because the log scale is normally distributed. Whereas the relative risk scale is not normally distributed. If you don't get that and you are curious, come ask me. If you don't get it and you are not curious it's not going to affect anything in your life. We'll take it out of log scale later. Everything goes on log scale. All the relative risk, all the confidence intervals we put it on the log scale. Simple. We calculate this BI. Let's call it a coefficient. Basically this is the log of the relative risk. Nice and simple. You can do this in Excel. Or on a calculator or anywhere else. All right. And again take the log of our relative risk and take the logs of our confidence intervals. That's what that is, log, log, log. Nice and simple. All right. And then we want to weight each one of these, each study based on our study size. But what is study size? Is it the number of subjects in the study? Is that study size? There's a problem with that. Where you could run into situations like this. You could have two studies that exactly the same number of people in them. 200000 people. But this study only has one case. Right? Versus this study has a lot more cases. Now can you really, people with the disease, people with the out come. Can you really tell much about what's going on with the disease if you only have one person with that disease in your entire study? Probably not. You are probably going to get a lot more information out of this study. So you can see there's potential problems with just weighting the individual studies based on the number of total subjects. And so, and also did you talk about statistical power in this class? Art Reingold: Later in the semester. Craig Steinmaus: You'll learn about statistical power, how much power your study has to identify an effect. That is really driven not all that much by this, but that's really driven by how many people with your outcome that you have. This is really the driving factor. So maybe we want to weight our studies based on the number of people in our study with the outcome of interest. But there's potential problem with that. We could run into a scenario like this. You know, where we have studies with the same number of people with the disease but we only have one non case here. Again, can you really get a whole lot of information about a disease if you only have one person in the comparison group. Not much. Not much. I think you can see you'd probably want to give this study more weight. We can't really weight based on the number of people with the outcome of interest. Or even the total sample size. There's problems there. So what do we do? It turns out we weight by the variance of the relative risk. When you get the variance of the relative risk that incorporates all that stuff. It incorporates the study size, it incorporates the distribution of the cases and the controls. So the study that gives you the most information that's most precise information will have the lowest variance. Okay? Big studies have low variance. All right? And by big I mean that not only the total study size but that distribution of cases and controls. So we actually do end up weighting by the variance. Or I should say the inverse of the variance. Does that help you? It doesn't help you that much. What's the variance of the relative risk? There's a whole bunch of different equations to calculate the variance. The bottom line is the variance incorporates the study size, big studies have smaller variance and it incorporates that distribution. There's a whole bunch of different equations for variance. This is one. I think this is an estimate of the variance. Look at this equation for a case control study to calculate the variance. The A, B, C, D in your two by two table boxes. What would happen to the variance in A, B, C and D are really big? Right? They are in the denominator. If they are really big the variance is going to drop. You can sort of see how study size is related to the variance. Bigger studies, lower variance. That still doesn't help you, does it? How do you get the variance? It turns out we can get the variance of the relative risk from the standard error. Great. That still doesn't help you. How do you get the standard error? It turns out we can get the standard error from the confidence interval and guess what? We have the confidence interval. We can take the confidence interval, calculate the standard error and then use that, square it to calculate the variance. Look at this equation here. And remember what I said, bigger studies the confidence interval will get narrower. So look what happens when you have a big study. What happens to this quantity right here? It gets smaller. All right? So big studies this quantity will get smaller. And as this gets smaller this gets bigger. As this gets bigger this gets bigger, this gets bigger. Did I say that right? No I didn't say that right. Let's start all over again. Ready? How come nobody raised their hand? You're not listening. As this gets smaller this gets smaller. Right? As this gets smaller this gets smaller. As this gets smaller this gets smaller. For bigger studies. And as the variance gets smaller the weight gets bigger. So we will end up giving greater weight to bigger study. Bigger study, tighter confidence interval. This gets smaller, this gets smaller, this gets smaller, this gets bigger. Bigger studies get greater weight by weighting on the confidence interval. That's what we do in meta-analysis, that's how we weight studies. So we'll do that for each individual study we calculate the standard error using that equation I showed you. We have the confidence interval. We do that, we calculate the standard error for each individual study. And then we use that equation then, calculate the variance and calculate the weight. Nice and simple. Simple equations. Nothing is complicated here. You may ask about that 3.92. It turns out if you look at the equations on how to calculate a confidence interval for rate ratio, odds ratio, you had the 99 percent confidence interval. Do you remember the 1.96 from your stats class. That's the Z-score for the P value of 1.05. That's that double. If you look at the equations you end up doubling that when you calculate the standard error. That's where the 3.92 comes from. You guys don't remember that? Do you remember that Art? 1.96. All right. Okay. Again, look and see what happened. We had the Reynolds study which has a pretty tight confidence interval. It's pretty narrow. It was a big study. I can't remember how many people were in it. It was a big study. Compare that to the Sandler study where the confidence interval is wide. That was a smaller study. We end up giving greater weight to the bigger study. This is greater weight than the Sandler study. Okay. That's what we want it to do. And so what we can do then is take those weights, multiply each weight times each coefficient and then divide that by the sum of the weights. Just add up all the weights, divide that by the sum of the weights and we get an overall summary coefficient. We summarized all these. We calculated a weighted coefficient. This isn't 0. It should be past the decimal point. We summarize -- we calculated basically weighted average, weighted by the precision of each study where we got the precision based on the confidence interval. Okay. Remember what I said we did everything in the log scale. This number doesn't really mean all that much to me or probably to most people. Well, okay, this summary coefficient .32 what does that mean? Let's take it out of the log scale. Put it back on the relative risk scale by taking the exponential. It correlates to a relative risk of 1.37. I think if you go back and look at those data and in your own mind you kind of average those data, look at the confidence interval, see which are the bigger studies. You'll probably come up with a number right about here in your own mind. So, we, this is our overall weighted average, again weighted by the precision of each individual study. Art Reingold: Craig can you go back to the slide with relative risk? Craig Steinmaus: Yeah. In your mind you can tell which are the bigger studies because they have the smaller confidence intervals. This is a big study. Art Reingold: If I were looking at that in my mind I would come up with something bigger. Craig Steinmaus: I probably would too. These are smaller studies. That one did. A lot of these other ones. This 1, 3.6. That didn't get a whole lot of weight. It got 7.4 percent of the total weight. Oops. I'm sorry. Where my percent weight? I didn't put it on here. Look at Moriaba, it didn't get a lot of weight compared to some of the others. This one got a lot of weight and the relative risk was 1.1. That's a good point. I think I've been doing this enough I would have come up with about a 1.37 (laughter). Again, if you haven't maybe you see all the twos. Art Reingold: I look at all the twos and threes and seven. I would come up with a bigger number. Craig Steinmaus: All the more reason to not mix everything up in your brain. Go ahead and quantify this. And then you don't need to know this. This isn't going to be in the midterm these equations. Art Reingold: No. Craig Steinmaus: But you can get a 95 percent confidence interval. It's basically based on the sum of the waits. And these equations here. You have all this information already. It's nothing fancy so you get a confidence interval. We have a 1.37 statistically significant effect. All right. Now you can also calculate a heterogeneity statistic. Have you talked about consistency? You'll have a lecture I believe on causal inference where you'll talk about what are the things you look for to decide whether there's an association. One of the things is do all the studies give you the same result? Do all the studies give you a relative risk of around two or is there a lot of spread in the data? And that's a normal step of meta-analysis is we can do that quantitatively using this heterogeneity statistic. And again, you have this information. You have the coefficient for each individual study. You have the summary coefficient. You have the weights. So this is easy to calculate. And you can even once you get a chi-square in whatever computer program you are using if you are using Excel or STATA you can get a P value for this. You can get statistically significant heterogeneity or not. It does measure how much spread in the data that you have. Are these really far away from this? Are these all over the place? A lot really high or really low? Or are they really close? Is this two and all these about two. You can see how this does give you an indication of how spread out the data are? >>>: Quick question. You may have already mentioned this. Is there a confidence interval associated with the relative risk that you got the 1.36 for the entire meta-analysis. Craig Steinmaus: Yeah. That's this. >>>: Okay. Craig Steinmaus: Yeah. That's the confidence interval for that. All right. That's a heterogeneity. Now, quality, not all studies are created equal. We don't really want to do the quality scores like I told you because there's some difficulties there. There's other ways of assessing quality. By quality I mean is there potential bias? Is there potential confounding? There's ways of doing that. I'll just give you some examples. One way of doing that is subgroup analysis. All right? And so this is an example where these are the studies I just showed you. Some of the studies had good exposure assessment and some didn't. Some assessed all the major ways you can get exposed to environmental tobacco smoke. At work, at home as a child and at home as an adult. Those are the three major ways. Some of these studies assessed each one of those, I mean all three of them. And some of the studies did not. They only assessed one and they ignored all the other. Clearly you probably want to assess all three. If you are going to have a good idea of a person's true exposure to secondhand smoke you probably want to assess all three. Some did and some didn't. Mark wanted to assess what's the affect of the quality indicator on my meta-analysis. You can do a subgroup analysis. That means do one group analysis with all the studies. And do a separate meta-analysis where you only include these that did a good job of exposure assessment. Throw out the bad ones. That's a subgroup analysis. He did that and here's what we found. Studies that use all three sources he got a much higher relative risk. Better data, you get a stronger effect. That's one way of assessing quality, doing these subgroup analysis. You can do that for a whole bunch of exposure classifications. Here's one done by my colleagues. Diesel exhaust in lung cancer. They had all these studies and came up with an overall relative risk, statistically significant. Some of the studies included data on smoking and some didn't. It's a study of lung cancer. You worry about confounding from smoking. What they did was they did one meta subgroup analysis that only included smoking adjusted studies and a separate one where they didn't give any data on smoking to see did it make a difference? Were the smoking adjusted studies better or worse? Was the relative risk higher or lower? Here's the results. You can see relative risk 1.5 here. Overall 1.33. This is the confidence interval lower and upper. For smoking adjusted studies you get a relative risk of about 1.3. And for unadjusted studies you get about the same thing. It didn't make a difference. This is evidence, again, it doesn't prove anything. It's pretty strong evidence that adjusting for smoking doesn't make a difference in diesel and lung cancer studies. That smoking was not a major confounder in these studies. Does anybody know why smoking wasn't a major confounder? It causes lung cancer. You would think people that did these dirty jobs that had a lot of diesel smoke, they were blue collar workers. Maybe they would smoke more. Any idea why it didn't cause major confounding? A lot of these studies were done in the 50s and 60s and back then everybody smoked. It didn't matter if you were white collar, blue collar, rich, poor. Everybody smoked. It wasn't associated with diesel exhaust. People in the office smoked, people in the diesel truck yards smoked. This is a meta-analysis we just completed. Keep this confidential. We just completed this. It's on chromium six and does it cause stomach cancer. Chromium six is a well known cause of lung cancer. That's important because we're passing regulations in our state based on chromium six causes stomach cancer in animals. We want to know does it cause stomach cancer in humans. We reviewed all the different studies. You can see we had 71 different studies. Different job types exposed to chromium six. You can see we got an overall effect of 1.32. I'm not sure why all my examples are 1.3 something. Then we did these different subgroup analysis based on what work you did. Chromium six is a known cause of lung cancer. We know that. We separated out those studies that also found elevated lung cancer and those studies that didn't where they had data on lung cancer but it wasn't elevated. We did separate meta-analysis of those studies. We found the studies that had positive lung cancer findings also has higher relative risk for stomach cancer. We figured if a study was good enough to find an association that we know it was probably a good study. And it probably had high chromium exposures. So basically what we're doing is we're finding a higher association we know are good and found a known association versus those that weren't. It's higher here than here. We also did this. We figured maybe, I have to be careful with this. Because I work for industry too. I consult for industry. We figured that maybe if you are funded by the chromium industry you might tend not to find something. If you work in academia, maybe you tend to find something. We did studies that were funded by industry and whoa, they didn't find much. Not statistically significant. Versus studies done by academia. And again, it found something. So this gives us evidence that maybe, potentially there's some bias here. Right? Now, again don't get me wrong. I'm not saying all industry funded studies are wrong because I work for industry too. But there you go. All right. Let's skip this one. Just another example. And I want to show you this one just because this BCG vaccine decrease leprosy. Art sends me all the best students. It's great. It's not just chemical exposure. It's also infectious disease you can do meta-analysis. Does BCG vaccine decrease leprosy. It's for tuberculosis. So it's not all good. I gave you all the good stuff here. You can do meta-analysis on a lot of different stuff. Summarize literature, look at confounding, look at exposure misclassification. All that different stuff. It's not all good. There's a lot of problems. Sometimes studies don't give you all the data you need. Some studies will give you relative risk but won't give you a confidence interval. Some studies will say we didn't find anything and won't give you the results. There's also this issue of publication bias. Publication bias is the tendency of journals to publish results that are statistically significant. And not to publish results that are negative. All right. That's publication bias. And it turns out it actually is true. It actually is out there. There does seem to be this tendency for journals to publish results that are statistically significant. This is perhaps the earliest study. This is major journals in psychology at the time. It was one of the classical studies at the time. They took every single study in each one of these journals over an entire year and they saw how many studies gave a test of statistical significance, a P value. How many studies gave a P value and how many of the P values were less than 0.05. How many results were statistically significant? You can see in these journals 97 percent. 95 percent of higher of all results were statistically significant. That's huge. Maybe researchers are just really good at deciding to study things where they are going to find an effect. Maybe there's this publication bias where only positive studies, only journals tend to report positive studies. There's been subsequent studies like this. It's not nearly as bad as it used to be. I don't think we've still sort of solved the problem of publication bias. Yes. >>>: I'm trying to get a sense of how bad this is and sort of what implications you could draw from it? I look at that. You know, those percentages are greater than the 95 percent confidence interval. So isn't that telling you the studies you get that are showing, you have a likelihood that, good likelihood the studies are showing 95 confidence are happening just by chance and they are happening just by chance with the same likelihood they are happening because there's an actual association there. Craig Steinmaus: I think what you are getting at if the journals only have positive studies and you only use journal articling for your meta-analysis and you only use journal articles for your meta-analysis it's always going to be positive. >>>: Does that you couldn't make the call that a study that shows statistical significance is due to anything other than chance alone? Craig Steinmaus: I think what it says is it says that you are getting, if you just go to the journals you are getting a biased selection of studies a lot of time. You are only getting those positive studies. Those negative studies are out there and they are not being published. It's not the editor's fault. It's not the editors saying well I have ten positive studies and ten negative studies. I'm only going to publish the positive ones. It turns out it's the researcher's fault. You have two studies. One positive and one negative. The negative one will sit on your desk for a while. I'll publish it later. You never get to it. Researchers tend not to publish their negative results. That's the cause of this. Again, if you are only publishing positive results your meta-analysis is always going to be positive based on that. There's ways of getting around this or not so much getting around this. There's ways of assessing how bad this is. If anybody is going to do a meta-analysis come to me and we can talk about those ways. There's funnel plots. There's statistical tests you can look at. There's a whole variety of ways you can see not necessarily correct for it but to see how big of a problem it is. Remember, that's epidemiology. You are not looking at whether there's a bias or whether there's confounding. There's always bias and there's always confounding. What you are looking at is how big is it. And there are statistical tests and other ways you can see if it's a big problem or a little problem and it's different for different topics. Some topics it's a huge problem and others it's not a big problem. There's ways to check that out. I don't have time to talk about each one. Two minutes. Let me skip this and I'll get to this. Which is there's a couple of good programs. I showed you the mathematical calculations. You don't have to do those by hand. There's a couple of programs that can do it. Perhaps the two best are STATA and this one that was named after me, Craigs program. I have an Excel spreadsheet that you plug in the relative risk and confidence interval. All the numbers you want pop up. If you are comfortable with STATA and not so comfortable with Excel you can do it in STATA. STATA is good for meta-analysis. You can supposedly do it in SAS but it's miserable. Any questions? About any of that. Yes. >>>: I came in a few minutes late. Would you mind defining the BI coefficient? Craig Steinmaus: It's the log of the relative risk. But again you don't need to worry too much about that. Remember, we take it back out of the log later on. Any other questions? Okay. Free to go (applause).