This year my research life has been a bit of a roller-coaster. A little less than a year ago, my thesis research involved apply a reasonable-sounding methodology to an overarching question -- what sets the size of a hurricane? -- and on my first "try" in its application I basically got a beautiful and simple result. In very generic and universal terms, all the dots lined up along straight lines, and those straight lines and slopes that made intuitive sense.
In the subsequent 9 months, I proceeded to question various aspects of the details of my research which ranged from the stupid (e.g. finding simulations where I never actually changed the one or two parameters that I thought had been changed), to the scientifically legitimate (e.g. why did I choose to represent this physical process in this way?). This led me to, not once but twice, rerun all of my simulations. The first time, I made a couple of simpler changes, but not long afterwards I realized that I had simply been lazy in implementing certain aspects of the simulations which probably don't matter, but it's impossible to know for sure. Thus, I hunkered down and reran everything again, this time applying a much greater degree of quantitative precision in determining if I was happy with how I was doing things each step of the way.
Thus, by September, I finally, for the first time, felt good about my work: I was confident that the results I was getting were not tainted by small but easily avoidable errors in scientific judgment. I also realized really for the first time how crucial personal responsibility is to doing good science -- after all, no one, not even your advisor, will ever know the details of how you reach the answers that you do. Obviously in some fields this enables bad behavior such as scientific fraud. Often, though, such fraud can easily be blanketed over because "that's science". However, it's plainly evident to me that "that's science" encompasses both legitimate and unavoidable human error as well as pure scientific negligence, and in most cases it is entirely up to the researcher and his/her code of ethics to determine to what extent the former is minimized and the latter is avoided. Either way, at this stage in my scientific career, it became clear that if I want to do science that I can be proud of, I need to make sure its done right -- to the best that I can define it as so -- because no one else is going to decide that for me.
Nonetheless, since September I've been faced with a new set of scientific questions, namely why, after all of the changes that I made to make my simulations more scientifically valid, I was now getting results that were far messier (read: dots no longer so neatly aligned) than they were at the outset?
Consciously and subconsciously, I worried about this new problem but, because the overall results made sense (now for a variety of reasons), I in any case set about writing up a paper on my work. I sent it to my advisor, who approved it with very minimal criticism. Following a couple of additional points to address, I felt it was ready to submit to the top journal in our field.
Even then, my uneasiness continued and I still somehow felt unconvinced by my own results. Then, a couple of weeks ago, I found a systematic bias in my results. I had no idea why, but this was a bit of a smoking gun that I cannot simply explain away the "noise" as noise. After a day or so of complete freaking out -- my results are wrong, I shouldn't be a scientist etc. etc. -- I told myself that I was going to figure this out one way or another. I could look more directly at my simulations and see plainly with my eyes that my fundamental conclusions were not wrong, which at the least restored some basic confidence in myself that I wasn't a complete idiot.
After a couple more days or intense thinking and a willingness finally to open up and complain/talk about this with my friends/colleagues, I realized something difficult but important: I was a victim of my own early success -- I really wanted to see the pretty result I got a year ago -- and what I needed to do was extricate my thought processes about this problem from the methodological approach that I have been using. In other words, I needed to step back and ask: if I just showed up now, how would I approach the problem?
It turns out that, despite the fears of finding out you've done it wrong all along, a fresh start can do wonders. I now see why it is that, for example, tech companies and grow and then fail, themselves doomed to never think outside of the very box that they built and that led to their success.
This past weekend, a flurry of insights and understanding came through, and it soon became clear that in fact my original answer was never wrong, but there simply was more going on than I had thought. Importantly, I would not have realized this without having redone all of my simulations correctly, as it's clear that my initial results from one year ago got what had seemed to be the right answer (i.e. the simple one) likely out of sheer luck.
What might have been plausibly termed "human error" was in fact, in the most negative of respects, scientific negligence on my part. Being lazy may obscure the signal you are looking for or, worse, accidentally lead you to either a signal that's not actually there or, in my case, a mis-interpretation of the answer. Precision matters.
Ah, graduate school. I suppose this is the learning experience that grad school is supposed to provide. It can be harsh. But it's obvious that such an experience depends a lot more on personal responsibility and self-confidence than I had ever imagined.
In the subsequent 9 months, I proceeded to question various aspects of the details of my research which ranged from the stupid (e.g. finding simulations where I never actually changed the one or two parameters that I thought had been changed), to the scientifically legitimate (e.g. why did I choose to represent this physical process in this way?). This led me to, not once but twice, rerun all of my simulations. The first time, I made a couple of simpler changes, but not long afterwards I realized that I had simply been lazy in implementing certain aspects of the simulations which probably don't matter, but it's impossible to know for sure. Thus, I hunkered down and reran everything again, this time applying a much greater degree of quantitative precision in determining if I was happy with how I was doing things each step of the way.
Thus, by September, I finally, for the first time, felt good about my work: I was confident that the results I was getting were not tainted by small but easily avoidable errors in scientific judgment. I also realized really for the first time how crucial personal responsibility is to doing good science -- after all, no one, not even your advisor, will ever know the details of how you reach the answers that you do. Obviously in some fields this enables bad behavior such as scientific fraud. Often, though, such fraud can easily be blanketed over because "that's science". However, it's plainly evident to me that "that's science" encompasses both legitimate and unavoidable human error as well as pure scientific negligence, and in most cases it is entirely up to the researcher and his/her code of ethics to determine to what extent the former is minimized and the latter is avoided. Either way, at this stage in my scientific career, it became clear that if I want to do science that I can be proud of, I need to make sure its done right -- to the best that I can define it as so -- because no one else is going to decide that for me.
Nonetheless, since September I've been faced with a new set of scientific questions, namely why, after all of the changes that I made to make my simulations more scientifically valid, I was now getting results that were far messier (read: dots no longer so neatly aligned) than they were at the outset?
Consciously and subconsciously, I worried about this new problem but, because the overall results made sense (now for a variety of reasons), I in any case set about writing up a paper on my work. I sent it to my advisor, who approved it with very minimal criticism. Following a couple of additional points to address, I felt it was ready to submit to the top journal in our field.
Even then, my uneasiness continued and I still somehow felt unconvinced by my own results. Then, a couple of weeks ago, I found a systematic bias in my results. I had no idea why, but this was a bit of a smoking gun that I cannot simply explain away the "noise" as noise. After a day or so of complete freaking out -- my results are wrong, I shouldn't be a scientist etc. etc. -- I told myself that I was going to figure this out one way or another. I could look more directly at my simulations and see plainly with my eyes that my fundamental conclusions were not wrong, which at the least restored some basic confidence in myself that I wasn't a complete idiot.
After a couple more days or intense thinking and a willingness finally to open up and complain/talk about this with my friends/colleagues, I realized something difficult but important: I was a victim of my own early success -- I really wanted to see the pretty result I got a year ago -- and what I needed to do was extricate my thought processes about this problem from the methodological approach that I have been using. In other words, I needed to step back and ask: if I just showed up now, how would I approach the problem?
It turns out that, despite the fears of finding out you've done it wrong all along, a fresh start can do wonders. I now see why it is that, for example, tech companies and grow and then fail, themselves doomed to never think outside of the very box that they built and that led to their success.
This past weekend, a flurry of insights and understanding came through, and it soon became clear that in fact my original answer was never wrong, but there simply was more going on than I had thought. Importantly, I would not have realized this without having redone all of my simulations correctly, as it's clear that my initial results from one year ago got what had seemed to be the right answer (i.e. the simple one) likely out of sheer luck.
What might have been plausibly termed "human error" was in fact, in the most negative of respects, scientific negligence on my part. Being lazy may obscure the signal you are looking for or, worse, accidentally lead you to either a signal that's not actually there or, in my case, a mis-interpretation of the answer. Precision matters.
Ah, graduate school. I suppose this is the learning experience that grad school is supposed to provide. It can be harsh. But it's obvious that such an experience depends a lot more on personal responsibility and self-confidence than I had ever imagined.
