Why grad students benefit from open science practices

Open science badges
Open science badges created and maintained by the Open Science Collaboration

The Context

We are in the midst of a paradigm shift in psychology. It has been less than ten years since the Simmons, Nelson, and Simonsohn “False-Positive Psychology” paper demonstrated “how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis,” calling into question the legitimacy of standard research procedures used in many psychology labs. A month later, the Open Science Framework lead a team of 270 psychologists in a massive endeavor to reproduce 100 findings published in top psychology journals. In 2015, the results of this ‘Reproducibility Project’ revealed that only 39% of the original reported effects were reproduced, leading many to conclude that psychology had entered a replication crisis. The scientific integrity of the field’s research was being called into question by those both within and outside it, as one by one famous effects fell like dominoes.

The sudden thrust of the field under the scrutiny of many ‘small telescopes’ led to a valiant, collective attempt at course correction. New tools such as pcurve and statcheck made it easier to detect whether reported statistics were incorrect or improbable, in published studies or one’s own work. 2016 saw the first meeting of the Society for the Improvement of Psychological Science, catalyzing collaborative projects aimed at strengthening the evidenciary value of our research such as constraints on generality and StudySwap. Journals increasingly started to employ open science badges and TOP guidelines, and accept registered reports. Reproducible science had become a movement in its own right within psychology, and its proponents believed that the values of open materials, pre-registration, and multilab collaboration were the keys to ending the crisis. With the emergence of this movement the field seemed to be on the verge of a scientific revolution.

"Reproducible science had become a movement, and its proponents believed that the values of open materials, pre-registration, and multilab collaboration were the keys to ending the crisis."

The Concerns

During any paradigm shift the group that has the most at risk are those with the least power, who have no choice but hold on tight and try to ride out the clashes between new and old high-status figures. Part of the reason for this post is to honestly discuss some of the potential consequences of psychology’s shift from the perspective of a low-power position within the academic hierarchy: that of a graduate student. Grad student education and professional development are inherently dependent upon one’s home department and faculty advisor, and access to prestige-building opportunities such as publications, awards, and speaking invitations are additionally dependent on the standards and interests of the field. Thus, a graduate student’s future life outcomes, especially their potential employment prospects, are shaped by a diverse assortment of gatekeepers who pass on their own standards for scientific excellence. With the ongoing conflict between traditional and open science schools of psychology, current graduate students are seeing the definition of what it means to be an employable psychologist shift under their feet. Where to land on either side of the dividing line between these two schools of thought can therefore be a risky decision for current graduate students preparing for the job market while the field itself is unsettled.

The concerns that I have heard both first- and second-hand from other graduate students indeed indicate that evaluating this risk may lead to reluctance to adopt open science practices in their own work. The prevailing wisdom is that to be competitive for academic jobs one should maximize first-author publications and publications in high-impact journals, and some grads worry that open science methods are (currently) practically at odds with these goals. It is true that for some areas, switching from a traditional psychology research framework to one focused on openness and reproducibility can slow the progression of research question to published manuscript down. Pre-registration, testing replicability of effects, and making data and materials open all take time. It may therefore seem risky to not instead spend this time pursuing status quo publication, especially if your peers (and potential fellow job applicants) are not expending the time on these efforts themselves. Similarly, the time and effort spent conducting a replication study or contributing to a many labs initiative is comparable to that of individual, novel projects, but may appear less prestigious to some journals or hiring committees (especially if one is not fist author). When time, funding, and other resources are scarce and one’s future is on the line, it may therefore seem less risky to maintain the status quo - and this perceived risk only looms larger for students who are working in traditional psychology labs or under advisors reluctant (or opposed) to open science efforts.

But having advanced through grad school simultaneously with the building of the open science movement, and having worked in both traditional and open psychology labs, my message to fellow grads is this: the most valuable use of your limited resources is in practicing open and reproducible science. The benefits of these initiatives include learning an array of marketable skills, shrinking the timeline from project initiation to citable product, and empowering networking and collaboration across areas, fields, and continents. I hope that the following section can convince you that adopting open practices is not a risk, but rather a boon to your professional development.

"My message to fellow grads is this: the most valuable use of your limited resources is in practicing open and reproducible science."

The Benefits

  • Open science encourages deep thinking.
    As academics we are, of course, in the business of thinking. But many of us have had the experience of jumping head first into the deep end of a new research project and later facing unanticipated issues, or making off-the-cuff decisions because a protracted discussion would halt a project's momentum. While these events may seem small and inconsequential for a study's integrity in the moment, they could actually be meaningfully changing how the results should be interpreted. For example, let's say you are halfway through data collection for a study with deception and discover that during debriefing a handful of participants were wise to the manipulation. Now what? Your decision could influence your results by removing data not-at-random (dropping all wise participants from final tests), artifically inflating your p values (conducting a significance test to see if wise participants show the effect, then continuing with data collection if so), or contaminating the controlled part of your experiment (changing the study design mid-collection). And, more concerning, this decision may wind up as a footnote in your methods section rather than a substantial piece of your conclusion given its potential to alter the statistical assumptions behind your significance tests.

    The open science practice of pre-registration offers a buffer against making quick decisions with potential drastic effects by formally incorporating in-depth thinking about study design and statistical analysis into project planning. Pre-registrations are written and frozen on a repository such as the Open Science Framework prior to the collection and analysis of data. This process is beneficial to grad students' statistical thinking by training them to answer the questions of 'how','why', and 'what if' in planning their research. There are a variety of pre-registration templates available for use on the OSF, but they share similar questions such as:
    • How will you choose your sample size?
    • How will you operationalize your variables?
    • Why are your hypotheses directional or not?
    • Why are you reporting that statistic?
    • What if your data are heavily skewed?
    • What if you observe outliers in your data?
    Upon completing a pre-registration you will have a 'recipe' for your study that includes (as much as humanly possible) answers to issues that could pop up during the life of the project. You'll also have taken the time to think about and record the justification for your design decisions. And ideally you'll be able to step back and consider how all of the components in this document relate to the interpretion of the results you'll eventually find. Thus, the benefit is in making statistical thinking habitual, planned, and thoughtful, improving not only your skills, but also the veracity of your work.

  • Open science shortens the timeline from project initiation to citable product.
    In contrast to the concern that I have heard expressed that open science initiatives slow down the publication process, my personal experience is that it can speed things up. There are two open science initiatives that are excellent vehicles for getting your work out of the lab and on to your CV. Registered reports are essentially pre-registrations that are sent to journals for peer review. In Phase One of a registered report, authors submit an introduction section as well as sections for planned methods and data analysis. If granted an in principle acceptance, the manuscript is accepted for publication. Authors complete the manuscript by reporting the results of the registered analyses in Phase Two, but the article is citable (and sometimes even made available online) at the point of acceptance. Registered reports can be especially beneficial for graduate students who collect longitudinal data, work with difficult-to-access samples, or who are tackling otherwise ambitious projects that take a lot of time to complete. And again they can help increase the veracity of the science reported by providing a platform for good ideas and well-planned methodology, regardless of the findings.

    Registered reports do require some forethought as most journals won't accept proposals using data that have already been collected (see this list of participating journals). However, even if your project is complete you can post it publicly as a preprint and receive a citable DOI. Preprints are completed manuscripts that have either yet to have been accepted for publication at a journal, or have been accepted but are in-press. In most cases pre-prints are not the same as the final published article, but they allow the authors to present their work before their peers for feedback. Psychology article preprints can be posted on the PsyArXiv (and you can follow the friendly bot on twitter to be alerted when a new preprint is posted). While posting a preprint by no means that your work is done - the manuscript does need to undergo peer review, after all - it does mean that you have an easy, accesible way to share your project with colleagues while you endure the formal review process.

  • Open science builds your marketable skills.
    Psychology grad students are spoiled with potential career options. We can stick to the academic life and profess, run regressions for lots of money in Silicon Valley, use our data wrangling skills to tackle social issues in non-profits, or become a superstar science writer with our talents for written and verbal communication. Any field that values teaching, coding, analyzing, and communicating combined with the ability to distinguish truth from fiction is a match for psychologists - and open science initiatives refine these skills. As mentioned before, pre-registration helps grads refine their statistical thinking and get in the habit of moving this process to the forefront of their study design. Opening one's work to a broader audience of peers via registered reports and preprints means more opportunities to improve written communication of complex ideas. And posting data and analysis scripts with the intention of making one's work open and reproducible means constant practice at coding. Since open science is as much of a community as it is a movement in psychology, getting involved also means exposure to peers all over the world whom you can learn from. You don't have to rely only on the skills that are taught in your home department or lab, and this will benefit your development as a well-rounded scientist who is competitive for jobs in a variety of fields.

  • Open science helps you connect.
    There's a truth that I don't hear often spoken aloud but which is nevertheless important for grad students to be aware of: networking matters, even in science. Opportunities for collaborations on projects, speaking in symposia, postdoctoral fellowships, and yes, even full-time careers are to an extent distributed through social networks. Psychology is a pretty big field full of interminably busy people, so it makes sense that we choose to work with people in our immediately accessible environment - people who we know, whose work we've read, and/or who reach out to us to collaborate. The open science movement is therefore beneficial for grad students as it provides many routes to making connections. As previously mentioned, preprints can help get your name and your work out to the community - I've seen many an early career researcher have their preprints praised by some pretty big names on twitter (retweeted often, of course, from the the helpful bot) which then get picked up across the field. Producing work that is not traditionally rewarded in psychology, such as coding an experimental task or developing an open dataset, can again be assigned a citable DOI through the OSF or other repositories and lead to new collaborations and projects. The Society for the Improvement of Psychological Science (SIPS) hosts an annual conference focused on furthering open science which continues to have a max attendance of a couple hundred people, and is therefore a great opportunity for establishing relationships with other scientists. Being open is therefore beneficial not just for the field overall, but also for your professional development and reputation. Engaging with the open science community can be especially beneficial for those who come from staunchly traditional labs or schools, as you'll no doubt find people to learn from and work with.
"Any field that values teaching, coding, analyzing, and communicating combined with the ability to distinguish truth from fiction is a match for psychologists - and open science initiatives refine these skills."

This is by all means not a comprehensive list of the benefits of open science, but hopefully it is enough to encourage you to try it out if you haven’t before. I do believe that while the concerns I’ve heard from fellow grads are valid and need to be addressed by the field overall (for example, by reframing the heuristics used to evaluate job candidates), many of them may be due to a lack of experience working within an open framework. Since engaging fully with the movement in my mid-grad school career, I have seen improvements in my statistical reasoning and understanding, my written code, and my productivity. I also feel more confident in the results that I report, even if those results are null, and more comfortable admitting when I just don’t have enough evidence to write a tidy conclusion- and I believe these things improve my quality of a scientist as much as the practical skill-building does. The plural of anecdote is not data, but I encourage you to conduct your own research and see if perhaps open science is right for you- because even if the future is unknown, at least the methods you took to predict it will be.