Choose your own adventure to a reproducible scientific article

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[
## Choose your own adventure to a reproducible scientific article
### **learnings from ReproHack**

Florencia D'Andrea

---

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[

**Link to the slides**: [bit.ly/reprohack_toronto](bit.ly/reprohack_toronto)

]

---

<img src="imagenes/openresearch.jpg" width="700" style="display: block; margin: auto;" />
.footnote[[The Turing Way Community, & Scriberia. (2020)](http://doi.org/10.5281/zenodo.3695300)]

---

<img src="imagenes/circulo.png" width="250" style="display: block; margin: auto;" />
## More articles with published code and data are accessible and **could be reproduced by everyone**

---

### The scientific community should learn about **reproducible practices**

* Facilitate **collaboration** and review

* Publish **validated research**

]

.pull-right[
<img src="imagenes/researchtools.jpg" width="700" style="display: block; margin: auto;" />
]

.footnote[[The Turing Way: A Handbook for Reproducible Data Science (2019)](https://the-turing-way.netlify.app/) | [The Turing Way Community, & Scriberia. (2020)](http://doi.org/10.5281/zenodo.3695300)]

---

### The scientific community should learn about **reproducible practices**

* Write your papers **efficiently**

* **Get credit** for your work fairly
]

.pull-right[
<img src="imagenes/researchtools.jpg" width="700" style="display: block; margin: auto;" />
]

## Reproducibility is consider a proof of **Research Quality**

.pull-up[
<img src="imagenes/codecheck.png" width="400" style="display: block; margin: auto;" />
.footnote[[Codecheck](https://codecheck.org.uk/)]
 ]
 
.pull-down[
<img src="imagenes/codecheck2.png" width="700" style="display: block; margin: auto;" />
]

---

## ¿Are we experiencing a cultural **change**?

<blockquote class="twitter-tweet" data-lang="ens"><p lang="en" dir="ltr">The advert asks for: 
"A commitment to following the best [..] practices in science, such as [..] sharing of computer code and writing reproducible research reports, [..]sharing of data whenever feasible"

Have you come across job descriptions asking for such a commitment before?</p>&mdash; ReproHack(@ReproHack) <a href="https://twitter.com/ReproHack/status/1296061566484385792?s=20">19 August 2020</a></blockquote>

---

## ¿Are we experiencing a cultural **change**?

---

## **ReproHack**
Learning from the experience

---

## Hackathon

Send their papers with **publicly available associated code and data**

<img src="imagenes/circulo.png" width="300" style="display: block; margin: auto;" />
]

Attempt to **reproduce published research** of their choice from a list of proposed papers

<img src="imagenes/selecciona.png" width="500" style="display: block; margin: auto;" />
]

At the end, the participants give feedback to the authors

---

## An example of a **submitted paper**

Thanks [Luis Verde](https://twitter.com/LuisDVerde)

---

## As researchers, we know how a **manuscript** is organized

---

## How **code** and **data** could be organised?

---

## Research Compendium

1. **Organize files** according to a prevailing convention.

2. Provide **separation between data, methods and results** expressing unambiguously the relationship between the three.

3. Specify the environment 
(+ **reproducibility**).

]

.footnote[[The Turing Way: A Handbook for Reproducible Data Science (2019)](https://the-turing-way.netlify.app/)]
---

## Research Compendium

<img src="imagenes/compendio1.png" width="450" style="display: block; margin: auto;" />
]

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[
[`rrtools`](https://github.com/benmarwick/rrtools) provides instructions, templates and functions to make a basic compendium suitable for writing **reproducible research with R **.]

* Anna Krystalli's [`rrtools` Tutorial](https://annakrystalli.me/rrresearch/10_compendium.html)

]

---

## How data and code is shared?

---
class: inverse, center, middle

# 1

## 📄

## `README`

Guide the reviewer to the results

---

## What a `README` should have?

* General information

* **Data and file** overview

* **Sharing and access** information

* **Methodological** information

### Is all of this information **necessary to reproduce the article?**

[Example `rrtools README`](https://github.com/annakrystalli/rrcompendiumDTB)

.footnote[[Guide to writing "README" style metadata - Cornell University](https://data.research.cornell.edu/content/readme)]

---

## 🧭

### What do we need?

Reading a license will not give us the steps we must take to reproduce the article.

---

## Explain HOW

<table class=" lightable-minimal table" style='font-family: "Trebuchet MS", verdana, sans-serif; margin-left: auto; margin-right: auto; margin-left: auto; margin-right: auto;'>
 <thead>
  <tr>
   <th style="text-align:left;"> Category </th>
   <th style="text-align:left;"> Example section headings </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> What </td>
   <td style="text-align:left;"> Introduction, project, background </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Why </td>
   <td style="text-align:left;"> Advantages of the project, comparison with related work </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;color: white !important;background-color: #6a0dad !important;"> How </td>
   <td style="text-align:left;font-weight: bold;color: white !important;background-color: #6a0dad !important;"> Getting started, how to run, installation, how to update, configuration, languages, dependencies,... </td>
  </tr>
  <tr>
   <td style="text-align:left;"> When </td>
   <td style="text-align:left;"> Project status, versions, roadmap,... </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Who </td>
   <td style="text-align:left;"> Project team, community, contact,... </td>
  </tr>
  <tr>
   <td style="text-align:left;"> References </td>
   <td style="text-align:left;"> feedback, getting support, more information </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Contribution </td>
   <td style="text-align:left;"> Contribution guidelines </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Other </td>
   <td style="text-align:left;">  </td>
  </tr>
</tbody>
</table>

.footnote[[Table extracted from Prana *et al.* (2019)](https://link.springer.com/article/10.1007/s10664-018-9660-3#citeas)]

---

## What I find useful to reproduce the article results?

A clear and detailed description in the `README` of the files/links necessary to run and where they are

---

# 2

## 💻

#### How the computational environment is captured could change the experience of reproducing the article
### And what to write -> **README**

---

## Computational environment

## 💻

Characteristics of a computer that can affect the behavior of the work done on it, such as:

* your **operating system**

* what **software** do you have installed

* software **package versions** are installed

.footnote[[The Turing Way: A Handbook for Reproducible Data Science (2019)](https://the-turing-way.netlify.app/)]

---
## Computational Environments

Important to capture them to avoid:

* Problems when sharing your code

* That you code breaks with time

---

### **3** examples of information to include in the `README` depending on how you are capturing the computational environment

**A.** Provide a plain text file with the **detailed description** of dependencies and software/hardware used

**B.** Describe how to **install** required software

**C.** Share a **link**

---

### **A.** Provide a plain text file with the **detailed description** of dependencies and software/hardware used

👩‍💻  ** Hackathon participant experience** <svg viewBox="0 0 576 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;fill:currentColor;position:relative;display:inline-block;top:.1em;">  [ comment ]  <path d="M259.3 17.8L194 150.2 47.9 171.5c-26.2 3.8-36.7 36.1-17.7 54.6l105.7 103-25 145.5c-4.5 26.3 23.2 46 46.4 33.7L288 439.6l130.7 68.7c23.2 12.2 50.9-7.4 46.4-33.7l-25-145.5 105.7-103c19-18.5 8.5-50.8-17.7-54.6L382 150.2 316.7 17.8c-11.7-23.6-45.6-23.9-57.4 0z"></path></svg><svg viewBox="0 0 576 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;fill:currentColor;position:relative;display:inline-block;top:.1em;">  [ comment ]  <path d="M528.1 171.5L382 150.2 316.7 17.8c-11.7-23.6-45.6-23.9-57.4 0L194 150.2 47.9 171.5c-26.2 3.8-36.7 36.1-17.7 54.6l105.7 103-25 145.5c-4.5 26.3 23.2 46 46.4 33.7L288 439.6l130.7 68.7c23.2 12.2 50.9-7.4 46.4-33.7l-25-145.5 105.7-103c19-18.5 8.5-50.8-17.7-54.6zM388.6 312.3l23.7 138.4L288 385.4l-124.3 65.3 23.7-138.4-100.6-98 139-20.2 62.2-126 62.2 126 139 20.2-100.6 98z"></path></svg><svg viewBox="0 0 576 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;fill:currentColor;position:relative;display:inline-block;top:.1em;">  [ comment ]  <path d="M528.1 171.5L382 150.2 316.7 17.8c-11.7-23.6-45.6-23.9-57.4 0L194 150.2 47.9 171.5c-26.2 3.8-36.7 36.1-17.7 54.6l105.7 103-25 145.5c-4.5 26.3 23.2 46 46.4 33.7L288 439.6l130.7 68.7c23.2 12.2 50.9-7.4 46.4-33.7l-25-145.5 105.7-103c19-18.5 8.5-50.8-17.7-54.6zM388.6 312.3l23.7 138.4L288 385.4l-124.3 65.3 23.7-138.4-100.6-98 139-20.2 62.2-126 62.2 126 139 20.2-100.6 98z"></path></svg>

> *"Could be easier"*

---

###  **B.** Describe how to **install** required software

example: Install Docker

☕   **Hackathon participant experience** <svg viewBox="0 0 576 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;fill:currentColor;position:relative;display:inline-block;top:.1em;">  [ comment ]  <path d="M259.3 17.8L194 150.2 47.9 171.5c-26.2 3.8-36.7 36.1-17.7 54.6l105.7 103-25 145.5c-4.5 26.3 23.2 46 46.4 33.7L288 439.6l130.7 68.7c23.2 12.2 50.9-7.4 46.4-33.7l-25-145.5 105.7-103c19-18.5 8.5-50.8-17.7-54.6L382 150.2 316.7 17.8c-11.7-23.6-45.6-23.9-57.4 0z"></path></svg><svg viewBox="0 0 576 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;fill:currentColor;position:relative;display:inline-block;top:.1em;">  [ comment ]  <path d="M259.3 17.8L194 150.2 47.9 171.5c-26.2 3.8-36.7 36.1-17.7 54.6l105.7 103-25 145.5c-4.5 26.3 23.2 46 46.4 33.7L288 439.6l130.7 68.7c23.2 12.2 50.9-7.4 46.4-33.7l-25-145.5 105.7-103c19-18.5 8.5-50.8-17.7-54.6L382 150.2 316.7 17.8c-11.7-23.6-45.6-23.9-57.4 0z"></path></svg><svg viewBox="0 0 576 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;fill:currentColor;position:relative;display:inline-block;top:.1em;">  [ comment ]  <path d="M528.1 171.5L382 150.2 316.7 17.8c-11.7-23.6-45.6-23.9-57.4 0L194 150.2 47.9 171.5c-26.2 3.8-36.7 36.1-17.7 54.6l105.7 103-25 145.5c-4.5 26.3 23.2 46 46.4 33.7L288 439.6l130.7 68.7c23.2 12.2 50.9-7.4 46.4-33.7l-25-145.5 105.7-103c19-18.5 8.5-50.8-17.7-54.6zM388.6 312.3l23.7 138.4L288 385.4l-124.3 65.3 23.7-138.4-100.6-98 139-20.2 62.2-126 62.2 126 139 20.2-100.6 98z"></path></svg>

> *"This will take some time. Let's go for a coffee"*

---

### **C.** Share a **link**

example: Binder

😎 **Hackathon participant experience** <svg viewBox="0 0 576 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;fill:currentColor;position:relative;display:inline-block;top:.1em;">  [ comment ]  <path d="M259.3 17.8L194 150.2 47.9 171.5c-26.2 3.8-36.7 36.1-17.7 54.6l105.7 103-25 145.5c-4.5 26.3 23.2 46 46.4 33.7L288 439.6l130.7 68.7c23.2 12.2 50.9-7.4 46.4-33.7l-25-145.5 105.7-103c19-18.5 8.5-50.8-17.7-54.6L382 150.2 316.7 17.8c-11.7-23.6-45.6-23.9-57.4 0z"></path></svg><svg viewBox="0 0 576 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;fill:currentColor;position:relative;display:inline-block;top:.1em;">  [ comment ]  <path d="M259.3 17.8L194 150.2 47.9 171.5c-26.2 3.8-36.7 36.1-17.7 54.6l105.7 103-25 145.5c-4.5 26.3 23.2 46 46.4 33.7L288 439.6l130.7 68.7c23.2 12.2 50.9-7.4 46.4-33.7l-25-145.5 105.7-103c19-18.5 8.5-50.8-17.7-54.6L382 150.2 316.7 17.8c-11.7-23.6-45.6-23.9-57.4 0z"></path></svg><svg viewBox="0 0 576 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;fill:currentColor;position:relative;display:inline-block;top:.1em;">  [ comment ]  <path d="M259.3 17.8L194 150.2 47.9 171.5c-26.2 3.8-36.7 36.1-17.7 54.6l105.7 103-25 145.5c-4.5 26.3 23.2 46 46.4 33.7L288 439.6l130.7 68.7c23.2 12.2 50.9-7.4 46.4-33.7l-25-145.5 105.7-103c19-18.5 8.5-50.8-17.7-54.6L382 150.2 316.7 17.8c-11.7-23.6-45.6-23.9-57.4 0z"></path></svg>

> *"So, I click a link and I get the results?"*

.footnote[[Image from Mendez *et al.*(2019)](https://link.springer.com/article/10.1007/s11306-019-1588-0)]
---

.footnote[[Publishing reproducible logbooks](https://opendreamkit.org/2017/11/02/use-case-publishing-reproducible-notebooks/)]
---

Binder is an open source web service that lets users create **sharable, interactive, reproducible environments** in the cloud.

.footnote[[Binder 2.0 - Reproducible, interactive, sharable environments for science at scale](https://pdfs.semanticscholar.org/c043/bef741a9616d1144e0205ac21ceae881485d.pdf)]
---

## Capturing Computational Environments

.footnote[[The Turing Way: A Handbook for Reproducible Data Science  (2019)](https://the-turing-way.netlify.app/reproducible-research/renv/renv-options.html)]

---
## The dark side of links 🌒

---

* **What happens when the technology fails?** 
* Should we provide more than one option?
* Should our code have **maintainers**?

<img src="imagenes/FixingLinks.jpg" width="600" style="display: block; margin: auto;" />
.footnote[[The Turing Way Community, & Scriberia. (2020)](http://doi.org/10.5281/zenodo.3695300)]

---

# **{**Definition**}**
### Research software

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[
#### Software that is used to generate, process or analyse results that you intend to appear in a publication (either in a journal, conference paper, monograph, book or thesis).

#### Research software can be anything **from a few lines of code written by yourself, to a professionally developed software package**]

---

## 👩‍💻

### Did you ever consider that your next scientific article **can have not only readers but also users**?

---

# 3

## 👩‍💻

## ~~REVIEWERS~~

## **USERS**

---

# 4

## 📝

## Documentation

#### Define **who** are your users and **write for them!**

Are you writing the documentation for you or for someone that doesn't know the technology you are using?

---

## ReproHack **feedback form**

Reproducibility ✅

Reusability ✅

Documentation ✅

Transparency ✅

---

## What I have learned in **ReproHack**? ♻️

---

## What I have learned in **ReproHack**? ♻️

#### - It is important to write a **good README ** file

#### - How the **computational environment** is captured could change the experience of reproducing the article

#### - Write **detailed documentation** for your users

#### - **Researchers develop software** for their scientific articles (even if their are not aware of it)

---

## 👉 You can have **your own experience!**

---

### Organize a remote hackathon

Attend our events!

---
class: middle, center
# Join us!

[Twitter](https://twitter.com/ReproHack)

[Slack](https://reprohack-autoinvite.herokuapp.com/)

[Email](reprohack.team@gmail.com)

---

# ReproHack in [LatinR 2020](https://latin-r.com/blog/reprohack)

[Playlist with 6 talks on reproducibility in Spanish](https://www.youtube.com/playlist?list=PL9-E3cL2KgKliN3DFBWfUAUNXco_NOAMQ)

---

## Ana Krystalli - UseR! 2020

[Video](https://www.youtube.com/watch?v=KHMW8fV2NXo)

---

## 🤝 🏆 🤝

## Thanks to ReproHack **Team**!

Daniela Gawehns<br>
Anna Krystalli <br> 
Linda Nab <br> 
Ricarda Proppert <br> 
Paloma Rojas-Saunero <br> 
Florencia D'Andrea

---

## References

---

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[

* Cornell University - [Guide to writing "readme" style metadata](https://data.research.cornell.edu/content/readme)

* Hettrick, S., Antonioletti, M., Carr, L., Chue Hong, N., Crouch, S., De Roure, D., … Sufi, S. (2014). UK Research Software Survey 2014 [Data set]. Zenodo. http://doi.org/10.5281/zenodo.14809

* Jupyter Project, Bussonnier, M., Forde, J., Freeman, J., Granger, B., Head, T., ... & Willing, C. (2018, July). [Binder 2.0-Reproducible, interactive, sharable environments for science at scale](https://pdfs.semanticscholar.org/c043/bef741a9616d1144e0205ac21ceae881485d.pdf?_ga=2.134990321.1289472605.1614439362-796733345.1614439362). In Proceedings of the 17th python in science conference (Vol. 113, p. 120).

]

---

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[

* Marwick, B., Boettiger, C., & Mullen, L. (2018). [Packaging data analytical work reproducibly using R (and friends). The American Statistician 72(1), 80-88.](https://doi.org/10.1080/00031305.2017.1375986)

* Prana, G.A.A., Treude, C., Thung, F. et al. (2019) [Categorizing the Content of GitHub README Files.](https://link.springer.com/article/10.1007/s10664-018-9660-3#citeas) Empir Software Eng 24, 1296–1327 https://doi.org/10.1007/s10664-018-9660-3

* The Turing Way Community, Becky Arnold, Louise Bowler, Sarah Gibson, Patricia Herterich, Rosie Higman, … Kirstie Whitaker. (2019, March 25). [The Turing Way: A Handbook for Reproducible Data Science (Version v0.0.4). Zenodo. http://doi.org/10.5281/zenodo.3233986](https://the-turing-way.netlify.app/)

]

---

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[

#### Other Videos - Talks - Books - Blogs

* [Improve your workflow for reproducible science - Mine Çetinkaya-Rundel](https://mine-cetinkaya-rundel.github.io/improve-repro-workflow-reproducibilitea-2020/slides/improve-repro-workflow-reproducibilitea-2020.pdf)

* [How To Make Your Data Analysis Notebooks More Reproducible - Kartik Rham](https://rstudio.com/resources/rstudioconf-2019/a-guide-to-modern-reproducible-data-science-with-r-karthik-ram/)

* [Reproducible Environments - RStudio](https://environments.rstudio.com/)

* [renv: Project Environments with R - RStudio blog](https://blog.rstudio.com/2019/11/06/renv-project-environments-for-r/)

* [ROpenSci - Reproducibility in Science A Guide to enhancing reproducibility in scientific results and writing](https://ropensci.github.io/reproducibility-guide/)

* [Putting the R into Reproducible Research - Anna Krystalli](https://annakrystalli.me/talks/r-in-repro-research.html#1)
]

---

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[

#### Illustrations

* The Turing Way Community, & Scriberia. (2020, March 3). Illustrations from the Turing Way book dashes. Zenodo. http://doi.org/10.5281/zenodo.3695300

]

---
class: center, middle

## Thank you!

.pull-right[ 
<img src="https://res.cloudinary.com/flor/image/upload/w_1000,c_fill,ar_1:1,g_auto,r_max/v1614343937/flor_pic_ceyztn.jpg" width="250" />
]

---
background-image:  url(imagenes/75581.jpg)
background-size: cover
class: bottom

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[

**Link to the slides**: [bit.ly/reprohack_toronto](bit.ly/reprohack_toronto)

<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.

[Cover image from: rawpixel.com - www.freepik.es](www.freepik.es)
]

]