Open Science –Introduction · Open Science in the research process Formulate hypotheses &...

OpenScience– Introduction

OpenScienceWorkshop

moreinformationavailableonthelastslide

OpenScienceϵ Good Science

Goodresearchpractices OpenScience

Pillars of OpenScience

Ope

nScience

OpenData

OpenMaterial

OpenAccess

OpenSource(Software)

OpenPeerReview

OpenEducationalResources

Whytransparency?

Formulatehypotheses&analysisplan

Collectdata

Analyzedataaccordingtoanalysisplan

Interpret&reportresults

Replicateresults

Publish&distributeresearchoutput

TheConfirmatoryResearchProcess

Wagenmakers etal.(2012)

Whytransparency?


Collectdata

Analyzedataaccordingtoanalysisplan

Interpret&reportresults

Replicateresults


Howcanyouknowthatitdoesnotlooklikethis?

interesting*

butonlywiththosewhopay

*p<.05;thatfitatheory;thataresurprising/publishable…

inventsomeshinynewhypotheses

orfakeit

OpenScienceintheresearchprocess


Collectdata

AnalyzedataInterpret&reportresults

Replicateresults

Preregistration

OpenLabNotebook


RegisteredReport(1st phase)

OpenAnalysisCode

OpenDataOpenMaterials

OpenAccess

RegisteredReport(2nd phase)

Replicationstudy

getallmaterialhere:https://osf.io/zjrhu/

PowerAnalysis

Whytransparency?

Increasetrustinscience,don’twastepublicresources

Getconstructivefeedback

Beinternationalandinclusive

Increasethespeedofdiscovery

Picturesfromfreepik.comby@brgfx,@makyzz;flaticon.combyIconPond,Dimitry Miroliubov

However…

Thepresentsituationisnotsobright…

PublicationBias

Begg &Berlin(1988)

Definition:

“Thephenomenoninwhichstudieswithpositiveresultsaremorelikelytobepublishedthanstudieswithnegativeresults.”

Study

significant notsignificant

published notpublished published notpublished

p=80% p=20% p=20% p=80%

PublicationBias:Efficacyofanti-depressants(Turneretal.2008;Meta-Analysiswithk =74)

https://twitter.com/eturnermd1/status/737436322344927232Turner,E.H.,Matthews,A.M.,Linardatos,E.,Tell,R.A.,&Rosenthal,R.(2008).SelectivePublication of Antidepressant Trialsand Its Influence onApparent Efficacy.NewEnglandJournalof Medicine,358,252–260.doi:10.1056/NEJMsa065779

Trialspublished injournals:48positive,3negative

TrialsregisteredatFDA:38positive,36negative

QuestionableResearchPractices

ScientificMisconduct

Iconfrom flaticon.comby Smashicons

Fabrication(Makingupdata)

Falsification(Distortingdata)

QuestionableResearchPractices(p-hacking,HARKing,selectivereporting...)

HunterCollege(2016),Johnetal.(2012)

QuestionableResearchPractices

Simonsohn etal.,2014;Johnetal.(2012)

p-hacking (n.).Tuneyour data analysis inaway that youachieve asignificant p-value insituations where it wouldhave been non-significant.

Questionable research practices (QRPs)(n.).Practicesofdata collection and data analysis that are notoutrightfraud,butalsonotreally kosher.

Picturesfrom pxhere.com/de/photo/494938,pxhere.com/de/photo/634757

OptionalStopping

John,Loewenstein,&Prelec (2012)

Initialsamplesize

p<0.05?

Collectsomemoredata

Publishresults

Toolsforp-Hacking

OptionalStopping

Armitage,P.,McPherson,C.K.,&Rowe,B.C.(1969).Repeatedsignificancetestsonaccumulatingdata.JournaloftheRoyalStatisticalSociety.SeriesA(General),132,235–244.

Toolsforp-Hacking

HARKing:Hypothesizingaftertheresultsareknown

Wagenmakers (2018);Kerr(1998)©ChrisHankin

Toolsforp-Hacking

Outcomeswitching

Toolsforp-Hacking

http://compare-trials.org/ http://blogs.discovermagazine.com/neuroskeptic/2015/07/23/social-priming-money-for-nothing/#.VuKRSRi5KJM

• 2outcomevariables:false positiverate5%➙ 9.5%

• 5outcome variableswith one-sidedtesting:false positiverate5%➙ 41%

Thegardenofforkingpaths/Analyticaldegreesoffreedom

John,Loewenstein,&Prelec (2012),Gelman &Loken (2013)

Toolsforp-Hacking

Data

Typeofoutlierrejection

Testequalvarianceassumption?

Variableencoding

Usearobuststatistic

p<0.05

Elson(2016):http://www.flexiblemeasures.com

Thereare currently 130publications inwhichresultsarebasedontheCompetitiveReactionTimeTask,andtheyreported 156differentquantificationstrategies intotal!

Toolsforp-Hacking

Intentional?• Evilresearcherwhoonlycaresabouthis/hercareerandnotat

allabouttruth-seeking?

Unintentional?• Wrongeducation?• Wrong/uncriticalstandardsofthefield?• Pushedbysupervisors,reviewers,oreditors?

• à Distortingeffectsonthepublishedrecordareprobablycomparable,buttheethicalevaluationsdiffersstrongly.

P-Hacking

Wemighthaveareproducibilitycrisis

Baker(2016)

90%YES(thereisacrisis)

Wemighthaveareplicationcrisis

36% 49%

11% 21%

78%

64% 51%

89% 79%

22%

Psychology(2015;N=97)

Economics*(2015;N=67)

Cancerresearch1

(2011;N=53)

Cancerresearch2

(2012;N=67)

ExperimentalPhilosophy(2018;N=40)

NotReplicated

Replicated

*Thedataoneconomicsisaboutreproducibility;i.e.theattempttogetthesameresultsifyouapplytheoriginaldataanalysisontheoriginaldataset.OpenScienceCollaboration(2015);Chang&Li(2015);Begley&Ellis(2012);Prinz etal.(2011);Cova etal.(2018)

Whatcanyoudo?

PublicationbiasQRPs/p-hacking

Iconfrom flaticon.comby Baianat,en.wikipedia.org/wiki/File:We_Can_Do_It!.jpg

Scientificprogressissloweddown

Publishedresultscannotbetrusted

Resourcesarewasted

(1) IdentifyQuestionableResearchPractices(2) PracticeOpenScience:Makeyourown

researchtrustworthy(3) Helptochangeincentivestructures

TheOpenResearchProcess

StudyDesign DataCollection Publication&Distribution Replication

Preregistration

“Thespecificationofaresearchdesign,hypotheses,andanalysisplanpriortoobservingtheoutcomesofastudy”

Nosek &Lindsay(2018)

Why?• PreventHARKing• Reduceanalyticalflexibility• Makeselectivereportingvisible• Getearlyfeedback• Takecreditforyourideas• Regulatoryagenciesrequireit



ApplyforRegisteredReport

“RegisteredReportsareaformofempiricaljournalarticleinwhichmethodsandproposedanalysesarepre-registeredandpeer-reviewedpriortoresearchbeingconducted.High-qualityprotocolsarethenprovisionallyacceptedforpublicationbeforedatacollectioncommences.”

COS(2018):www.cos.io/rr/

Why?• Advantagesofpreregistration• Guaranteedpublicationindependent

ofresults• Peerreviewforyourdesign



OpenLabNotebook

“Researchersusealabnotebooktodocumenttheirhypotheses,experiments,andinitialanalysisorinterpretationoftheseexperiments.Thelabnotebookservesasanorganizationaltool,amemoryaid,andcanhavearoleinprotectingintellectualpropertythatcomesfromtheresearch.”

Goyal,Malviya,&Kapoor (2012)

Why?• Gain&shareproceduralknowledge• Increaseauthenticity• Protectyourintellectualproperty



OpenData

“Opendatashouldbeavailabletoeveryonetoaccess,use,andshare.”

GOFAIR(2018)

Why?• Makeyouranalysesreproducible• Enablere-useofdataforanswering

otherresearchquestions• Neverlosevaluabledatainafile

drawer• Fundingagenciesrequireit



OpenMaterials

“Makingcomponentsoftheresearchmethodologyneededtoreproducethereportedprocedureandanalysispubliclyavailable.”

OSF(2016)

Why?• Makeyourstudyreproducible• Enablere-useofmaterialsforother

experiments



OpenAccess

Literaturewhichis“digital,online,freeofcharge,andfreeofmostcopyrightandlicensingrestrictions”.

Suber (2015)

Why?• Enablefasterprogressinresearchby

openingtheaccesstoknowledge• Givebackvaluetothecommunity

thatfundedyouandnotonlytopublishers



OpenAnalysisCode

“Clean,repeatable,script-basedworkflow[…]thatlinksrawdatathroughtocleandataandtofinalanalysisoutputs.”

BritishEcologicalSociety(2017)

Why?• Enableotherstoreproduceyour

analyses• Understandyourowncode(after

sometime)• Recreateyourresultswithoneclick



Replication

“replicationisascientificmethodtoverifyresearchfindingsand[…]referstoarepetitionofaresearchproceduretochecktheaccuracyortruthofthefindingsreported.”

Schmidt(2009)

Why?• Enhancecredibilityofyourresearch• Gainconfidenceinyourfindings&

solidifythebasisofyourresearch

Let’snotrestonourlaurels:Currentchallenges.

1. Blindspots2. Highopenness,lowquality3. Empiricalevidenceforeffectivenessofreforms4. Incentivestructures


1. Blind spots2. Highopenness,lowquality3. Empiricalevidenceforeffectivenessofreforms4. Incentivestructures

Leaveyourbubble!

• Blindspotswithinpsychology?• Project:AnalyseconferenceprogramsofthesubsectionsoftheDGPs(„Fachgruppen“)forkeywordssuchasreplication,reproducibility,open science.

• Leaveyourbubbleandspreadthewordtoyourcommunity.

35


1. Blindspots2. High openness, low quality3. Empiricalevidenceforeffectivenessofreforms4. Incentivestructures

Paperdoesnotmatchpreregistration

38

https://twitter.com/BrianNosek/status/1003989856643674113

„However,theanalysisplanwaspostedtoOSFbutunfortunatelynotactuallyregistered“

„Also,theStudy3designwaspartoftheregistration,butitdidnotincludeananalysisplan.“

➙ half-way preregistration?

„Oneofthefivestudies(Study3)waspreregistered“

➙ Preregister one trivial research question, get the badge for the whole paper?

Open-washing

Open-washing

39https://twitter.com/jamesheathers/status/1004330301626208256https://www.sciencedirect.com/science/article/pii/S0195666317313041

Whoisresponsibleforchecking/enforcingthebadges?

•„Self-disclosure model“:Authorssignthestatement„Ihaveapreregistrationandmypapermatchestheprereg“,butverificationislefttocommunity(inpost-publicationpeerreview)➙ badgemeans:„Thisisverifiablein principle“(butsomebodystillhastodoit)•„verification model“:Reviewersandoreditorsdotheverification➙ badgemeans:„Thishasbeenverifiedandcanbetrusted“(butextraburdenforreviewersandeditors)•RegisteredReportsasamuchbettermodel?Preregistrationisthepaper,nomismatchpossible.Reviewerscheckitduringstage1review.

FAIRdata•Findable:Metadataanddatashouldbeeasytofindforbothhumansandcomputers.•Accessible:Oncetheuserfindstherequireddata,she/heneedstoknowhowcantheybeaccessed,possiblyincludingauthenticationandauthorisation.•Interoperable:Thedatausuallyneedtobeintegratedwithotherdata.Inaddition,thedataneedtointeroperatewithapplicationsorworkflowsforanalysis,storage,andprocessing.•Reusable:Metadataanddatashouldbewell-describedsothattheycanbereplicatedand/orcombinedindifferentsettings.

41

https://www.go-fair.org/fair-principles/

OpenDatavs.FAIRdata

• FAIRdatacanbenot open–e.g.,ifadatasetisfindable,reuseable,etc.,butonlyaccessiblewithinaclosedresearchgroup

•OpenDatacanbenot FAIR–e.g.,anundocumenteddatadumpinanuncuratedrepository,suchasOSF,whichisneitherfindable,norreuseable,norinteroperable

• FAIRdimensionsarequalitycriteriathatcanbeappliedtodatasets.Ideally,adatasetisopenandFAIR.

42

https://ask-open-science.org/1116/what-the-difference-between-fair-data-and-open-data-there-any

Open-Washing=Hey,let’sgamethenewsystem!

EndorseopenscienceonTwitterandyourCV,trytogetbadgeswithminimumeffort,pretendopennessbutdonotdeliver.


1. Blindspots2. Highopenness,lowquality3. Empirical evidence for effectiveness of reforms4. Incentivestructures

Meta-Science:Gatherempiricalevidence

•Hypothesis:Opensciencepracticesincreasethecredibility,theveridicality*,andthereplicabilityofresearch.•Acriticcouldsay:Where is the empirical evidence? You rush implementing all these interventions and reforms without having any evidence that they actually have the desired effect.

*thedegreetowhichatheoryorinterpretationaccuratelyrepresentsreality

Meta-Science:Gatherempiricalevidence

46


1. Blindspots2. Highopenness,lowquality3. Empiricalevidenceforeffectivenessofreforms4. Incentive structures

48

RichardHorton,EditorvonThe Lancet

Quantity,notquality

49

Abele-Brehm, A. E., & Bühner, M. (2016). Wer soll die Professur bekommen? Psychologische Rundschau, 67(4), 250–261. http://doi.org/10.1026/0033-3042/a000335

Actual(notdesired)relevanceatprofessorshiphiringcommittees: Rank

Number ofpeer-reviewedpublications 1Fitofresearchprofiletotheadvertisinginstitution 2Qualityofresearchtalk 3Number ofpublications 4Volume ofacquiredthird-partyfunding 5Number offirstauthorships 6… …

50

Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7(6), 543–554. http://doi.org/10.1177/1745691612459060Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3(9), 160384–17. http://doi.org/10.1098/rsos.160384

Idealstrategyforahighquantityofpublications:smalln +manystudies+questionableresearchpractices(QRPs),suchasp-hacking

„The rules of the game“ „Evolution of bad science“

Quantity,notquality

51


Actual(notdesired)relevanceatprofessorshiphiringcommittees: Rank

Number ofpeer-reviewedpublications 1Fitofresearchprofiletotheadvertisinginstitution 2Qualityofresearchtalk 3Number ofpublications 4Volume ofacquiredthird-partyfunding 5Number offirstauthorships 6… …Quality assessment ofthebestthreepublications 17… …Indicators of research transparency 41 (of 41)

Quality,notquantity

52


Job committees

Roadmap

Fastadoptionvs.High(FAIR)quality?

• Lowhurdles,onesmallstepatatime• Rewardsmallsteps

Sharing something - even badly documented data - is better than sharing nothing.

• LearningbydoingWith increasing practice, hopefully the quality gets better, too.

• But:(Initially)LowqualityBarely reusable data sets; trying to reproduce a result is a pain in the ass or impossible; data reuse very limited.

• Riskof„open-washing“Pretending openness without actual value.

• HighhurdlesMainlyenthusiasts/computerscientistswillableandmotivateduseit

• RewardbigstepsCurated repositories with input quality control.

• InstanthighqualityThe data sets which are open are instantly FAIR.

Hiring committees: Make „open science“ a desirable or essential job characteristic

55

https://docs.google.com/document/d/1ty43Syw0Flkh8ncjW8MZArIkvYe8hLwwhLlIwbtSk_Y/edit?usp=drive_web&ouid=108982640291853577145

…

…

Since 2015: All professorship job descriptionsuse this requirement

Seemoresuchprofjobadsat:https://osf.io/7jbnt/

Dissertation agreement

Xhttp://www.fak11.lmu.de/dep_psychologie/osc/dissertation_agreement/index.html

Open science in curricula

Xhttp://www.fak11.lmu.de/dep_psychologie/studium/lehrelounge/kerncurriculum_empra/index.html

Hiring committees: Require an annotated CV with limited items (e.g., <= 10)

Dougherty, M. R., Slevc, L. R., & Grand, J. (2018, February 2). Retrieved from psyarxiv.com/48qux

Authors&title Year Cit-

ationsSamplesizeperstudy

p-valueperstudy

OpenScienceindicators Dataset Applicants

contribution

Doe,John&Smith,Peter

2001 47n₁=21n₂=30n₃=19

p₁=.048p₂=.050p₃=.023

☐OpenData☐OpenMaterial☐Preregistered

☑Owndatacollection➙URLNA☐Archivaldata

• Analyzeddata

• Wrotemanuscript

Doe,John 2016 26 n₁=180n₂=158

p₁=.012p₂=.001

☑ OpenData☑OpenMaterial☑Preregistered

☑ Owndatacollection➙URLosf.io/as1cd☐ Archivaldata

• Designedstudy

• Wrotemanuscript

Paper-level citation metrics

Basic information for judging

evidential value

No journal; JIF is irrelevant or misleading

Open science indicators: Judging

reproducibility

Data: own collection or

reuse?

OpenScience:Whoto ask?

• Ask OpenScienceInitiative(Universityof Bielefeld)https://ask-open-science.org/

• Your local OpenScienceInitiativehttps://osf.io/tbkzh/wiki/home

• Reddit OpenSciencehttps://www.reddit.com/r/Open_Science/

OpenScience:3EasySteps

HowyoucanimproveyourOSrecord(almost)withouteffort1. Whenreviewingapaper:KeepaneyeonQRPsandaskfor

opendataandopenmaterial(https://opennessinitiative.org/)2. Whichpartsofyourresearchprocesscanyoumakeopen?

Startoutwiththeleastwork-intensivepartandgiveitatry!3. GetaTwitteraccountandjointhediscussion

OpenScience:What you learned

• OpenScienceas part of good research practice• Thescience hamsterwheel:Incentive structures inresearch• Thereplication crisis:Non-replicability inresearch and its

problems• Identify scientific misconduct:Fabrication,Falsification,

Questionable ResearchPractices• Methods of p-hacking:Optionalstopping,HARKing,selective

reporting,analytical flexibility• Theopenresearch process:Preregistration,Registered

Reports,OpenLabNotebooks,OpenData,OpenMaterials,OpenAccessPublishing,ReplicationStudies

• How to make achange:OpenSciencenetworks and initiatives

FurtherResources

• Arslan,R.(2018).Openscience vs.bad science:Thereplication crisis and possible reforms.Presentationslides available onosf.io/65mqz/

• Gelman,A.,&Loken,E.(2013).Thegardenofforkingpaths:Whymultiplecomparisonscanbeaproblem,evenwhenthereisno“fishingexpedition”or“p-hacking”andtheresearchhypothesiswaspositedaheadoftime. DepartmentofStatistics,ColumbiaUniversity.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf

• John,L.K.,Loewenstein,G.,&Prelec,D.(2012).Measuringtheprevalenceofquestionableresearchpracticeswithincentivesfortruthtelling.PsychologicalScience23(5),p.524-532.doi:10.1177/0956797611430953

• Schmidt,S.(2009).Shall we really doit again?Thepowerfulconcept of replication is neglected inthesocial sciences.Reviewof GeneralPsychology 13(2),p.90-100.doi:10.1037/a0015108

• Schönbrodt,F.D.(2017).P-hacking:What it is,how to prevent it.Presentation at GESIS,slides available ongesis.org/fileadmin/upload/events/Vortragsreihe/Schoenbrodt_GESIS_p-hacking.pdf

• Simmons,J.,Nelson,L.D.,&Simonsohn,U.(2011).False-positivepsychology:Undisclosedflexibilityindatacollectionandanalysisallowspresentinganythingassignificant.PsychologicalScience22(11),p.1359-1366.doi:10.1177/0956797611417632

• Wagenmakers ,E.-J.(2018).Thecaseforradicaltransparencyinstatisticalreporting.Presentationslidesavailableonbayesianspectacles.org/wp-content/uploads/2018/01/RadicalTransparency.pdf

References• Arslan,R.(2018).Openscience vs.bad science:Thereplication crisis and possible reforms.Presentation slides available onosf.io/65mqz/• Baker,M.(2016).1.500scientists lift the lid onreproducibility.Nature533(7604),p.452-454.doi:10.1038/533452a• BritishEcological Society(2017).Aguide to reproducible code inecology and evolution.Reportavailable onwww.britishecologicalsociety.org/wp-content/uploads/2017/12/guide-to-reproducible-

code.pdf• COS(2018).RegisteredReports:Peerreviewbeforeresultsareknowntoalignscientificvaluesandpractices.Availableoncos.io/rr/• Cova,F.,Strickland,B.,Abatista,A.,Allard,A.,Andow,J.,Attie,M.,...&Cushman,F.(2018).Estimatingthereproducibilityofexperimentalphilosophy. ReviewofPhilosophyandPsychology,1-36.

psyarxiv.com/sxdah/• *Begg,C.,&Berlin,J.(1988).PublicationBias:AProbleminInterpretingMedicalData. JournaloftheRoyalStatisticalSociety.SeriesA(StatisticsinSociety), 151(3),p.419-463.doi:10.2307/2982993• Begley,C.G.,&Ellis,L.M.(2012).Drugdevelopment:Raisestandardsforpreclinicalcancerresearch. Nature, 483(7391),531.www.gulfcoastconsortia.org/wp-content/uploads/2017/01/Begley-Ellis-

Nature-Comment-2012.pdf• Chang,A.&Li,P.(2015).Chang,A.C.,&Li,P.(2015).Iseconomicsresearchreplicable?Sixtypublishedpapersfromthirteenjournalssay'usually not'.

papers.ssrn.com/sol3/papers.cfm?abstract_id=2669564• Gelman,A.,&Loken,E.(2013).Thegardenofforkingpaths:Whymultiplecomparisonscanbeaproblem,evenwhenthereisno“fishingexpedition”or“p-hacking”andtheresearchhypothesiswas

positedaheadoftime. DepartmentofStatistics,ColumbiaUniversity.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf• GOFAIR(2018).Whatisthedifferencebetween‘FAIRdata’and‘Opendata’ifthereareany?Availableongo-fair.org/faq/ask-question-difference-fair-data-open-data/• Goyal,R.,Malviya,S.,&Kapoor,D.N.(2012).Labnotebook:Anindispensableresourceforresearchers.DHRInternationalJournalOfPharmaceuticalSciences2(1).Availableon

doublehelixresearch.com/DHRIJPS• HunterCollege(2016).Psych250-07:Extracredit3:Datafabrication.Availableonhunterdolkerpsych250.wordpress.com/2016/12/06/extra-credit-3-data-fabrication/• John,L.K.,Loewenstein,G.,&Prelec,D.(2012).Measuringtheprevalenceofquestionableresearchpracticeswithincentivesfortruthtelling.PsychologicalScience23(5),p.524-532.

citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.727.5139&rep=rep1&type=pdf• Mullard,A.(2011).Reliabilityof'newdrugtarget'claimscalledintoquestion.NatureReviewsDrugDiscovery10,p.643-644.doi:10.1038/nrd3545• Nosek,B.A.&Lindsay,S.(2018).Preregistration becoming the norminpsychological science.APSObserver,31(3).Available onpsychologicalscience.org/observer/preregistration-becoming-the-norm-

in-psychological-science• OpenScienceCollaboration (2015).OpenScienceCollaboration.(2015).Estimatingthereproducibilityofpsychologicalscience. Science, 349(6251),aac4716.osf.io/447b3/• OSF(2016).Badges to acknowledge openpractices.Available onosf.io/tvyxz/wiki/1.%20View%20the%20Badges/• Prinz,F.,Schlange,T.,&Asadullah,K.(2011).Believeitornot:howmuchcanwerelyonpublisheddataonpotentialdrugtargets?. NaturereviewsDrugdiscovery, 10(9),712.hopecenter.wustl.edu/wp-

content/uploads/2012/01/Nat-Rev-Drug-Disc-reproducibility-article.pdf• Schmidt,S.(2009).Shall we really doit again?Thepowerfulconcept of replication is neglected inthe social sciences.Reviewof GeneralPsychology 13(2),p.90-100.doi:https://goo.gl/3wEfdz• Schönbrodt,F.D.(2017).P-hacking:What it is,how to prevent it.Presentation at GESIS,slides available ongesis.org/fileadmin/upload/events/Vortragsreihe/Schoenbrodt_GESIS_p-hacking.pdf• Schönbrodt,F.D.(2016).Introducing the p-hacker app:Trainyour expertp-hacking skills.Blogpost available onnicebread.de/introducing-p-hacker/• Simmons,J.,Nelson,L.D.,&Simonsohn,U.(2011).False-positivepsychology:Undisclosedflexibilityindatacollectionandanalysisallowspresentinganythingassignificant.PsychologicalScience22(11),

p.1359-1366.doi:10.1177/0956797611417632• Simonsohn,U.,Nelson,L.D.,&Simmons,J.P.(2014).P-curveandeffectsize:Correctingforpublicationbiasusingonlysignificantresults. PerspectivesonPsychologicalScience, 9(6),666-681.doi:

10.1177/1745691614553988• Suber,P.(2015).OpenAccessOverview.Available onbit.ly/oa-overview• Wagenmakers,E.J.,Wetzels,R.,Borsboom,D.,vander Maas,H.L.,&Kievit,R.A.(2012).Anagendaforpurelyconfirmatoryresearch. PerspectivesonPsychologicalScience, 7(6),632-638.• Wagenmakers ,E.-J.(2018).Thecaseforradicaltransparencyinstatisticalreporting.Presentationslidesavailableonbayesianspectacles.org/wp-content/uploads/2018/01/RadicalTransparency.pdf*Unfortunately,thereisnoOpenAccessversionofthisarticleavailable

Credentials

Thecreation of this workshop materialwaspartially funded by the BerkeleyInitiativefor Transparency intheSocial Sciences (BITSS)Catalyst Program.For more information,please visit www.bitss.org,sign up for the BITSSblog,and [email protected] alsokindly thank the LMUGraduateCenter for their support.

TheseslideswerecreatedbyAngelikaStefan,JuliaBrandt,andFelixSchönbrodt.TheworkislicensedunderaCreativeCommonsAttribution4.0InternationalLicense.Thatmeans,youcanreusethisslidesinyourownworkshops,remixthem,orcopythem,aslongasyouattributetheoriginalcreators.

Open Science –Introduction · Open Science in the research process Formulate hypotheses &...

Documents

Transcript of Open Science –Introduction · Open Science in the research process Formulate hypotheses &...