Skip to main navigation Skip to search Skip to main content

Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses

  • Stefano Coretta
  • , Joseph v. Casillas
  • , Simon Roessig
  • , Michael Franke
  • , Byron Ahn
  • , Ali h. Al-Hoorie
  • , Jalal Al-Tamimi
  • , Najd e. Alotaibi
  • , Mohammed k. Alshakhori
  • , Ruth m. Altmiller
  • , Pablo Arantes
  • , Angeliki Athanasopoulou
  • , Melissa m. Baese-Berk
  • , George Bailey
  • , Cheman baira a Sangma
  • , Eleonora j. Beier
  • , Gabriela m. Benavides
  • , Nicole Benker
  • , Emelia p. Bensonmeyer
  • , Nina r. Benway
    • Grant m. Berry, Liwen Bing, Christina Bjorndahl, Mariška Bolyanatz, Aaron Braver, Alicia m. Brown, Alicia m. Brown, Alejna Brugos, Erin m. Buchanan, Tanna Butlin, Andrés Buxó-Lugo, Coline Caillol, Francesco Cangemi, Christopher Carignan, Sita Carraturo, Tiphaine Caudrelier, Eleanor Chodroff, Michelle Cohn, Johanna Cronenberg, Olivier Crouzet, Erica l. Dagar, Charlotte Dawson, Carissa a. Diantoro, Marie Dokovova, Shiloh Drake, Fengting Du, Margaux Dubuis, Florent Duême, Matthew Durward, Ander Egurtzegi, Mahmoud m. Elsherif, Janina Esser, Emmanuel Ferragne, Fernanda Ferreira, Lauren k. Fink, Sara Finley, Kurtis Foster, Paul Foulkes, Rosa Franzke, Gabriel Frazer-Mckee, Robert Fromont, Christina García, Jason Geller, Camille l. Grasso, Pia Greca, Martine Grice, Magdalena s. Grose-Hodge, Amelia j. Gully, Caitlin Halfacre, Ivy Hauser, Jen Hay, Robert Haywood, Sam Hellmuth, Allison i. Hilger, Nicole Holliday, Damar Hoogland, Yaqian Huang, Vincent Hughes, Ane Icardo isasa, Zlatomira g. Ilchovska, Hae-Sung Jeon, Jacq Jones, Mágat n. Junges, Stephanie Kaefer, Constantijn Kaland, Matthew c. Kelley, Niamh e. Kelly, Thomas Kettig, Ghada Khattab, Ruud Koolen, Emiel Krahmer, Dorota Krajewska, Andreas Krug, Abhilasha a. Kumar, Anna Lander, Tomas o. Lentz, Wanyin Li, Yanyu Li, Maria Lialiou, Ronaldo m. Lima, Justin j. h. Lo, Julio cesar Lopez otero, Bradley Mackay, Bethany Macleod, Mel Mallard, Carol-Ann mary Mcconnellogue, George Moroz, Mridhula Murali, Ladislas Nalborczyk, Filip Nenadić, Jessica Nieder, Dušan Nikolić, Francisco g. s. Nogueira, Heather m. Offerman, Elisa Passoni, Maud Pélissier, Scott j. Perry, Alexandra m. Pfiffner, Michael Proctor, Ryan Rhodes, Nicole Rodríguez, Elizabeth Roepke, Jan p. Röer, Lucia Sbacco, Rebecca Scarborough, Felix Schaeffler, Erik Schleef, Dominic Schmitz, Alexander Shiryaev, Márton Sóskuthy, Malin Spaniol, Joseph a. Stanley, Alyssa Strickler, Alessandro Tavano, Fabian Tomaschek, Benjamin v. Tucker, Rory Turnbull, Kingsley o. Ugwuanyi, Iñigo Urrestarazu-Porta, Ruben Van de vijver, Kristin j. Van engen, Emiel Van miltenburg, Bruce xiao Wang, Natasha Warner, Simon Wehrle, Hans Westerbeek, Seth Wiener, Stephen Winters, Sidney g.-J. Wong, Anna Wood, Jane Wottawa, Chenzi Xu, Germán Zárate-Sández, Georgia Zellou, Cong Zhang, Jian Zhu, Timo b. Roettger
  • Department of Linguistics and English Language, University of Edinburgh, Edinburgh, United Kingdom
  • Department of Spanish and Portuguese, Rutgers University, New Brunswick, New Jersey
  • Department of Linguistics, Cornell University, Ithaca, New York
  • Department of General and Computational Linguistics, University of Tübingen, Tübingen, Germany
  • Program in Linguistics, Princeton University, Princeton, New Jersey
  • Jubail English Language and Preparatory Year Institute, Royal Commission for Jubail and Yanbu, Jubail, Saudi Arabia
  • Université Paris Cité, Laboratoire de Linguistique Formelle, CNRS, Paris, France
  • School of Education, Communication and Language Sciences, Newcastle University, Newcastle upon Tyne, United Kingdom
  • Department of Linguistics, University of Arizona, Tucson, Arizona
  • Psychological and Brain Sciences, Washington University in St. Louis, St. Louis, Missouri
  • Departamento de Letras, Universidade Federal de São Carlos, São Carlos, Brazil
  • School of Languages, Linguistics, Literatures and Cultures, University of Calgary, Calgary, Alberta, Canada
  • Department of Linguistics, University of Oregon, Eugene, Oregon
  • Department of Language and Linguistic Science, University of York, York, United Kingdom
  • Department of Psychology, University of California, Davis, Davis, California
  • Institute of Phonetics and Speech Processing, University of Munich, Munich, Germany
  • Department of Linguistics, University of Pennsylvania, Philadelphia, Pennsylvania
  • Department of Communication Sciences and Disorders, Syracuse University, Syracuse, New York
  • Department of Spanish, Villanova University, Villanova, Pennsylvania
  • Department of English Language and Linguistics, University of Birmingham, Birmingham, United Kingdom
  • Department of Philosophy, Carnegie Mellon University, Pittsburgh, Pennsylvania
  • Department of Spanish and French Studies, Occidental College, Los Angeles, California
  • Department of English, Texas Tech University, Lubbock, Texas
  • Department of Spanish and Portuguese, University of Arizona, Tucson, Arizona
  • Division of Mathematics and Computer Science, Simmons University, Boston, Massachusetts
  • Analytics, Harrisburg University of Science and Technology, Harrisburg, Pennsylvania
  • Department of Psychology, University at Buffalo, Buffalo, New York
  • CLILLAC-ARP, Université Paris Cité, Paris, France
  • IfL-Phonetik, University of Cologne, Cologne, Germany
  • Department of Speech, Hearing and Phonetic Sciences, University College London, London, United Kingdom
  • Basque Center on Cognition Brain and Language, Donostia, Spain
  • Department of Linguistics, University of California, Davis, Davis, California
  • Laboratoire de Linguistique de Nantes (UMR6310), Nantes Université, CNRS, Nantes, France
  • Department of Linguistics and TESOL, University of Texas at Arlington, Arlington, Texas
  • School of Psychology, Newcastle University, Newcastle upon Tyne, United Kingdom
  • School of Psychological Sciences and Health, University of Strathclyde, Glasgow, United Kingdom
  • School of English Literature, Language and Linguistics, Newcastle University, Newcastle upon Tyne, United Kingdom
  • Department of Comparative Language Science, University of Zurich, Zurich, Swizerland
  • Department of Linguistics, University of Canterbury, Christchurch, New Zealand
  • IKER (UMR 5478), CNRS, Bayonne, France
  • Department of Neuroscience, Psychology and Behaviour, University of Leicester, Leicester, United Kingdom
  • Statistics Group, Association for Diversity in Linguistics, Cologne, Germany
  • Department of Music, Max Planck Institute for Empirical Aesthetics, Frankfurt, Germany
  • Department of Psychology, Pacific Lutheran University, Parkland, Washington
  • Department of Languages, Linguistics, and Translation, Université Laval, Quebec City, Quebec, Canada
  • New Zealand Institute of Language, Brain and Behaviour, University of Canterbury, Christchurch, New Zealand
  • Department of Languages, Literatures, and Cultures, Saint Louis University, St. Louis, Missouri
  • Department of Psychology, Princeton University, Princeton, New Jersey
  • LPC, Aix-Marseille Université, CNRS, France
  • Ao Tawhiti Unlimited Discovery, Christchurch, New Zealand
  • Department of Speech, Language, and Hearing Sciences, University of Colorado Boulder, Boulder, Colorado
  • Department of Linguistics and Cognitive Science, Pomona College, Claremont, California
  • California State University Los Angeles
  • Department of Modern and Classical Languages and Literatures, California State University, Northridge, California
  • School of Psychology, University of Birmingham, Birmingham, United Kingdom
  • School of Humanities, Language and Global Studies, University of Central Lancashire, Preston, United Kingdom
  • Programa de Pós-Graduação em Letras, Federal University of Rio Grande do Sul, Porto Alegre, Brazil
  • Department of Linguistics, University of Washington, Seattle, Washington
  • Department of Communication and Cognition, Tilburg University, Tilburg, the Netherlands
  • Department of Linguistics and Basque Studies, University of the Basque Country UPV/EHU, Vitoria-Gasteiz, Spain
  • Department of Psychology, Bowdoin College, Brunswick, Maine
  • Linguistic Convergence Laboratory, HSE University, Moscow, Russia
  • Institute of German Language I Linguistics, University of Cologne, Cologne, Germany
  • Department of English Language Studies, Federal University of Ceará, Fortaleza, Brazil
  • Department of Hispanic Studies, University of Houston, Houston, Texas
  • Department of English and American Studies, University of Salzburg, Salzburg, Austria
  • School of Linguistics and Language Studies, Carleton University, Ottawa, Ontario, Canada
  • Population Health Sciences Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, United Kingdom
  • Speech and Language Therapy, University of Strathclyde, Glasgow, United Kingdom
  • Department of Psychology, Faculty of Media and Communications, Singidunum University, Belgrade, Serbia
  • Institute of Linguistics, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
  • Graduate Program in Linguistics, Federal University of Ceará, Fortaleza, Brazil
  • Department of World Languages, Literatures and Cultures, University of Arkansas, Fayetteville, Arkansas
  • Deparment of Linguistics, Queen Mary University of London, London, United Kingdom
  • Department of Linguistics, University of Alberta, Alberta, Canada
  • Department of Linguistics, University of California, Berkeley, Berkeley, California
  • Department of Linguistics, Macquarie University, Sydney, Australia
  • Center for Cognitive Science, Rutgers University, Piscataway, New Jersey
  • Department of Speech, Language, and Hearing Sciences, Saint Louis University, St. Louis, Missouri
  • Department of Psychology, Witten/Herdecke University, Witten, Germany
  • Department of Linguistics, University of Colorado Boulder, Boulder, Colorado
  • Clinical Audiology, Speech and Language Research Centre, Queen Margaret University Edinburgh, Edinburgh, United Kingdom
  • Department of English and American Studies, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
  • Department of Linguistics, University of British Columbia, Vancouver, British Columbia, Canada
  • Department of Psychiatry and Psychotherapy, University Hospital Cologne, Cologne, Germany
  • Department of Linguistics, Brigham Young University, Provo, Utah
  • Department of Neuroscience, Max Planck Institute for Empirical Aesthetics, Frankfurt, Germany
  • Department of General Linguistics, University of Tübingen, Tübingen, Germany
  • Department of English and Literary Studies, University of Nigeria, Nsukka, Nigeria
  • Chinese and Bilingual Studies, Hong Kong Polytechnic University, Hong Kong SAR, China
  • Department of Modern Languages, Carnegie Mellon University, Pittsburgh, Pennsylvania
  • Geospatial Research Institute, University of Canterbury, Christchurch, New Zealand
  • Département de Lettres Modernes, LIUM, LST, Le Mans Université, Le Mans, France
  • Faculty of Linguistics, Philology and Phonetics, University of Oxford, Oxford, United Kingdom
  • Department of Spanish, Western Michigan University, Kalamazoo, Michigan
  • School of Information, University of Michigan, Ann Arbor, Michigan
  • Department of Linguistics and Scandinavian Studies, University of Oslo, Oslo, Norway
  • Institute of Linguistics, University of Cologne, Cologne, Germany
  • Department of Languages, Literature and Communication, Utrecht University, Utrecht, the Netherlands

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.
Original languageEnglish
Pages (from-to)1-29
JournalAdvances in Methods and practices in Psychological Science
Volume6
Issue number3
Early online date1 Jul 2023
DOIs
Publication statusPublished - 2023

Bibliographical note

Publisher Copyright:
© The Author(s) 2023.

Funding

S. Coretta was partially supported by European Research Council Grant No. 742289 (2017–2022; to J. Harrington); J. Al-Tamimi was partially supported by French Investissements d’Avenir - Labex EFL Grant ANR-10-LABX-0083, contributing to IdEx Université Paris Cité Grant ANR-18-IDEX-0001; G. Moroz and S. Carraturo received funding from the Basic Research Program at the National Research University Higher School of Economics; I. Urrestarazu-Porta received funding from French National Research Agency Grant ANR-20-CE27-0007 and Spanish Ministry of Science and Innovation Grant PID2020-118445GB-I00; C. Kaland, M. Grice, F. Cangemi, M. Lialiou, M. Spaniol, and S. Wehrle received funding from German Research Foundation Grant 281511265-SFB 1252; and N. R. Benway was supported in part through computational resources provided by Syracuse University Grants NSF ACI-1341006 and NSF ACI-1541396.

FundersFunder number
French Investissements d’Avenir
Syracuse University
European Research Council742289 (2017–2022
Deutsche Forschungsgemeinschaft281511265-SFB 1252
Agence Nationale de la RechercheANR-20-CE27-0007
LabexANR-18-IDEX-0001, ANR-10-LABX-0083
Norsk SykepleierforbundACI-1541396, ACI-1341006
Ministerio de Ciencia e InnovaciónPID2020-118445GB-I00
National Research University Higher School of Economics

    Keywords

    • acoustic analysis
    • crowdsourcing science
    • data analysis
    • scientific transparency
    • speech

    Access to Document

    Other files and links

      Fingerprint

      Dive into the research topics of 'Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses'. Together they form a unique fingerprint.
      View full fingerprint

      Cite this