Category Archives: Research

What if a failed replication… somehow fails to replicate?

Have I told you already that science can be messy? If not, welcome to this blog! 2 years ago I posted this replication of the infamous pencil in the mouth study. It has become one of the more well known examples of the replication crisis. But it also spurred a lot of debate. Was the replication really a true replication of the original research?

A new study adds fuel to this debate as it failed to replicate the failed replication. Ok, just kidding, the study is actually showing the original study might have been correct! But we can’t be really sure, as it’s actually even more complicated:

The paradigm diverged from the original facial feedback experiment in several respects. They include the classroom setting in which testing was conducted; the fact that each participant rated two cartoons rather than four; the fact that it featured a within-subjects rather than between- subjects design; the absence of a cover story about piloting a study for future research regarding populations with disabilities to explain the manipulation; the use of a 7-point scale rather than a 10-point scale; the fact that the experiment was part of a classroom lecture about learning (specifically, about the acquisition of conditioned associations) rather than following a line- drawing task; the fact that correct positioning of pens could be monitored only within the limits of a group setting; the fact that participants selected but did not write down their ratings with their pens in their mouths; and the lack of individualized follow-up with participants regarding their beliefs about the experiment, precluding exclusion of participants for suspicions regarding the study goals. (It is notable, however, that when the instructor presented students with their results in the ensuing class, the most commonly verbalized reaction was surprise or disbelief that the manipulation could have possibly affected their ratings.)

It seems the only thing that we seem to know for sure is that more research is needed…

Abstract of this new study:

The facial feedback effect refers to the influence of unobtrusive manipulations of facial behavior on emotional outcomes. That manipulations inducing or inhibiting smiling can shape positive affect and evaluations is a staple of undergraduate psychology curricula and supports theories of embodied emotion. Thus, the results of a Registered Replication Report indicating minimal evidence to support the facial feedback effect were widely viewed as cause for concern regarding the reliability of this effect. However, it has been suggested that features of the design of the replication studies may have influenced the study results. Relevant to these concerns are experimental facial feedback data collected from over 400 undergraduates over the course of 9 semesters. Circumstances of data collection met several criteria broadly recommended for testing the effect, including limited prior exposure to the facial feedback hypothesis, conditions minimally likely to induce self-focused attention, and the use of moderately funny contemporary cartoons as stimuli. Results yielded robust evidence in favor of the facial feedback hypothesis. Cartoons that participants evaluated while holding a pen or pencil in their teeth (smiling induction) were rated as funnier than cartoons they evaluated while holding a pen or pencil in their lips (smiling inhibition). The magnitude of the effect overlapped with original reports. Findings demonstrate that the facial feedback effect can be successfully replicated in a classroom setting and are in line with theories of emotional embodiment, according to which internal emotional states and relevant external emotional behaviors exert mutual influence on one another. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

1 Comment

Filed under Education, Psychology, Research

A new review study about Collaborative Problem Solving, still a lot to research

I found this new review study via Jeroen Janssen and it’s quite interesting as it tackles Collaborative Problem Solving and how much is already known about how to advance this. Spoiler: not that much, it seems, while it now is being measured e.g. by PISA.

But first what is it? That is already a bit more difficult question to answer, but I do think this description of what is needed makes it more clear:

…team members must be able to define the problem, understand who knows what on the team, identify gaps in what is known and what is required, integrate these to generate candidate solutions, and monitor progress in achieving the group goals. From the social perspective, the success of a team requires that members establish shared understanding, pursue joint and complementary actions, and coordinate their behavior in service of generating and evaluating solutions.

So what is (un)known? This overview at the end of the article with suggestions for further research, makes a lot more clear:

  1. CPS has been identified as an important skill in the international community and workforce, but recent assessments have revealed that students and adults have low CPS proficiency. This calls for an analysis of CPS mechanisms, frequent problems, and methods of solving these problems. Psychological scientists could play a major role in this broad effort by partnering with stakeholders.
  2. CPS is rarely trained in schools and the workforce, and the existing training is not informed by psychological science. This opens the door to the value of psychological scientists’ being part of national and international efforts drawing from their expertise in science, learning and training.
  3. Psychological scientists have developed a body of empirical research and theory of team science over the years, but much of this work has focused on group learning, work, memory, and decision making rather than CPS per se. We need to sort out how much of the existing research in team science applies to CPS. Psychological scientists are encouraged to direct their focus on CPS per se in the team science research landscape.
  4. Intelligent digital technologies have the potential to automatically analyze large samples of group interactions at multiple levels of language, discourse, and interactivity. This is landmark progress because existing research on teams has had small samples and time-consuming annotation of the interactions. There is a need for psychological scientists to partner with the developers of these technologies to recommend psychological characteristics to track and to scrutinize the validity of automated measures.
  5. A curriculum for training CPS competencies has not been developed and adequately tested. There is a need to develop a program of research on CPS curriculum design for both students and instructors. Psychological scientists are an important asset to generate potential curricula and to test their efficacies.

Abstract of the open access review:

Collaborative problem solving (CPS) has been receiving increasing international attention because much of the complex work in the modern world is performed by teams. However, systematic education and training on CPS is lacking for those entering and participating in the workforce. In 2015, the Programme for International Student Assessment (PISA), a global test of educational progress, documented the low levels of proficiency in CPS. This result not only underscores a significant societal need but also presents an important opportunity for psychological scientists to develop, adopt, and implement theory and empirical research on CPS and to work with educators and policy experts to improve training in CPS. This article offers some directions for psychological science to participate in the growing attention to CPS throughout the world. First, it identifies the existing theoretical frameworks and empirical research that focus on CPS. Second, it provides examples of how recent technologies can automate analyses of CPS processes and assessments so that substantially larger data sets can be analyzed and so students can receive immediate feedback on their CPS performance. Third, it identifies some challenges, debates, and uncertainties in creating an infrastructure for research, education, and training in CPS. CPS education and assessment are expected to improve when supported by larger data sets and theoretical frameworks that are informed by psychological science. This will require interdisciplinary efforts that include expertise in psychological science, education, assessment, intelligent digital technologies, and policy.

3 Comments

Filed under Education, Research, Review

Will the effect last? Math on a tablet helps low-performing second graders… for a while

There is a new Best Evidence in Brief with among others, this study that shows how important it can be to check the lasting effects of what you do.

Published in the Journal of Educational Psychology, Martin Hassler and colleagues carried out a randomized controlled trial of a mathematics intervention on tablets (iPads).
The trial involved 283 low-performing second graders spread across 27 urban schools in Sweden. The children were randomized to four groups:
  • A math intervention called Chasing Planets, consisting of 261 planets on a space map, each with a unique math exercise (addition or subtraction up to 12). Students practiced for 20 minutes a day.
  • The math intervention combined with working memory training, where students spent an additional 10 minutes each day on working memory tasks.
  • A placebo group who practiced mostly reading tasks on the tablet (again for 20 minutes each day), including Chasing Planets-Reading, which had a similar format to the math intervention.
  • A control group who received no intervention, not even on improving their skills on the tablets.
The intervention lasted for around 20 weeks, with children completing nine measures at pre- and post-test, and then after 6 and 12 months.
Both math conditions scored significantly higher (effect size = +0.53-0.67) than the control and placebo groups on the post-test of basic arithmetic, but not on measures of arithmetic transfer or problem solving. There was no additional benefit of the working memory training. The effects faded at the 6-month follow-up (effect size = +0.18-0.28) and even more so after 12 months (effect size = +0.03-0.13)
IQ was a significant moderator of direct and long-term effects, such that children with lower IQ benefited more than higher IQ students. Socioeconomic factors did not moderate outcomes.

1 Comment

Filed under Education, Research, Technology

How do teachers react in class to the Trump-presidency

The past 2 months I have been slowing down a bit on reading new research because too much work. Sorry for that. This study raises some interesting questions related to what people on the old continent would describe as pedagogy. Does a teacher need to political neutral in the classroom or not, and is being neutral an effective teaching tactic. Teaching in Trump times made teachers feeling immense pressure from school leaders and families to respond in a certain way — or not at all — in their classrooms.

From the press release:

“There were many teachers who said they wanted to talk with students about the election and related issues but were also afraid of backlash,” said Dunn, who conducted the nationwide questionnaire of more than 700 educators.

In the survey, some teachers said they felt election-related topics weren’t appropriate in schools or weren’t relevant to their subject. Others felt they shouldn’t, or couldn’t, share their political affiliations or feelings.

But the idea of neutrality, as this research indicates, doesn’t always work in schools, because “education is inherently political,” Dunn said.

She and her co-researchers argue that by remaining neutral, teachers are enacting the opposite of neutrality by “choosing to maintain the status quo and further marginalizing certain groups.”

Dunn and her colleagues, Hannah Carson Baggett of Auburn University and Beth Sondel of the University of Pittsburgh, say the election is just one example of a renewed call for all teachers to consider the ethics of neutrality in the classroom.

“Midterm voting and the impact of results are an opportunity for them to say, ‘I’m not going to be neutral,'” Dunn said. “Knowing what neutrality means, and how it can be a disservice to students and to themselves, teachers can think about how to adapt their curriculum leading up to and after the midterms and other major events.”

Dunn said many educators and administrators believe that because something is happening “outside of school,” it isn’t relevant in the classroom. But that mentality is an injustice, she argues, and undermines the fact that the classroom is part of the real world.

In a separate study using the same data, the scholars studied what teachers did – or didn’t do – in the days after the election. In that study, teachers reported their students were experiencing political trauma.

Abstract of the study:

Guided by perspectives on the sociopolitical contexts of schooling, control of teachers’ curriculum and instruction, and teaching of elections, we use findings from a national questionnaire to explore the contexts that shaped teachers’ pedagogical decision making following the 2016 U.S. presidential election. Our findings reveal that classroom, school, district, state, and national contexts often manifested in pressure from colleagues, parents, the administration, the district, and the public. This pressure is reflective of the lack of trust, autonomy, and professionalism for teachers in our current climate. The days immediately following the election revealed new understandings about teachers’ views on neutrality, opportunities for agency within control of teachers’ work, and a call for justice-oriented pedagogy. Implications for teacher education, practice, and research are discussed.

Leave a comment

Filed under Education, Research, Trends

Sad thing we still have to discuss this: spanking does more harm than good

It seems so obvious, but a new study confirms that spanking children is a bad idea and even harmful for children on a more global scale than previously known.

From the press release:

Most research on how spanking affects children has involved studying families in high-income countries, such as the United States and Canada, but less was known about how spanking affects children in low- and middle-income countries–or developing countries.

Spanking is one of the most common forms of child discipline used by parents worldwide.

The new international research used data collected by UNICEF in 62 countries–representing nearly one-third of the world’s countries–and demonstrated that caregivers’ reports of spanking were related to lower social development among 215,885 3- and 4-year-old children.

A parent or caregiver was asked in person if the child gets along well with other children; if the child hits, kicks or bites others; and if the child gets distracted easily. The question about spanking concerned the physical discipline used within the last month with the child or their sibling.

One-third of the respondents indicated they believed physical punishment is necessary to bring up, raise or educate a child properly. Among the children studied, 43 percent were spanked, or resided in a home where another child was spanked.

A child’s social development suffered in both cases in which he or she was spanked or during times when a sibling had been spanked, the study showed.

“It appears that in this sample … spanking may do more harm than good,” said Garrett Pace, the study’s lead author and a doctoral student of social work and sociology.

Pace also noted that “reductions in corporal punishment might do a great deal to reduce the burden of children’s mental health and improve child development outcomes globally.”

More effort to create policies that discourage spanking has occurred globally. In fact, 54 countries have banned the use of corporal punishment, which can only benefit children’s well-being long term, Pace and colleagues said.

Abstract of the study:

Spanking is one of the most common forms of child discipline used by parents around the world. Research on children in high-income countries has shown that parental spanking is associated with adverse child outcomes, yet less is known about how spanking is related to child well-being in low- and middle-income countries. This study uses data from 215,885 children in 62 countries from the fourth and fifth rounds of UNICEF’s Multiple Indicator Cluster Surveys (MICS) to examine the relationship between spanking and child well-being. In this large international sample which includes data from nearly one-third of the world’s countries, 43% of children were spanked, or resided in a household where another child was spanked, in the past month. Results from multilevel models show that reports of spanking of children in the household were associated with lower scores on a 3-item socioemotional development index among 3- and 4-year-old children. Country-level results from the multilevel model showed 59 countries (95%) had a negative relationship between spanking and socioemotional development and 3 countries (5%) had a null relationship. Spanking was not associated with higher socioemotional development for children in any country. While the cross-sectional association between spanking and socioemotional development is small, findings suggest that spanking may be harmful for children on a more global scale than was previously known.

1 Comment

Filed under At home, Education, Research

Very interesting and relevant talk by Robert A. Bjork on learning, memory and forgetting (and how bad we are in judging effective learning)

H/T to @triciatailored

1 Comment

Filed under Education, Research, Review

Low-cost tutoring boosted struggling students’ math results (Best Evidence in Brief)

There is a new Best Evidence in Brief with among others, this study:

An evaluation in the UK of the Education Endowment Foundation trial of Tutor Trust’s affordable instruction project found that low-cost tutoring in small groups increased math scores for disadvantaged students who are working below age-expected levels in math.
One hundred and five schools in Manchester and Leeds with double the average numbers of disadvantaged students participated in the effectiveness trial of the Tutor Trust project from September 2016 until July 2017. The aim of the project is to improve the math achievement of disadvantaged students by providing small-group tutoring sessions with trained university students and recent graduates.
Year 6 students (ages 10-11) who were struggling with math were selected by their teacher to receive extra support from Tutor Trust tutors, should their school be randomly allocated to the intervention group. The selected students in the intervention schools received 12 hours of additional instruction, usually one hour per week for 12 weeks, in groups of three. Students in the control schools continued with normal teaching. Achievement was measured using Key Stage 2 math scores (standardized tests in the UK).
 The report found that children who received tutoring from Tutor Trust progressed more in math compared to children in control schools (effect size = +0.19). Among children eligible for free school meals, the effect size was +0.25. There was also some evidence that students with lower prior achievement tended to benefit more from the tutoring.

Leave a comment

Filed under Education, Research

What do you need to succeed in life?

The answer is of course sheer luck, besides talent and intelligence. This new systematically review doesn’t say intelligence and talent aren’t needed, but suggests that non-cognitive skills can also be important, although there are also some serious warning lights surrounding the existing body of evidence.

From the press release:

The study, published in the journal Nature Human Behaviour is the first to systematically review the entire literature on effects of non-cognitive skills in children aged 12 or under, on later outcomes in their lives such as academic achievement, and cognitive and language ability.

“Traits such as attention, self-regulation, and perseverance in childhood have been investigated by psychologists, economists, and epidemiologists, and some have been shown to influence later life outcomes,” says Professor John Lynch, School of Public Health, University of Adelaide and senior author of the study.

“There is a wide range of existing evidence under-pinning the role of non-cognitive skills and how they affect success in later life but it’s far from consistent,” he says.

One of the study’s co-authors, Associate Professor Lisa Smithers, School of Public Health, University of Adelaide says: “There is tentative evidence from published studies that non-cognitive skills are associated with academic achievement, psychosocial, and cognitive and language outcomes, but cognitive skills are still important.”

One of the strongest findings of their systematic review was that the quality of evidence in this field is lower than desirable. Of over 550 eligible studies, only about 40% were judged to be of sufficient quality.

“So, while interventions to build non-cognitive skills may be important, particularly for disadvantaged children, the existing evidence base underpinning this field has the potential for publication bias and needs to have larger studies that are more rigorously designed. That has important implications for researchers and funding agencies who wish to study effects of non-cognitive skills,” says Professor Lynch.

Abstract of the study:

Success in school and the labour market relies on more than high intelligence. Associations between ‘non-cognitive’ skills in childhood, such as attention, self-regulation and perseverance, and later outcomes have been widely investigated. In a systematic review of this literature, we screened 9,553 publications, reviewed 554 eligible publications and interpreted results from 222 better-quality publications. Better-quality publications comprised randomized experimental and quasi-experimental intervention studies (EQIs) and observational studies that made reasonable attempts to control confounding. For academic achievement outcomes, there were 26 EQI publications but only 14 were available for meta-analysis, with effects ranging from 0.16 to 0.37 s.d. However, within subdomains, effects were heterogeneous. The 95% prediction interval for literacy was consistent with negative, null and positive effects (−0.13 to 0.79). Similarly, heterogeneous findings were observed for psychosocial, cognitive and language, and health outcomes. Funnel plots of EQIs and observational studies showed asymmetric distributions and potential for small study bias. There is some evidence that non-cognitive skills associate with improved outcomes. However, there is potential for small study and publication bias that may overestimate true effects, and the heterogeneity of effect estimates spanned negative, null and positive effects. The quality of evidence from EQIs underpinning this field is lower than optimal and more than one-third of observational studies made little or no attempt to control confounding. Interventions designed to develop children’s non-cognitive skills could potentially improve outcomes. The interdisciplinary researchers interested in these skills should take a more strategic and rigorous approach to determine which interventions are most effective.

1 Comment

Filed under Education, Psychology, Research, Review

Going viral… in academia? Prestige rules most of the time

This study actually answers a question that I’ve had for quite a while: how come some ideas move through academia even if they’re not that good, while great insights sometimes seem to take ages to get around. This new study from Allison Morgan and her colleagues suggests something that is both close related to epidemiology and memes, but it has most to do with… prestige – and once and a while with the quality of the idea.

From the press release:

How ideas move through academia may depend on where those ideas come from–whether from big-name universities or less prestigious institutions–as much as their quality, a recent study from the University of Colorado Boulder suggests.

The new research borrows a page from epidemiology, exploring how ideas might flow from university to university, almost like a disease. The findings from CU Boulder’s Allison Morgan and her colleagues suggest that the way that universities hire new faculty members may give elite schools an edge in spreading their research to others.

In particular, the team simulated how ideas might spread out faster from highly-ranked schools than from those at the bottom of the pile–even when the ideas weren’t that good. The results suggest that academia may not function like the meritocracy that some claim, said Morgan, a graduate student in the Department of Computer Science.

She and her colleagues began by drawing on a dataset, originally published in 2015, that described the hiring histories of more than 5,000 faculty members in 205 computer science programs in the U.S. and Canada.

That dataset revealed what might be a major power imbalance in the field–with a small number of universities training the majority of tenure track faculty across both countries.

“This paper was really about investigating the implications of the imbalance,” Morgan said. “What does it mean if the elite institutions are producing the majority of the faculty who are, in turn, training the future teachers in the field?”

To answer that question, the researchers turned the 2015 dataset into a network of connected universities. If a university placed one of its Ph.D. students in a job at another school, then those two schools were linked. The resulting “roadmap” showed how faculty might carry ideas from their graduate schools to the universities that hired them.

The researchers then ran thousands of simulations on that network, allowing ideas that began at one school to percolate down to others. The team adjusted for the quality of ideas by making some more likely to shift between nodes than others.

The findings, published in October in the journal EPJ Data Science, show that it matters where an idea gets started. When mid-level ideas began at less prestigious schools, they tended to stall, not reaching the full network. The same wasn’t true for so-so thinking from major universities.

“If you start a medium- or low-quality idea at a prestigious university, it goes much farther in the network and can infect more nodes than an idea starting at a less prestigious university,” Morgan said.

That pattern held up even when the researchers introduced a bit of randomness to the mix–allowing ideas to pop from one end of the network to another by chance. That simulated how university departments might learn about an idea through factors other than hiring, such as journals, conferences or word of mouth.

The results seem to paint a dim picture of academia, said study coauthor Samuel Way, a postdoctoral research associate in computer science. He explained that recent sociological research demonstrates that workplaces benefit by having a lot of diversity–whether in gender, race or in how employees are trained.

“If you have five people who all have the exact same training and look at the world through the same lens, and you give them a problem that stumps one of them, it might stump all of them,” Way said.

He added that it may be possible for the academic world to blunt the impact of the sorts of biases the team revealed, including by adopting practices like double-blind peer review–in which the reviewers of a study can’t see the names or affiliations of the authors.

“In a setting like science where it’s incredibly difficult to come up with an objective measure of the quality of an idea, double-blind peer review may be the best you can do,” Way said.

The study did, however, contain a bit of good news: The bias toward big-name universities mattered a lot less for high-quality ideas. In other words, great thinking can still catch fire in academia, no matter where it comes from.

“I think it’s heartwarming in a way,” Morgan said. “We see that if you have a high-quality idea, and you’re from the bottom of the hierarchy, you have as good a chance of sending that idea across the network, as if it came from the top.”

Abstract of the study:

The spread of ideas in the scientific community is often viewed as a competition, in which good ideas spread further because of greater intrinsic fitness, and publication venue and citation counts correlate with importance and impact. However, relatively little is known about how structural factors influence the spread of ideas, and specifically how where an idea originates might influence how it spreads. Here, we investigate the role of faculty hiring networks, which embody the set of researcher transitions from doctoral to faculty institutions, in shaping the spread of ideas in computer science, and the importance of where in the network an idea originates. We consider comprehensive data on the hiring events of 5032 faculty at all 205 Ph.D.-granting departments of computer science in the U.S. and Canada, and on the timing and titles of 200,476 associated publications. Analyzing five popular research topics, we show empirically that faculty hiring can and does facilitate the spread of ideas in science. Having established such a mechanism, we then analyze its potential consequences using epidemic models to simulate the generic spread of research ideas and quantify the impact of where an idea originates on its longterm diffusion across the network. We find that research from prestigious institutions spreads more quickly and completely than work of similar quality originating from less prestigious institutions. Our analyses establish the theoretical trade-offs between university prestige and the quality of ideas necessary for efficient circulation. Our results establish faculty hiring as an underlying mechanism that drives the persistent epistemic advantage observed for elite institutions, and provide a theoretical lower bound for the impact of structural inequality in shaping the spread of ideas in science.

1 Comment

Filed under Education, Myths, Research, Review

Are the youngest in class more likely to be diagnosed with ADHD? (Best Evidence in Brief)

There is a new Best Evidence in Brief with among others, this study:

Findings from a study published in the Journal of Child Psychology and Psychiatry suggest that children who are the youngest in their classroom are more likely to be diagnosed with attention deficit hyperactivity disorder (ADHD) than their older classmates.
Martin Whitely and colleagues conducted a systematic review of 22 studies that examined the relationship between a child’s age relative to their classmates and their chances of being diagnosed with, or medicated for, ADHD. Seventeen studies (with a total of more than 14 million children) found that it was more common for the youngest children in a school year to be diagnosed as ADHD than their older classmates. This effect was found for both countries that have a high diagnosis rate, like the USA, Canada and Iceland, and countries where diagnosis is less common, like Finland and Sweden.
The researchers suggest that some teachers may be mistaking normal age-related immaturity of the youngest children in their class for ADHD, and that these findings highlight the importance of being aware of the impact of relative age and give the youngest children in class the extra time they may need to mature.

1 Comment

Filed under Education, Psychology, Research