Original Discipline: Psychology
The Student Struggle
Students often struggle to critically evaluate published research. They may assume that findings reported in an academic journal are necessarily reliable or that statistically significant results automatically indicate important or meaningful conclusions.
Students may also have difficulty distinguishing between a researcher's interpretation of results and what the evidence actually supports.
When reading scientific articles, they may focus on the conclusions without considering the research design, sample, limitations, or whether other studies have reached similar findings.
Intended Learning Outcome
Students will be able to critically evaluate an empirical psychology article, identify strengths and limitations in its methodology, assess whether its conclusions are supported by evidence, and develop a balanced critique that considers relevant research and possible alternative explanations.
Thinking Students Practice
Primary Capacity: Judgment
Students evaluate the quality of research evidence, consider methodological strengths and weaknesses, and determine how much confidence to place in a study's conclusions.
Secondary Capacity: Intellectual Humility
Students recognize that scientific findings are subject to uncertainty, revision, and further investigation. They learn to acknowledge limitations without automatically dismissing research that contains them.
Secondary Capacity: Sensemaking
Students connect research questions, methodology, findings, and conclusions to develop an understanding of how scientific knowledge is constructed and evaluated.
The Learning Activity
Students examine an empirical psychology article as though they were peer reviewers.
Rather than simply summarizing the article, they investigate the relationship between its research questions, methods, results, and conclusions.
The original faculty-created unit uses research on laptop versus handwritten note-taking as its central example.
Students first examine a published study suggesting that handwritten note-taking may offer advantages for conceptual learning. They identify the research question, variables, methodology, sample, and findings.
Next, students investigate the study's limitations and consider whether the evidence supports the researchers' conclusions.
They then examine a subsequent replication study that reached different findings.
The activity culminates in a mock peer-review exercise in which students identify strengths and weaknesses, evaluate the significance of the evidence, and recommend whether the original research should be accepted, revised, or rejected.
The goal is not to teach students to distrust research. It is to help them develop the judgment necessary to evaluate scientific claims fairly and thoughtfully.
Try This With Your Students
Does Taking Notes by Hand Really Improve Learning?
The Scenario
You are part of a student research review committee.
Your committee has been asked to evaluate a published psychology study examining whether students learn more effectively when taking notes by hand rather than on a laptop.
The original study reported advantages associated with handwritten note-taking on some measures of conceptual learning.
However, subsequent researchers attempted to replicate the findings and reported different results.
Your committee must evaluate the available evidence and decide how much confidence readers should place in the original conclusions.
Your Challenge
Step 1: Understand the Research
Read the assigned research article.
Identify:
The main research question.
The researchers' hypothesis.
The independent and dependent variables.
The characteristics of the study's participants.
How the researchers collected and analyzed their data.
The main findings and conclusions.
Step 2: Evaluate the Evidence
Examine the relationship between the research methods and the conclusions.
Consider:
Were the participants representative of the broader population?
What did the researchers actually measure?
Were there limitations in how the study was designed?
Do the results support the conclusions?
What additional evidence would strengthen the findings?
Step 3: Examine the Replication
Read a summary of the subsequent replication study.
Compare its findings with those of the original research.
Discuss:
What similarities and differences exist between the studies?
Why might researchers obtain different results?
Does a failure to replicate a finding necessarily mean the original study was wrong?
How should the additional evidence influence your confidence in the original conclusion?
Step 4: Become a Peer Reviewer
Working in small groups, prepare a short peer-review report.
Your report should identify:
One important strength of the original study.
Two significant methodological concerns.
One question requiring additional investigation.
Your overall assessment of the strength of the evidence.
Step 5: Make a Recommendation
Decide whether you would recommend accepting, revising, or rejecting the original article if it had just been submitted for publication.
Explain your recommendation using specific evidence rather than personal opinion.
Share Your Thinking
Present your group's recommendation to the class.
Be prepared to explain which evidence most influenced your judgment, what remains uncertain, and what additional research might change your conclusion.
Remember: Good scientific evaluation requires both skepticism and openness. The goal is to assess research fairly, recognizing its strengths while carefully examining its limitations.
Making Student Thinking Visible
Students produce an article analysis, a methodological critique, and a justified peer-review recommendation.
The instructor evaluates whether students can identify relevant research evidence, connect methodological decisions to possible limitations, and explain how those limitations affect the strength of a conclusion.
Students should demonstrate an ability to distinguish between identifying a weakness and explaining why that weakness matters.
The comparison between the original and replication studies provides additional evidence of how students respond when new information complicates an earlier conclusion.
The instructor can also examine whether students acknowledge uncertainty and avoid overstating what the research establishes.
Adapt This Scenario
The underlying instructional approach asks students to evaluate a published claim, examine the evidence supporting it, consider limitations, and develop a justified conclusion.
Healthcare: Students evaluate research supporting a medical intervention and consider whether the findings justify changes in professional practice.
Business: Students critique a market research study and determine whether its findings support a proposed organizational decision.
Environmental Science: Students examine competing studies about an environmental issue and evaluate differences in methods, evidence, and conclusions.
Education: Students review research on a teaching strategy and consider whether its findings support implementation in a particular learning environment.
Information Technology: Students evaluate claims about a technology's effectiveness, examine performance evidence, and identify limitations in testing methods.
Across disciplines, students practice moving beyond accepting published conclusions toward evaluating the quality and relevance of the evidence supporting them.
Original AI Prompt and Faculty Revisions
The original faculty prompt asked an AI system to develop a one-week instructional unit for first-year psychology majors focused on reviewing and critiquing scientific articles.
The requested unit included suggested readings, group activities, discussions, and a final assessment.
The AI-generated response, titled Reading Like a Reviewer, assumed three 50-minute class sessions with students working in small groups.
The first session introduced the structure of empirical research articles and asked students to identify research questions, hypotheses, methods, and findings.
The second session focused on evaluating research methodology and statistical evidence, including internal, external, construct, and statistical validity.
The third session introduced replication research and asked students to prepare mock peer-review reports.
The original unit used the following research as its central case:
Mueller and Oppenheimer (2014), The Pen Is Mightier Than the Keyboard, examining laptop and handwritten note-taking.
Morehead, Dunlosky, and Rawson (2019), a subsequent replication examining the original findings.
Open Science Collaboration (2015), Estimating the Reproducibility of Psychological Science, providing broader context about research replication.
The full AI-generated unit also included suggested readings, timed lesson plans, discussion prompts, team roles, assessment instructions, a grading rubric, and optional activities.
Students ultimately completed an individual two- to three-page research critique.
This library version presents the central instructional approach as a shorter, adaptable classroom scenario. It does not reproduce the complete week-long lesson plan or rubric.
The original contribution does not document subsequent faculty revisions or classroom implementation.