Original Discipline: Research Methods / Health Sciences
The Student Struggle
Students may assume that information presented in a dataset is complete, objective, or representative of the population being studied.
They often focus on the results of an analysis without considering how the information was collected, who was included or excluded, or what limitations might affect the conclusions.
Students may also struggle to recognize how sampling decisions, missing information, or measurement methods can influence what researchers can reasonably claim.
Intended Learning Outcome
Students will be able to investigate how a publicly available dataset was collected, identify potential sources of bias or limitations, and evaluate how those factors affect the conclusions and generalizations that can be drawn from the data.
Thinking Students Practice
Primary Capacity: Judgment
Students evaluate the quality and limitations of data collection methods and determine how much confidence to place in conclusions drawn from a dataset.
Secondary Capacity: Intellectual Humility
Students recognize that data may not provide a complete picture and acknowledge uncertainty when information about collection methods, sampling, or representation is limited.
Secondary Capacity: Sensemaking
Students connect decisions about data collection to the interpretation of research findings, recognizing how the methods used to produce information influence what it can tell us.
The Learning Activity
Students investigate a publicly available dataset, preferably related to health, nutrition, medicine, public health, or another area of interest.
Rather than immediately analyzing the dataset's findings, students first investigate how the information was collected.
They identify the source of the data, the population or sample represented, the collection methods, and any available documentation describing the process.
Students then evaluate possible limitations, such as sampling bias, missing information, nonresponse, or limited representation.
The activity concludes with an online discussion in which students explain how these limitations might influence the conclusions researchers can draw.
Students also respond to classmates, considering how different limitations could affect the interpretation and generalizability of the data.
Try This With Your Students
Before Trusting the Data: Investigating How Information Is Collected
The Scenario
You have been asked to evaluate a publicly available dataset that researchers or policymakers might use to understand an important issue.
Before anyone uses this information to make decisions, you need to investigate where the data came from and whether the way it was collected might limit what it can tell us.
Your goal is not simply to identify what the dataset contains. It is to determine how confidently its findings could be interpreted or applied.
Your Challenge
Step 1: Select a Dataset
Choose a publicly available dataset that interests you.
Whenever possible, select one related to health, nutrition, medicine, public health, or another health-science topic.
Possible sources include:
Centers for Disease Control and Prevention (CDC)
World Health Organization (WHO)
U.S. Census Bureau
Pew Research Center
State or local health departments
Identify the dataset's name, the organization responsible for it, and a link to the dataset or its documentation.
Briefly describe what information it contains.
Step 2: Investigate How the Data Were Collected
Using the available documentation, determine:
How were the data collected?
Where and when were they collected?
Who was included in the population or sample?
How were participants or observations selected?
What sampling method was used, if one is identified?
Record your findings in three to five bullet points.
If the documentation does not provide an answer, identify what information is missing rather than making assumptions.
Step 3: Identify Potential Limitations
Examine the collection process and identify at least one potential limitation or source of bias.
Possibilities include:
Undercoverage
Nonresponse bias
Voluntary response bias
Selection or sampling bias
Measurement or response bias
Missing or incomplete data
Limited geographic or demographic representation
Explain why the limitation matters and how it might affect the dataset.
Step 4: Evaluate What the Data Can Tell Us
Consider whether the data could reasonably be used to draw conclusions about a broader population.
What populations or situations are represented?
Which groups might be missing or underrepresented?
What conclusions would require greater caution?
Step 5: Discuss Your Findings
Post your analysis to the class discussion board.
Read at least two classmates' posts and respond to their evaluations.
Consider:
How could the limitation they identified affect conclusions drawn from the data?
How might researchers improve the data collection process?
To what population could the findings reasonably be generalized?
Would applying those findings to a different population be justified? Why or why not?
Share Your Thinking
Your initial discussion post should include:
The dataset and its source.
A brief explanation of how the data were collected.
At least one potential limitation or source of bias.
An explanation of how that limitation might affect the conclusions someone could draw.
Include a citation or link to the dataset and the documentation used to investigate its collection.
Respond to at least two classmates with questions, observations, or suggestions that extend their analysis.
Remember: Identifying a limitation does not necessarily make a dataset unusable. The important question is whether you understand what the data can reasonably tell you and where greater caution is needed.
Making Student Thinking Visible
Students produce a written evaluation of a dataset's collection methods, potential limitations, and implications for interpretation.
The instructor examines whether students can move beyond simply naming a type of bias to explaining how that bias might influence the information collected.
Students demonstrate judgment by identifying which conclusions are supported by the available data and which would require additional evidence.
Their discussion responses provide further evidence of whether they can consider alternative interpretations, evaluate the reasoning of classmates, and recognize the limitations of their own conclusions.
Adapt This Scenario
The underlying instructional approach asks students to investigate how information was produced before deciding what conclusions it supports.
Business: Students evaluate customer survey data and consider how sampling methods might influence marketing or product-development decisions.
Psychology: Students examine participant recruitment and data collection in a published study and identify limitations affecting generalizability.
Education: Students investigate institutional survey results and consider whether the data adequately represent different groups of students.
Environmental Science: Students examine environmental monitoring data and evaluate how collection locations, frequency, or measurement methods may influence reported trends.
Information Technology: Students evaluate data collected from system performance monitoring or user feedback and consider how missing or unrepresentative information might affect technical decisions.
Across disciplines, students practice asking not just what the data show, but how the data came to exist and whether their limitations affect the conclusions being drawn.
Original AI Prompt and Faculty Revisions
The original workshop contribution was a discussion activity titled Data Collection Methods & Potential Flaws.
It asked students to locate a publicly available dataset, investigate the data collection process, and identify potential limitations or sources of bias.
The activity emphasized health-related datasets but allowed students to select other topics of interest.
Students were instructed to document the dataset's source, describe its collection methods, identify limitations, and explain how those limitations could influence interpretation.
The original contribution also required students to respond to at least two classmates, examining the implications of identified biases and considering possible improvements to data collection methods.
This library version preserves the original discussion structure and central learning objectives while presenting the instructions within our standard scenario format.
The intended learning outcome, intellectual capacity categories, and cross-disciplinary adaptations are suggested additions developed for this library.
The original document does not include the AI prompt used to develop this activity or document subsequent faculty revisions.