The Ideal versus the Real Deal in Assessment of Physics Lab Report Writing
DOI:
https://doi.org/10.14738/aivp.112.14406Keywords:
Science writing assessment, Physics lab reports, Analytic rubrics, Writing assessment reliability.Abstract
Effective writing is important for communicating science ideas, and for writing-to-learn in science. This paper investigates lab reports from a large-enrollment college physics course that integrates scientific reasoning and science writing. While analytic rubrics have been shown to define expectations more clearly for students, and to improve reliability of assessment, there has been little investigation of how well analytic rubrics serve students and instructors in large-enrollment science classes. Unsurprisingly, we found that grades administered by teaching assistants (TAs) do not correlate with reliable post-hoc assessments from trained raters. More important, we identified lost learning opportunities for students, and misinformation for instructors about students’ progress. We believe our methodology to achieve post-hoc reliability is straightforward enough to be used in classrooms. A key element is the development of finer-grained rubrics for grading that are aligned with the rubrics provided to students to define expectations, but which reduce subjectivity of judgements and grading time. We conclude that the use of dual rubrics, one to elicit independent reasoning from students and one to clarify grading criteria, could improve reliability and accountability of lab report assessment, which could in turn elevate the role of lab reports in the instruction of scientific inquiry.
References
D. Kuhn, “What is Scientific Thinking and How Does It Develop?” in Blackwell Handbook of Childhood Cognitive Development, U. Goswami, Ed., Malden, MA, USA: Blackwell Publishers Ltd, 2002, pp. 371–393. doi: 10.1002/9780470996652.ch17.
C. W. Keys, “Revitalizing instruction in scientific genres: Connecting knowledge production with writing to learn in science,” Science Education, vol. 83, no. 2, pp. 115–130, 1999, doi: 10.1002/(SICI)1098-237X(199903)83:2<115:AID-SCE2>3.0.CO;2-Q.
S. Graham, S. A. Kiuhara, and M. MacKay, “The Effects of Writing on Learning in Science, Social Studies, and Mathematics: A Meta-Analysis,” Review of Educational Research, vol. 90, no. 2, pp. 179–226, Apr. 2020, doi: 10.3102/0034654320914744.
A. R. Gere, N. Limlamai, E. Wilson, K. MacDougall Saylor, and R. Pugh, “Writing and Conceptual Learning in Science: An Analysis of Assignments,” Written Communication, vol. 36, no. 1, pp. 99–135, Jan. 2019, doi: 10.1177/0741088318804820.
M. Huerta and T. Garza, “Writing in Science: Why, How, and for Whom? A Systematic Literature Review of 20 Years of Intervention Research (1996–2016),” Educational Psychology Review, vol. 31, no. 3, pp. 533–570, Sep. 2019, doi: 10.1007/s10648-019-09477-1.
E. B. D. Clabough and S. W. Clabough, “Using Rubrics as a Scientific Writing Instructional Method in Early Stage Undergraduate Neuroscience Study,” J Undergrad Neurosci Educ, vol. 15, no. 1, pp. A85–A93, 2016.
P. D. Klein and P. Boscolo, “Trends in research on writing as a learning activity,” Journal of Writing Research, vol. 7, no. 3, pp. 311–350, 2016, doi: Journal of Writing Research.
B. E. C. Timmerman, D. C. Strickland, R. L. Johnson, and J. R. Payne, “Development of a ‘universal’ rubric for assessing undergraduates’ scientific reasoning skills using scientific writing,” Assessment & Evaluation in Higher Education, vol. 36, no. 5, pp. 509–547, Aug. 2011, doi: 10.1080/02602930903540991.
B. O’Donovan, C. Rust, and M. Price, “A scholarly approach to solving the feedback dilemma in practice,” Assessment & Evaluation in Higher Education, vol. 41, no. 6, pp. 938–949, Aug. 2016, doi: 10.1080/02602938.2015.1052774.
C. Evans, “Making Sense of Assessment Feedback in Higher Education,” Review of Educational Research,
J. B. Garfield, D. Ben-Zvi, B. Chance, E. Medina, C. Roseth, and A. Zieffler, “Assessment in Statistics Education,” in Developing Students’ Statistical Reasoning, Dordrecht: Springer Netherlands, 2008, pp. 65–89. doi: 10.1007/978-1-4020-8383-9_4.
A. Pisano, A. Crawford, H. Huffman, B. Graham, and N. Kelp, “Development and Validation of a Universal Science Writing Rubric That is Applicable to Diverse Genres of Science Writing,” J Microbiol Biol Educ., vol. 22, no. 3, pp. e00189-21, Dec. 2021, doi: 10.1128/jmbe.00189-21.
V. Sampson, P. Enderle, J. Grooms, and S. Witte, “Writing to Learn by Learning to Write During the School Science Laboratory: Helping Middle and High School Students Develop Argumentative Writing Skills as They Learn Core Ideas,” Science Education, vol. 97, no. 5, pp. 643–670, 2013, doi: 10.1002/sce.21069.
J. Trace, V. Meier, and G. Janssen, “‘I can see that’: Developing shared rubric category interpretations through score negotiation,” Assessing Writing, vol. 30, pp. 32–43, Oct. 2016, doi: 10.1016/j.asw.2016.08.001.
T. H. Sundeen, “Instructional rubrics: Effects of presentation options on writing quality,” Assessing Writing, vol. 21, pp. 74–88, Jul. 2014, doi: 10.1016/j.asw.2014.03.003.
R. L. Bangert-Drowns, M. M. Hurley, and B. Wilkinson, “The Effects of School-Based Writing-to-Learn Interventions on Academic Achievement: A Meta-Analysis,” Review of Educational Research, vol. 74, no. 1, pp. 29–58, Mar. 2004, doi: 10.3102/00346543074001029.
B. Hand, Y.-C. Chen, and J. K. Suh, “Does a Knowledge Generation Approach to Learning Benefit Students? A Systematic Review of Research on the Science Writing Heuristic Approach,” Educational Psychology Review, Aug. 2020, doi: 10.1007/s10648-020-09550-0.
B. Hand, M. C. Shelley, M. Laugerman, L. Fostvedt, and W. Therrien, “Improving critical thinking growth for disadvantaged groups within elementary school science: A randomized controlled trial using the Science Writing Heuristic approach,” Sci. Ed., vol. 102, no. 4, pp. 693–710, Jul. 2018, doi: 10.1002/sce.21341.
N. S. Stephenson and N. P. Sadler-McKnight, “Developing critical thinking skills using the Science Writing Heuristic in the chemistry laboratory,” Chem. Educ. Res. Pract., vol. 17, no. 1, pp. 72–79, 2016, doi: 10.1039/C5RP00102A.
M. M. Balgopal, A. M. A. Casper, A. M. Wallace, P. J. Laybourn, and E. Brisch, “Writing Matters: Writing-to-Learn Activities Increase Undergraduate Performance in Cell Biology,” BioScience, vol. 68, no. 6, pp. 445–454, Jun. 2018, doi: 10.1093/biosci/biy042.
J. Airey and C. Linder, “A disciplinary discourse perspective on university science learning: Achieving fluency in a critical constellation of modes,” J. Res. Sci. Teach., vol. 46, no. 1, pp. 27–49, Jan. 2009, doi: 10.1002/tea.20265.
P. Black and D. William, “Assessment and Classroom Learning,” Assessment in Education: Principles, Policy & Practice, vol. 5, no. 1, pp. 7–74, Mar. 1998, doi: 10.1080/0969595980050102.
C. S. Kalman, Successful Science and Engineering Teaching: Theoretical and Learning Perspectives. in Innovation and Change in Professional Education. Springer, Cham, 2018. [Online]. Available: https://doi.org/10.1007/978-3-319-66140-7
United States Chamber of Commerce, “Bridging the soft skills gap: How the business and education sectors are partnering to prepare students for the 21 st century workforce,” Center for Education and Workforce, Washington, DC, 2017. [Online]. Available: https://www.uschamberfoundation.org/sites/default/files/Closing%20the%20Soft%20Skills%20Gap.pdf
National Science & Technology Council, “Charting a course for success: America’s strategy for STEM education,” Office of Science and Technology Policy, Washington, DC, 2018. [Online]. Available: https://www.energy.gov/sites/default/files/2019/05/f62/STEM-Education-Strategic-Plan-2018.pdf
C. Zimmerman, “The development of scientific reasoning: What psychologists contribute to an understanding of elementary science learning,” National Research Council Committee on Science Learning Kindergarten through Eighth Grade, 2005. [Online]. Available: https://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_080105.pdf
L. Schauble, R. Glaser, R. A. Duschl, S. Schulze, and J. John, “Students’ Understanding of the Objectives and Procedures of Experimentation in the Science Classroom,” Journal of the Learning Sciences, vol. 4, no. 2, pp. 131–166, Apr. 1995, doi: 10.1207/s15327809jls0402_1.
K. Hogan and M. Maglienti, “Comparing the epistemological underpinnings of students’ and scientists’ reasoning about conclusions,” J. Res. Sci. Teach., vol. 38, no. 6, pp. 663–687, Aug. 2001, doi: 10.1002/tea.1025.
D. Kuhn, E. Amsel, M. O’Loughlin, L. Schauble, B. Leadbeater, and W. Yotive, The development of scientific thinking skills. Academic Press, 1988.
F. Reif and J. H. Larkin, “Cognition in scientific and everyday domains: Comparison and learning implications,” J. Res. Sci. Teach., vol. 28, no. 9, pp. 733–760, Nov. 1991, doi: 10.1002/tea.3660280904.
K. E. Stanovich and R. F. West, “Reasoning independently of prior belief and individual differences in actively open-minded thinking.,” Journal of Educational Psychology, vol. 89, no. 2, pp. 342–357, Jun. 1997, doi: 10.1037/0022-0663.89.2.342.
P. A. Klaczynski, D. H. Gordon, and J. Fauth, “Goal-oriented critical reasoning and individual differences in critical reasoning biases.,” Journal of Educational Psychology, vol. 89, no. 3, pp. 470–485, Sep. 1997, doi: 10.1037/0022-0663.89.3.470.
J. D. Bransford, A. L. Brown, and R. R. Cocking, How People Learn: Brain, Mind, Experience, and School: Expanded Edition. Washington, D.C.: National Academies Press, 2000, p. 9853. doi: 10.17226/9853.
R. Bybee, B. McCrae, and R. Laurie, “PISA 2006: An assessment of scientific literacy,” J. Res. Sci. Teach., vol. 46, no. 8, pp. 865–883, Oct. 2009, doi: 10.1002/tea.20333.
L. C. McDermott, P. S. Shaffer, and M. L. Rosenquist, Physics by inquiry: an introduction to physics and the physical sciences, vol. 1 and 2. New York: J. Wiley, 1996.
D. R. Sokoloff, R. K. Thornton, and P. W. Laws, RealTime physics: active learning laboratories, Modules 1-4. New York: Wiley, 2004.
L. Bao, K. Koenig, Y. Xiao, J. Fritchman, S. Zhou, and C. Chen, “Theoretical model and quantitative assessment of scientific thinking and reasoning,” Phys. Rev. Phys. Educ. Res., vol. 18, no. 1, p. 010115, Feb. 2022, doi: 10.1103/PhysRevPhysEducRes.18.010115.
A. Zeineddin and F. Abd-El-Khalick, “Scientific reasoning and epistemological commitments: Coordination of theory and evidence among college science students,” J. Res. Sci. Teach., vol. 47, no. 9, pp. 1064–1093, Nov. 2010, doi: 10.1002/tea.20368.
W. C. Sá, C. N. Kelley, C. Ho, and K. E. Stanovich, “Thinking about personal theories: individual differences in the coordination of theory and evidence,” Personality and Individual Differences, vol. 38, no. 5, pp. 1149–1161, Apr. 2005, doi: 10.1016/j.paid.2004.07.012.
S. L. Eddy and K. A. Hogan, “Getting Under the Hood: How and for Whom Does Increasing Course Structure Work?,” LSE, vol. 13, no. 3, pp. 453–468, Sep. 2014, doi: 10.1187/cbe.14-03-0050.
L. Tsui, “Effective Strategies to Increase Diversity in STEM Fields: A Review of the Research Literature,” Journal of Negro Education, vol. 76, no. 4, pp. 555–581, 2007.
D. Baldwin, M. Fowles, and S. Livingston, “Guidelines for constructed-responses and other performance assessmen,” Educational Testing Service, Princeton, NJ, TOEFL BT 02 RR-07-02, 2008.
E. D. Turley and C. W. Gallagher, “On the uses of rubrics: Reframing the great rubric debate,” The English Journal, vol. 97, no. 4, pp. 87–92, 2008.
J. Cohen, “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, Apr. 1960, doi: 10.1177/001316446002000104.
M. J. Warrens, “Conditional inequalities between Cohen’s kappa and weighted kappas,” Statistical Methodology, vol. 10, no. 1, pp. 14–22, Jan. 2013, doi: 10.1016/j.stamet.2012.05.004.
K. Krippendorff, Content analysis: an introduction to its methodology, Fourth Edition. Fourth Edition.Los Angeles: SAGE, 2018.
A. R. Feinstein and D. V. Cicchetti, “High agreement but low Kappa: I. the problems of two paradoxes,” Journal of Clinical Epidemiology, vol. 43, no. 6, pp. 543–549, Jan. 1990, doi: 10.1016/0895-4356(90)90158-L.
D. V. Cicchetti and A. R. Feinstein, “High agreement but low kappa: II. Resolving the paradoxes,” Journal of Clinical Epidemiology, vol. 43, no. 6, pp. 551–558, Jan. 1990, doi: 10.1016/0895-4356(90)90159-M.
B. North, “Scales for rating language performance: Descriptive models, formulation styles, and presentation forma,” Educational Testing Service, Princeton, NJ, TOEFL Monograph MS-24, 2003.
G. A. Miller, “The magical number seven, plus or minus two: Some limits on our capacity for processing information.,” Psychological Review, vol. 63, no. 2, pp. 81–97, Mar. 1956, doi: 10.1037/h0043158.
A. Jonsson and G. Svingby, “The use of scoring rubrics: Reliability, validity and educational consequences,” Educational Research Review, vol. 2, no. 2, pp. 130–144, Jan. 2007, doi: 10.1016/j.edurev.2007.05.002.
G. Janssen, V. Meier, and J. Trace, “Building a better rubric: Mixed methods rubric revision,” Assessing Writing, vol. 26, pp. 51–66, Oct. 2015, doi: 10.1016/j.asw.2015.07.002.
J. R. Landis and G. G. Koch, “The Measurement of Observer Agreement for Categorical Data,” Biometrics,
M. Gunel, B. Hand, and V. Prain, “Writing for Learning in Science: A Secondary Analysis of Six Studies,” Int J of Sci and Math Educ, vol. 5, no. 4, pp. 615–637, Oct. 2007, doi: 10.1007/s10763-007-9082-y.
A. Moon, A. R. Gere, and G. V. Shultz, “Writing in the STEM classroom: Faculty conceptions of writing and its role in the undergraduate classroom,” Science Education, vol. 102, no. 5, pp. 1007–1028, 2018, doi: 10.1002/sce.21454.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Rebecca J. Passonneau, Kathleen Koenig, Zhaohui Li, Josephine Soddano
This work is licensed under a Creative Commons Attribution 4.0 International License.
