DDDM 5221 Resources

Welcome to the week on technology supports for data-driven (DDDM) and evidence-based (EBDM) decision making in schools. Please read the following to prepare for lab.

Making Decisions about how best to teach is never an easy process.  For this week’s lab, you should begin by considering what it might be like to be in an annual teacher evaluation meeting with your principal. You can get a sense of this by viewing This Teacher Evaluation video that includes a typical review of student progress that is, in part, as a metric of successful teaching. [WARNING: the audio on this video is not of high quality, as is often the case with real classroom video. So listen closely to what the principal (seated on the left behind the desk, and the teacher seated on the right) are discussing).] As you view this teacher evaluation meeting, consider the following:

  • How did the 5th grade teacher generate these graphs? Do you see any other such charts around the principal’s desk?
  • (around the 0:53 mark) In the classrooms you visited, how could you obtain data from the same students for 4 years (in the video it is 2nd grade to 5th grade)? Don’t students come in and out of the classroom? How did she get these data?
  • (around the 1:00 mark) What does the grading language of “moving from ‘basic,’ to ‘proficient,’ and ‘advanced'” mean or indicate to you?
  • (around the 1:30 mark) the 5th grade teacher makes the point that students should set goals to “maintain” rather than be ‘advanced’ – how does that relate to rubric grades in EPSY 5221?
  • How do these educators use proximal level assessments to make decisions about individual students?
  • How do these same results, that guide instruction, also speak to teacher skills? Are they valid for both purposes?
  • (around the 2:13 mark) A specific student’s name is mentioned, and the teacher and principal exchange comments- drawing from your Ethics module content, what concerns does this event (in a YouTube public video) raise for you?

First, you should understand that there are many types of data, quantitative and qualitative, and that there may be additional forms of evidence that can be used to understand the impact of instruction and teacher success. To make this point, consider this diagram that shows that teachers can do many small “tests” and observations at the “Close” level, even within a single class period, but such data must be quick and easy to obtain, analyze, and use.

But they also can obtain data from unit end tests at the “Proximal” level, that test knowledge, attitudes, and skills related to the standards-based curriculum. And of course, there are always those high-stakes tests like the Smarter-balanced tests, the CMT and the SATs, at the “Distal” level, that we all love, that are norm referenced on national samples and have purposes well beyond a single classroom or teacher’s work.

All of this testing is often used to compute alphanumeric report card grades (e.g., 95% A). Certainly we have all experienced this system of grading and its limitations. In comparison, Standards-based or mastery-based grading, provide evidence of student success while removing confusing factors such as effort/attitude, extra credit, and student behavior (such as submitting things precisely on time). These other factors are graded separately from content-related mastery based on State and National standards.

PARCC and Smarter Balanced.

Testing in K-12 schools has changed, mainly related to technology access. Traditional standardized tests (like the SAT MCAT GRE LSAT) have been upgraded with artificial intelligence in the form of Computer Adaptive Test.  The Smarter Balanced test is computer adaptive and so it changes with each answer students give, and then selects the next question presented based on the test takers history. Underlying the approach of computer adaptive testing are Item Response Theory, which uses research to optimize the diagnostic value of each wrong answer (distractors). Here are some slides with the core ideas of CAT and a 16 minute YouTube tutorial about key concepts. More issues with using CATs are address in “This test is for you.” Note ETS (for the GRE) and the College Board (for the SAT) are also considering test revisions that might incorporate computer adaptive programming and perhaps alignment with content objectives.

Purposes of Assessments.

Assessment literate teachers understand the difference between assessment for, of, or as learning. Unfortunately, the general public and novice teachers seem to focus mainly on assessments of learning, because people most often recall high stakes testing or major unit tests and projects (summative assessments) that were meant to sample understanding concept or skill acquisition along a broad range of topics (Deluca & Bellara, 2013; Stiggins, 1991). Technology has become a key part of high stakes testing, with most standardized tests (SBAC, PSAT, SAT,  ACT) are administered using computers and adaptive or tailored testing methods. Teachers can integrate tech more easily into formative assessments. Assessments for learning, also called formative assessment, are ongoing and serve as “dipsticks” of how students are doing throughout a unit. Tech tools can help with pretests designed to help students construct their own learning goals. Assessments as learning, which also have a formative purpose, involve students in measuring and being aware of their own progress in a metacognitive and reflective way (Brookhart, 2011; Stiggins, 2018).

Making the Grade.. On your mark… get set…

OK.. so reporting percent correct on a single test is different from calculating percentages across topics or units. If, as a future physician, I get an an A in my knowledge of the heart but a C on my knowledge of the brain, does that mean I am a B doctor, or would your still prefer me as your heart surgeon rather than your brain surgeon? Are grades supposed to reflect how much students have learned (criterion-based) or are they meant to show the relative performance of students to rank the top students (normative)? Grades cannot do both these things. Can you meaningfully give yourself or anyone, 1 single grade that meaningfully represents their knowledge of world history, or is it inevitable that I may know a bit more about my home country than I know about all other countries? Any what about effort and extra credit– should a student’s final grade reflect their level of effort as well as the level of their final achievement of understanding?   Tom Gruskey asks these questions in his slides and handouts to get future teachers to think more deeply about how and why they grade and what those score mean when used for various purposes.

The role of Technology in DDDM

Take a peek at Infinite Campus, a school-based data system being widely adopted, including in my favorite schools in Ellington CT. Teacher, parents, and even students will have mobile access to a variety of school performance data, to help with decision making. Consider what it will be like to be a teacher in schools implementing these parent, teacher and student dashboard to data. Moving beyond data-driven and evidence-based toward “purpose driven” decision making with a variety of types of evidence, the technology is driving the availability of data and teachers will need to be fluent in data reduction and data summarization to access and help others makes sense of all types of data.

Resources

Technologies

Supplemental: References and Read More about…

t-tests vs ANCOVA analysis of gain scores

Analyzing gain scores vs using the pretest as covariate– the saga continues

Brookhart, S. M. (2011). Educational assessment knowledge and skills for teachers. Educational Measurement: Issues and Practice, 30(1), 3-12. http://onlinelibrary.wiley.com/doi/10.1111/j.1745-3992.2010.00195.x/abstract
DeLuca, C., & Bellara, A. (2013). The current state of assessment education: Aligning policy, standards, and teacher education curriculum. Journal of Teacher Education, 64(4), 356-372.

Stiggins, R. (2018). Better Assessments Require Better Assessment Literacy. Educational Leadership, 75(5), 18-19.

Enduring Understanding- Student Learning Outcome from this module

This module intends to prepare teachers for ISTE Standard for Educators 7c that states: Use assessment data to guide progress and communicate with students, parents and education stakeholders to build student self-direction.

Stretch Goals:
  • Understand the Learning Sciences approach to co-research in schools
  • Discuss the relative value of mastery vs alphanumeric grades
  • Interrogate how to prepare students for Computer-Adaptive Tests