DDDM 5220 Resources

Welcome to the module on technology supports for data-driven and evidence-based decision making in schools.

Making Decisions about how best to teach is never an easy process.  Yet teachers are asked increasingly to present data as part of annual teacher evaluations and parent conferences. Further, personalized learning is best created on the basis of evidence of learning. Technology-based learning with its built-in analytics and ability to mine trends from big data is also a new factor to consider as we experience the good and bad of data mining from Amazon suggestions and political data breeches (see for example Cambridge Analytica article).

First, you should understand that there are many types of data, quantitative and qualitative, and that there may be additional forms of evidence that can be used to understand the impact of instruction and teacher success. Data driven decision making (DDDM) can be just about the data or it can be storytelling about individual students. Teachers may in fact prefer the latter.

To further this point, consider this diagram that shows that teachers can do many small “tests” and observations at the “Close” level, even within a single class period, but such data must be quick and easy to obtain, analyze, and use.

But they also can obtain data from unit end tests at the “Proximal” level, that test knowledge, attitudes, and skills related to the standards-based curriculum. And of course, there are always those high-stakes tests like the Smarter-balanced tests, the CMT and the Common Core aligned  School Day SATs, at the “Distal” level, that we all love, that are norm referenced on national samples and have purposes well beyond a single classroom or teacher’s work.

Distal Level. There is no doubt that high stakes testing is driving a large portion of classroom activity. In Connecticut, the high stakes testing data (EdSight portal) is used to evaluate teachers, principals, schools, districts, towns, and the State. Residents use such data to decide where to live and build a house, and directly and indirectly drive economic development. Smarter Balanced data will be used once the initial trials settle.

What are the ways to consider interpreting high stakes test scores, instead of thinking of the SAT or Smarter Balanced as pass-fail, but rather on a growth continuum. And how can additional data, such as chronic absences, help improve the validity of the decision making process? In Connecticut, there are efforts to more wisely use the 12 indicators of school success.

But why might many parents “opt out” of high stakes tests? Other states are dealing with parents who opt out of high stakes tests. In CT, school districts can lose funding if more than 10% of their students opt out of high stakes test. Looking solely at numbers may be misleading, but ignoring quantitative assessments may be limiting too. Perhaps we should seek a balance.

Are high stakes test data a legitimate measure of teacher effectiveness? Student performance on high stakes tests has only an indirect connection to what teachers do in their classroom that works or needs improvement (see this Bergin (2015) report prepared for NEE). There are several issues constraining the value of student test scores as a measure of teacher effectiveness.

All of this testing is often used to compute alphanumeric report card grades (e.g., 95% = A). Certainly we have all experienced this system of grading and its limitations. In comparison, Standards-based or mastery-based grading, provide evidence of student success while removing confusing factors such as effort/attitude, extra credit, and student behavior (such as submitting things precisely on time). These other factors are graded separately from content-related mastery based on State and National standards.

Purposes of Assessments.

Assessment literate teachers understand the difference between assessment for, of, or as learning. Unfortunately, the general public and novice teachers seem to focus mainly on assessments of learning, because people most often recall high stakes testing or major unit tests and projects (summative assessments) that were meant to sample understanding concept or skill acquisition along a broad range of topics (Deluca & Bellara, 2013; Stiggins, 1991). Technology has become a key part of high stakes testing, with most standardized tests (SBAC, PSAT, SAT,  ACT) are administered using computers and adaptive or tailored testing methods. Teachers can integrate tech more easily into formative assessments. Assessments for learning, also called formative assessment, are ongoing and serve as “dipsticks” of how students are doing throughout a unit. Tech tools can help with pretests designed to help students construct their own learning goals. Assessments as learning, which also have a formative purpose, involve students in measuring and being aware of their own progress in a metacognitive and reflective way (Brookhart, 2011; Stiggins, 2018)

Teacher-made tests: Proximal Level. Teachers can make up their own tests and often do so for each Unit or Chapter that they present. These are the grades that go into the teacher’s gradebook and result in grades that go home on Report Cards. Note: many schools are moving away from graded report cards toward competency-based grading. While teacher made tests are much easier to create, they often are not very reliable, so the decisions they guide may not always be the right ones. Of course many teachers are not assessment design experts so they often make common errors when creating these assessment. There are many things to consider when designing a test.

What might be the value of comprehensive student e-portfolios that follow a student through their career, between schools, and even on to college?

Exit Tickets, Recitation and Real-time data Checks for Understanding: Close Level. Teachers have always asked students to recite answers in class, raise their hands for class questions, and respond to in-class questions from the teacher to check for understanding. Technologies, such as classroom clickers, cell phones, and other smart devices have made such in-class instant checks easy to administer and instantly analyzable. Tools for conducting such rapid checks include SocrativePollEverywhereSurveyMonkey, the gamed-up Kahoot!, the collaborative Mentimeter, and a host of student response system clicker types. Here is a pretty good overview of the various types of clicker systems that use a device that students buy, so not their own cell phone, computer, or smart device.

School-based data systems and Learning Management Systems. FinalSite, Naviance, Powerschool, Schoology, Infinite Campus, PeopleSoft, HuskyCT (Blackboard). The number and names of school management and data systems is a long list. Sometimes it is hard for school data systems to talk to one another and this fragmented data can hamper efforts to coordinate data-driven decision making.

Analytics and Data mining. Here is a relatively comprehensive literature review of the topic for you edification. Enjoy, Papamitsiou & Economides (2014).

The role of Technology in DDDM

Take a peek at Infinite Campus, a school-based data system being widely adopted, including in my favorite schools in Ellington CT. Teachers, parents, and even students will have mobile access to a variety of school performance data, to help with decision making. Consider what it will be like to be a teacher in schools implementing these parent, teacher and student dashboard to data. Moving beyond data-driven toward “purpose driven” decision making with a variety of types of evidence, the technology is driving the availability of data and teachers will need to be fluent in data reduction and data summarization to access and help others makes sense of all types of data.

More About Grading Schemes and Scoring Options

You likely think you know how to grade a test. Count the number of correct answers, and divide by the total number of questions to get a percentage, right? Well, what if you deducted 1 point for every incorrect answer, so theoretically you could get a negative score (this would impose a penalty for guessing)? And what about blanks- conceptually is no answer the same thing as an incorrect response (what if you just overlooked 1)? Should a test have only 1 correct answer per question or should some questions allow multiple correct responses? What about asking multiple choice questions that also ask the student to provide a short written explanation for why they made a particular choice?

Next consider grading “on a curve,” meaning a bell curve or normal distribution of few A’s and F’s and mostly C’s. Students may like “curving up” when no one gets an A or the average grade is 60%, but will they also like curving down, when too many students get A’s? Conceptually it’s only fair to do both.

Finally consider the controversy of the “toxic zero.” (Edutopia 2017). This issue is intertwined with the issue of using averages (means) as the measure for learning across a variety of specific topics, or even an entire grading period or year of content. When averages are used, they can be strongly influenced by “outlier” scores that are very high or very low. In which case, a single late or missed assignment can make the difference between an average grade of A or C. Ultimately this is an issue of how grades are used in decision-making such as course placement or college admissions, and how other evidence might be needed to improve DDDM in education.

Resources

Technologies

  • Poll Everywhere
  • Socrative
  • Kahoot!
  • Google Forms for surveys

District level data analysis tools

 

Read More about…

For details about “smart” or adaptive testing read Zhang & Chung (2017) online (pdf).

t-tests vs ANCOVA analysis of gain scores

Analyzing gain scores vs using the pretest as covariate– the saga continues

Enduring Understanding- Student Learning Outcome from this module

This module intends to prepare teachers for ISTE Standard for Educators 7c that states: Use assessment data to guide progress and communicate with students, parents and education stakeholders to build student self-direction.