Why we need to be careful when analyzing student data…

square digital-388075_1920Achievement data can be really powerful in helping us better understand our students (their strengths, their needs, their interests, etc).  Analyzing this data can help us meaningfully plan interventions, supports, and approaches to engage our students.

But how much can we trust the data we’re looking at?

Well it depends… some data sets are more accurate then others.  There are certain factors that can make a certain data set less reliable.

Let me give you an example.

In schools, we often look at cohort data to analyze how a group of students is doing as they progress from year to year.  Let’s say we follow this year’s group of Kindergarten students in our school for the next 4 years (i.e. from K to Gr. 3).  Perhaps we analyze their report card achievement for Reading.  As we compare their Kindergarten results to Grade 1, 2 and 3, we can see if the cohort is improving or falling behind over time.

But again, how reliable is this data?

There are many factors that come into play that can greatly skew our results.  Let’s just consider one factor: students coming and going. Within a given school year many students come and go. Depending on your school, you may have a lot more coming and going then others.  Let’s say for argument’s sake that each year you have 5% of your students leave to go elsewhere and also have 5% new students come into your school.  Think about what this means when analyzing cohort data that spans 4 years  After each year, 10% of the students in the data are new!  When analyzing how students did in Kindergarten compared to Grade 3 in a given cohort, you may be looking at a group that has had 40% turnover over those 4 years (i.e. only 60% of the students in your Grade 3 cohort actually attended Kindergarten at your school and contributed to the Kindergarten data collected).

This is just one factor that can greatly skew your data.  Another factor that can affect the data is students having different teachers each year (with potentially very different approaches to assessment).

Now, don’t get me wrong.  I am not saying we should avoid looking at this data. I do think however that we need to be careful when analyzing it and be sure to triangulate it with other data sets (both quantitative and qualitative).

There are some factors in data sets that should raise a red flag and signal you to be more careful when analyzing it. Factors such as:

  • Data that is collected over a longer period of time (multiple terms or multiple years)
  • Data that involves large cohorts of students (grades, schools, districts)
  • Data that involves more subjective assessment (ex. analyzing achievement in the Arts)
  • Data that involves different cohorts of students (ex. analyzing Grade 2 data of many years… this would involve a new cohort each year, and we know how unique student cohorts can be)

There are certainly more factors than those listed here that can skew your data.  So be careful when analyzing student data, and be sure to consider any factors that could be affecting this data to make it less reliable.  And be sure to triangulate the data with other data sets, as well as your observations/conversations.

 

Leave a Reply

Your email address will not be published. Required fields are marked *