Wednesday, April 24, 2013

You can't really reassess an individual skill

Depending on how you define Standards Based Grading, it gets a lot of things right. You're more likely to get an accurate picture of what someone knows by assessing a skill more than once. What you know now matters more than what you didn't in October. Students need an accurate picture of what they're studying, and "Test #4" doesn't provide that.

Great. But here's something about SBG that's been bugging me for a while.


There's something wrong here, but what is it? 
  • The kid showed that she knows all the triangley stuff, but dropped the ball on the square root side of things. She gets a 5/5 on Finding Sides of Right Triangles, but gets a 2/5 on Understanding Square Roots.
  • The kid got a question about right triangles wrong, so she gets a 3/5 on Finding Sides of Right Triangles.
Neither of these ideas is quite right. Knowing how to find the square root of 1 is not an all-or-nothing affair. Understanding isn't binary. Rather, understanding comes in degrees, and if a piece of knowledge is weakly understood then it's especially likely to falter when under pressure. 

If you aren't super-comfortable working with right triangles, trying to solve a right triangle problem will be mentally taxing, and when you engage in mentally taxing behavior, you mess things up. But you don't mess up the things that are rock-solid. I doubt that I'll mess up single-digit addition when working on a Calculus problem. Rather, when you're using up mental resources it's the infirm and tentative knowledge that falls apart.

It's the sort of thing that we see all the time on mathmistakes.org


This student said something silly, but it's artificial to attribute this to either his understanding of solving quadratic equations or his understanding of what the equation symbol means. It's both.


Would you ask this student to reassess on Doing Arithmetic with Negative Numbers or Finding Equations Given 2 Points? Neither? Both?

There's a larger point here. The idea that you can create a quality assessment that targets an individual skill is a myth. Take the slope question above. You could make the numbers easier so that the arithmetic probably wouldn't be a problem. For instance, you could use (0, 4) and (2, 10). But this is far too easy -- understanding means being able to apply a skill to a difficult context. So you toss in more difficult numbers, but then you're no longer purely assessing a kid's ability to find a line that passes through two points.

I don't know what this means for SBG or reassessing, and I hope that (in addition to challenging the premise of my post) we hash this out in the comments. Maybe this is an argument for fewer, but more substantive standards, like "Doing Stuff With Lines." I'm not sure, though.

12 comments:

  1. This is the problem that I have with SBG and (part of) the reason that I stopped after doing it for a year. I'm looking forward to the comments.

    ReplyDelete
  2. I agree that you can't focus in on just one skill and I have found that out the hard way. Years ago when I started writing my SBG quizzes, when I would grade them I would find that the questions I might have asked ended up assessing something in addition to what I intended. Or the mistake that were made were related to skills other than what I was assessing.

    However, we can't make skill lists of every little skill (can we?)...the ones in our curriculum and the ones that are prerequisites to our curriculum...if we did, the list would be long and overwhelming for both student and teacher. I see the embedded skills usually encompassing skills they should have mastered prior to my course, or the basic skills if I may call them that. They really make a lot of number sense mistakes and while they may understand the major skill I am testing them on, do I take points off because they aren't attending to precision? I don't know now, but I can say that I currently do. They get a 3.5 instead of a 4 and man do they dislike those 3.5's.

    I think this post is great and should open up a great conversation about SBG, grading and how to break out skills. I will be interested to see where this goes!

    ReplyDelete
  3. Yep, that's why I do something a bit bigger. I set up units that break the material up into chunks that work for me. After each unit I give a test that's composed of 2 to 4 mini-tests, which are the concepts that go nicely together. Students may re-take any mini-test. (Re-take means they get a different test with similar sorts of problems.)

    Regardless of this student's weakness with square roots, they also don't get that the hypotenuse must be longer than the legs.

    ReplyDelete
  4. I think if you dig deep enough, you may even find that labeling skills as different from each other is somewhat challenging.

    We do our best in our curriculum documents, but the space between one skill and another is somewhat murky at best, which is certainly not represented by any gray area in, for example, the Common Core.

    ReplyDelete
  5. There's some interesting fodder for discussion here, but a couple caveats first. One, I've only been doing SBG for a few months, so I'm a relative newbie. Second, I do it at the college level, so the context is somewhat different. Not sure if that matters, though. Also, I teach physics, not math, but I don't think that matters.

    My way of thinking about this conundrum is to try to separate out the summative and formative purposes of assessment as much as possible. For summative purposes, each assessment in my classes is tied to exactly one standard. So there's never any question (for summative assessment purposes) about which standard is being evaluated or which standard would have to be reassessed. I just evaluate each student on his or her ability to demonstrate mastery of that standard, regardless of the reason the student might fail. (So maybe part of the resolution is that I'm evaluating "ability to demonstrate mastery" instead of mastery itself?) When I'm grading an assessment for summative purposes, I don't worry about the root cause of a student's difficulty in demonstrating mastery. So, for example, in your middle example, assuming the standard is something like "I can find the roots of quadratic equations", it's clear the student has not demonstrated mastery. I don't care (for summative purposes) whether that's because the student doesn't understand something fundamental about variable or equations or something else.

    Of course, the assessments also serve a formative purpose. Now things like root causes and trying to understand why a student made a particular mistake (or, more generally, why they wrote or said what they did, and what they were thinking) is the critical thing. But it's okay if this trying to untangle things is messy or the skills overlap or can't be cleanly separated or whatever. So long as both instructor and student get the right feedback on how to move forward, I don't see a big problem here. I'm not saying this part is easy, I just see it as a very different process, with a different purpose, than the summative part of the assessment.

    ReplyDelete
  6. If SBG is done at a granular level (find the slope between two points), I don’t see how you can tell whether the student has understanding or is memorizing.

    You get a better gauge of understanding if SBG is less detailed (effectively answer problems that require analyzing a linear relationship). This does not provide very specific feedback to the student or teacher, but it is probably more honest feedback. The main issues are not that a student can’t accurately calculate slope, find intercepts, etc. The main issue is that they cannot or do not analyze linear relationships. In your slope example, the student would likely not continue to make that mistake if they analyzed the relationships involved each time they did a problem like this instead of following steps and moving on.

    I like the re-assessment aspect of SBG but don’t really understand how it can work as a detailed diagnostic tool. Doesn’t that reduce your assessed coursework to a whole bunch of little ideas?

    ReplyDelete
  7. Yeah, this is a big problem I've been having too. My best attempt at SBG (nowhere near expert) thus far resulted in quizzes with different questions all around something small. So my "exponents quiz" had four or five different questions that all had to do with the basic definition of exponents, some whole numbers, some negative, some fractional, some where the exponent was the variable you were trying to find. By having those different aspects, I could give the kid a grade on their understanding of "exponents", but then they had to look at what they got wrong to see what they needed to work on. This was not clear to them.

    ReplyDelete
  8. I agree, you can't assess one discrete skill without making the problem overly simple. But like Cornally says, SBG is a gateway drug to awesome!

    I think one of the keys to making SBG work is having different categories of targets or assessments (as described by Mathy McMatherson http://tiny.cc/8yy3vw). I use the terms Fundamental, Core, and Advanced (get it? F,C,A). In physics we have a learning target right now that says "Calculate energies using observable quantities". For this learning target, Ss are given various measurements and asked to calculate either kinetic or potential energy. It's the smallest chunk we're willing to put out there and the feedback is simple ("you dont have the right answer, better fix it!"). A Core target has a little more depth to it. For instance, we could ask them to apply conservation of energy to a system. In this case, it's a little tougher to break down exactly where a mistake may have come from, but that just means more complex feedback is required ("How about making some bar graphs to compare the energies before and after the event?"). Then, I believe an Advanced target should require Ss to create something. Currently, we have Ss creating screencasts in which they decribe the energy transfers going on in a youtube video of their choice. Part of that target is the ability to choose a good video in the first place. (That's a great example of a skill that makes no sense as it's own target.) Advanced targets require even more detailed feedback ("this video is no good, there's not enough obvious energy transfers there! what might make for a better video?")

    So, I'm definitely assessing more than one skill in any of these categories, but the student still has an idea of what they need to do to get better. So while I agree with the blog title, I'll also never give up my SBG system. One of the main reasons is that right now as I write this at lunch, my room is full of Ss helping each other and getting help from me. And I think the reason for this is that they either know exactly what they did wrong and want the opportunity to fix it, or they're not sure but they want to know.

    ReplyDelete
  9. Wow. I suppose I'll make my comment short.
    This year, we (8th grade Algebra teachers) had the same warm-ups in every class. One integer problem, one "Solve an Equation" problem. We drilled those two skills the whole year.
    The result? Students made fewer integer errors and equation errors because they saw them every day.
    Certain skills are embedded into all mathematics, and viewing thoseskills as a checklist is--as you said--silly. Before we begin the unit on factoring, integer operations must be solid.

    Those skills aren't boxes to be checked, they're more like the foundation of a pyramid; without them, the higher-level mathematics blocks building upon them will crumble.

    ReplyDelete
    Replies
    1. Those skills aren't boxes to be checked, they're more like the foundation of a pyramid.

      I bet those kids knew their integer operations and equations. No doubt.

      I think the pyramid metaphor, though, needs to be modified. While it may be true that some skills are more foundational than others, no skill is *truly* foundational. With credit to Quine, I prefer to think of skills/knowledge as a web, one with tons of connections. In a web, some skills are more central than others, but it's impossible to say which need to come first. Any weakness in the web, in principle, weakens the whole thing.

      Students go at a hard math problem with *all* of their math knowledge.

      Delete
  10. "Maybe this is an argument for fewer, but more substantive standards, like "Doing Stuff With Lines.""

    I teach a foreign language, so everything is related to everything else and it is really difficult (impossible?) to tell where the student's misunderstanding is coming from. I have actually reduced the total number of standards to three--yes, three. In this way we continually reassess the same standards over and over but the content of the assessments gets more and more difficult as we move on and acquire more language.

    ReplyDelete
  11. Why couldn't you set up two or three other problems that aren't "h = 1" and let the student solve those?

    Assuming she knows her "triangle stuff", if the problem proves to be a lack of understanding of square roots or something like that, you can tackle that separately.

    Yes, I've struggled with that in writing my standards. However, wouldn't that student's answer just be "wrong" under traditional grading? And wouldn't that wrongness be hidden under a heading like "Chapter 8 Section 3 Quiz"?

    ReplyDelete