Friday, September 14, 2012

The hard problem of online learning

Idea: Kids learn things on their own, at their own pace.
Problem: They'll get stuck, and frustrated.

Idea: We'll give kids feedback automatically that doesn't need the attention of a person, so that they can still learn stuff on their own.
Problem: That's complicated.

Idea: We'll use computers.
Problem: Computers are pretty cheap, but the quality of the feedback isn't very good.

(See also: Turing Test)

This is a really tough problem. Nobody has any ideas that seem entirely promising. Maybe we just have to wait for the technology to improve.

Here's my pitch for a shift in the way we think about this problem.

In the world of "giving feedback to kids on math" there are two contestants. There's (1) people and (2) machines, and people are beating the stuffing out of machines. Truth is, machines haven't ever shown that they're up to snuff, as far as quality goes.

But humans have issues too. A single human can only be in one place at a time and can only focus on giving feedback to one kid at a time. We people have all sorts of stuff to do besides slavishly providing kids with quality feedback. I mean, unless you're a teacher. But, then you have to figure out a way to give quality feedback to a few dozen kids at once. Humans are limited in a way that machines aren't.

But humans are making progress. Technology is the key, here. Advances in machines have allowed humans to group together to overcome some of the limitations of being a person. If you mess around with Wikipedia,   a human being will find out and fix it. Closer to our discussion, if you ask a question on Math StackExchange you'll get a good, quality response before long. If you ask a teaching question on twitter, you can also depend on getting a good answer.

The lesson here is that the humans are making progress.

Nobody has figured out a way to create a site where math students can get quality human feedback on any topic they're studying. Nobody has figured out a way to get quality feedback out of machines. Pretty much everybody who's working on online learning is trying to figure out a way to help the machines. In the meantime, sites like StackExchange or Physics Forums keep on building and improving their communities.

I've got no idea how to build a site that has a quality, supportive community of students and adults who will provide quality feedback to K-12 students. But there are places on the web that are like this. And if you can sustain a vibrant online community, then you can start creating more difficult tasks for students online. The rate of innovation in online curricula could speed up quite a bit.

Creating a quality community is a hard problem. Creating a piece of software that can give excellent feedback is an attractive and lucrative problem.

The pitch is: focus on the hard problem, not the attractive one.


  1. The extra layer of challenge is that sites like StackExchange work solely on giving you the answer. Using your expert knowledge to answer someone's question is relatively easy.

    However, good teaching doesn't hand you the answer - it *hints*. It prompts you just far enough that you get unstuck and then the moment you're going again, it sits back and watches you fly. What's worse, sometimes good teaching means letting someone struggle with some amount of discomfort before you jump in to offer assistance. Hint too early and you shortcut the learning.

    How do you do that when a) you have no non-verbal cues to gauge someone's discomfort levels, and b) you're communicating asynchronously?

    1. (1) I agree that those are problems, though I've had some success in my own learning in the "homework" section of Math StackExchange. There are communal norms that are built in there: give hints, not answers.
      (2) Imagine you created software that could give super specific answers to questions that kids had while they're learning, even if they weren't hints. We'd consider that a huge breakthrough. But I think leveraging online communities to get to that breakthrough is way more available, if less flashy and lucrative.

  2. There is a guy who I saw speak at AMATYC in Boston, fall 2010 who has developed a system that gives useful feedback on students' algebraic equation solving. The really cool thing is that it has students input their work (not just the answer-imagine that!) and can find errors and suggest appropriate next steps that are responsive to what the student seems to be doing.

    I don't remember the guy's name, nor his software. He makes it freely available and talked a lot of smack about traditional publishing companies and their online systems that only provide right/wrong as feedback and that are unresponsive to student strategies.

    Perhaps some intelligent Googling would turn it up. You'd find it an interesting crude first step at some of what you point to here, I think.

  3. Really interested in this problem. To clarify, it seems that what you are most interested in is feedback, not grading . . .
    A few thoughts:
    1. I think that a lot of thinking would have to go into providing adequate incentives for answerers. On MathStackExchange, StackOverflow, etc, the questions asked are (presumably) hard questions about real initiatives. The person who answers one of these questions can derive a real sense of accomplishment (for having the info) and contribution (for adding to a real enterprise). K-12 math problems, by contrast, are usually easy, and are almost always "fake," which might limit answerers' sense of achievement and contribution - meaning the community would have to rely on altruism, compensation OR an as-yet-unseen (by me) alternative.
    2. The no-tech approach to good feedback is to spend less class time previewing and more class time reviewing student work (a la Exeter). As far as I understand it, Exeter does very little previewing, assigns very well-scaffolded problems, and has students put most or all of the problems up on the board the next day.
    3. I'm working on a platform for a similar community, but focused on creation of good problems instead of good feedback. Working on it has me thinking that if problems and "resources" (e.g. videos) are sufficiently granular and aligned, pseudo-feedback could be achievable (similar to what Khan is trying to do, but with less multiple choice and more targeted videos).

    1. Good points. I want to focus on your first point. Who could conceivably be motivated to give good feedback online?

      1. Other kids. 8th grade math is trivial for many adults, but not for 10th graders.
      2. Education students.
      3. People interested in math education.
      4. People interested in developing online math tools, or piloting them. Imagine an open-source testing ground for curricular materials or modules that is supported by this kind of robust feedback system.

    2. Good thoughts.
      A. Do you think teachers would be motivated to contribute?
      B. Really interesting fourth point - took me a while to wrap my head around it. I think the idea is that curriculum developers could "test drive" lessons or tasks and interact with struggling students in real time (--> optimization of materials).
      C. Another interesting question is - what technology is best for reaching the maximum number of students? A friend and I did some research on a similar (but much simpler idea) and concluded that camera phones are the way to go (95% of high schoolers have them). Our idea was that students could send a picture of their homework to an email address (using their camera phone), and the site would respond with an answer key. Then, students could correct their own work and re-submit before class the next day. Not as sophisticated as real-time feedback, but could abbreviate misunderstanding.

  4. This post discusses online math communities that support teachers and students. I discover such a community 6 years ago. Through support of other professional teachers I realized that learning math is a journey and kids do learn at their own pace, and so do we as adults. My students worked on one problem over a two week span. Through an online tool they posted their thoughts and frustrations. As a teacher I went in and gave them encouragement through a piece of information to lead them along their way.

    If I was uncertain I could always ask Dr. Math at Within this organization they have a library of online tools that I can explore for any grade and math concept. They have not developed them but organized them into one virtual library. I have found it a valuable tool to quickly look up something to visually teach a concept.
    Your comment about ¨I've got no idea how to build a site that has a quality, supportive community of students and adults who will provide quality feedback to K-12 students.¨ Math Forum, as far as I know, does not have a community of student providing feedback but does have a group of adult that will provide feedback. I do not believe computers can replace teachers but are a tool t9 use to support learners. I agree correct answers at the end of a book do not support learning, they just tell the student that they have it wrong.

    1. I am also part of that adult math community with Glenys at I agree there is value and I alo think this is how technology can help us grow and learn as teachers. Another thing the Math Forum provides is professional development online courses. Within these workshops mathematical discussions are centered around problem solving and teaching. Communities are formed within the workshop and can continue beyond the the 8 week workshop. Glenys and I are an example of these continued teacher communities six years later.

      Technology can't replace teachers but it certainly has its place in creating teaching and learning collaboration communities such as the Math Forum.
      Technology can provide limited feedback and tutoring that is just one thread of the complicated path of learning. In addition, technology collects data and information that can generate a rich classroom discussion deepening concepts and problem solving thinking when it comes to the hard problems. Learning is a complex process and we teachers should take advantage of any valuable technology that supports this process.

  5. I noticed that you said, “Pretty much everybody who's working on online learning is trying to figure out a way to help the machines.” I wonder if this is the main problem. As you inferred, teachers are overwhelmed and can only help so many children at once. However, I think that too many software developers (and, unfortunately many educators) think that the solution is to take teachers out of the equation. I worry about many of the math practice options that are out there. If one of my students added 1/3 + 4/9 and got 5/12 I want more than a pop-up that alerts her that she needs a common denominator. Similarly, I worry about many of the popular online tutoring sites. Even though there is an actual human being there is no guarantee that their philosophy of mathematics education is the same as mine. I care more about the process than the product and providing procedural hints to get the correct answer – whether done via machine or people – does not constitute quality feedback in my book.

    I agree with Glenys that The Math Forum is the sort of “quality community” that you referenced. They just celebrated their 20th anniversary. For the first nine years, they wrote back to every student who submitted a Problem of the Week solution. A software revision in the late 1990’s made it possible for volunteer mentors and these actually did include students. When they made the move from Swarthmore to Drexel, their staff was cut by more than 50% and they moved to a “self-mentoring” system where students could check their own answers. What I love, though, is that whether the answer is correct or not, students are given questions to consider and help move their thinking forward. Staff members and volunteers continue to provide more individualized feedback for a small fee. In addition, teachers have the option of being trained to mentor their own students.

    I did this as part of my coursework in Drexel’s Master of Science in Mathematics Learning and Teaching. To help manage the workload, I usually gave my students a one-week deadline so that the submissions were staggered and offered a 48-hour turnaround time for feedback. Of course, it can still be overwhelming at times, but the benefits far outweigh the drawbacks. Students who are too shy or embarrassed to ask questions in class may express their confusion privately in writing. Writing also allows students to examine, express, and keep track of their reasoning, which is especially useful when ideas are too complex to keep in their heads. Furthermore, when writing is collected and saved like it is by The Math Forum it can act as a platform on which other ideas can be built because students can go back and re-examine, reflect on, and refine their ideas as they integrate new learning with old. Finally, the online environment helps students develop an appreciation of the need for precision in communicating ideas in math.

    You might want to check out their upcoming EnCoMPASS Project ( It’s focus is on enhancing teachers ability to provide quality feedback through the development of an online professional community.

  6. Every now and then, I get my nose far enough away from the grindstone to explore issues about teaching math, and sometimes (not often enough, I'm afraid) I'm able to take corrective actions. Teachers need feedback, too! This blog provides just such a blast of great thinking about teaching math... I've also just read the September issue of Educational Leadership (publication of ASCD) which focuses on feedback... mostly teacher-to-student, but there were mentions of other kinds.

    In a possible answer to Michael Pershan: Chris Sangwin at the University of Birmingham (England) has developed software that can engage with students and provide feedback (STACK, details at His students use it collaboratively; they have different randomized problems, and discuss the process and feedback given by the system. I'm trying to use LON-CAPA in a similar vein, though with this the students press a "feedback" button which passes their questions to me. They can also use a "discussion" button, which opens up their questions to the entire class. Pretty limited, and still requires (allows?) teacher interaction.

    Teacher feedback is vital to our professional growth. It is all too easy to stagnate. A great value to me of a community such as MathForum is that I can share ideas and questions (things I notice and things I wonder about the art of teaching), and learn both from what others say and from how others respond to what I say, do, and try with my students.

  7. I am also part of the Math Forum community. You need to explore their website. It is awesome! I have used their Problems of the Week both with paper and pencil (old school) and by having them submit their answers with detailed explanations online. They love it! They have the chance to use technology and they have to explain to me their mathematical reasoning. I in turn give feedback and encourage those who need it to keep working towards a reasonable solution to the problem.