Fear of Black Boxes

A popular internet meme involves listing steps to reach an outcome, except one of the crucial steps is missing. An example might look something like this:

1. Build product
2. Give product away for free
3. Grow user base
4. ???
5. Profit!

This would be funny if it didn't represent the kind of thinking that is prevalent in education today. I attended an ed tech meetup last night where we discussed adaptive learning. There is a widespread assumption that the educational components needed to make an adaptive learning system work are in place and that adaptive learning is poised to revolutionize education once the technological challenges can be worked out. What makes this assumption completely ridiculous and more than a little terrifying is that it is being made by people who can't be bothered to do the minimum due diligence to see if it is true or not.

One person walked up to me after the meetup and asked me if and how adaptive learning systems are being used in schools. Only one out of ten decided that it might be worth engaging in some inquiry before pronouncing adaptive learning systems as the next big thing. He was surprised to learn that schools have been using adaptive learning systems ever since I could remember. A teacher gives you a quiz. The quiz is scored. The teacher reaches into the filing cabinet and pulls out the next worksheet based on that score. You do that worksheet. Lather. Rinse. Repeat. Adaptive learning systems are most commonly used for remedial math or for math skills practice throughout the year.

When I was curriculum coordinator in Holliston, I was sometimes called upon to assess a student and then suggest strategies to the teacher for working with that student. Erin had been a top math performer all through sixth-grade, but was suddenly struggling to get C's in seventh-grade. Erin had an eidetic (photographic) memory, and as I was working with her and probing her thinking, it became clear her primary strategy for solving a math problem was to flip through her memories until she could find one where her teacher was solving a similar problem on the whiteboard. She would then mimic what the teacher had done. This learning strategy had served her well and it was all she knew, but suddenly it was no longer working and she was in tears over it. Her entire sense of self was on the line.

Increasing Erin's engagement level wouldn't help. Giving her more practice on skills she hadn't mastered or presenting concepts in alternative ways wouldn't help. She needed to transition from her dominant learning strategy to another one. While Erin's situation may sound extreme, it isn't. Her learning strategy stands out as different from most (I picked this example because most people would have no trouble seeing that her particular learning strategy doesn't scale very well), but most of us have primitive learning strategies that don't scale very well. We just don't see it because our learning strategies are similar to the ones that everyone else has. Just ask someone how to study for a science test. The most common approach is to re-read your notes or the textbook. In math, it would be to redo old homework problems. We do these things because we don't have better strategies. It's all we know. But do they sound like good strategies for learning math or science?

My point isn't that adaptive learning systems are incapable of diagnosing issues like Erin's. I don't think they can right now, but I have little doubt that eventually an expert system will come along that can pretty much do everything that I can do. My point is that, in trying to work with Erin, I encountered dozens of open questions... questions that the research base doesn't cover. To make an adaptive learning system work, we need to start filling in those gaps. Unfortunately, I see lots of gaps that no one is working on.

As educators, we drastically overestimate the number of best practices that have been identified. Most practices used in classrooms developed out of expedience. Teachers need to move students from point A to point B, and they need to do it quickly. Just like many businesses are focused on their next quarter, many teachers are focused on what they need to cover and what their students will be tested on at the end of the year. The result is practices that are designed to cover over issues instead of uncovering and fixing them. Uncovering and fixing issues takes time, and if you throw resources at it, you may not make next quarter's numbers. What makes this even more insidious is that even well-intentioned teachers will see a practice that everyone else is using and assume that it must be the best practice, and not simply a practice that we started doing because it gave us quick and dirty results. This is why most highly trained educators erroneously believe that the best approach for working with struggling students is to break things down for them and make learning more procedural.

For most people, what happens inside the classroom represents a black box that no one wants to peek inside. Policymakers use test scores to "motivate" teachers. They assume that if they increase the stakes and the level of accountability, that the stuff that happens inside of the classroom will take care of itself. They assume that teachers know what to do and just need to do it. But I can tell you, that is not the case. We don't know what to do. When you put pressure on us, we will do more of what we know how to do, but that really doesn't work. You can put a gun to my head, but flapping my arms faster and harder is not going to help me to fly.

Increasing student engagement is another way to avoid the black box. So is the idea of using technology to enable students to produce knowledge instead of consuming it. Walk me through, step-by-step, how these increase student learning without any other parts of the system changing because I've got plenty of counter examples to show that it doesn't happen automatically. If you want to have an impact on learning, you can't be afraid of the black box. Writing "then a miracle occurs" in a cartoon or "here be dragons" on a map may be cute, but it just isn't going to cut it.