Friday, October 28, 2011

Lies, Damned Lies, and Statistics




Somewhere in your school district is a group of people who care a lot more about standardized testing than you do. Since you're reading this, you may be a Slacker, which means it's possible that someone in your school district cares about lots of things more than you do. Still, I'm guessing it's true for everyone else as well, assuming, of course, that your name doesn't start with "Dr. Betsy," or "Dr. Dennis." Oh, dear, I hope your name doesn't start this way...

Because there are very smart people who care a great deal about standardized testing, there are lessons that can be gleaned from what your school is currently doing to game the system. I don't mean cheating, although people have tried that too (see here and here). I mean perfectly legitimate ways that schools use to improve test scores, many of which require little or no additional effort. Here are a few tricks, and ways to adapt them your own Slacker ways.

What they know:
Statistics matter - Schools must make (or "meet," as we all seem to say) Adequate Yearly Progress. If they do, they get to keep their funding, get to stay open, and get a little plaque celebrating their status as "Adequate." This is determined, however, through a more complicated formula than you might expect. Rather than a simple matter of reaching a passing score, it is tied to all sorts of statistical trickery. Like the near cousins lies and damned lies, statistics can be used to tell a story--and best of all, you choose the story. My school met AYP not by having our scores reach a certain threshold (they didn't), or by making significant improvements in scores (they may even have gone down). We met it through a morass of "safe harbor" and "confidence intervals" and other mathematical voodoo that defies simple explanation. If you'd like a complicated explanation, see the 54 page document called, and this is funny, "Understanding Your Adequate Yearly Progress (AYP)."
How you can use this:
Spin and statistics are closely linked to one another. You're probably already well-versed in some forms of comparative analysis--but everyone failed that quiz, or no one turned that in on time. However, you should experiment with some others as well--yes, I failed that, but right now I'm failing many fewer things than last (month, year, time I was in 9th grade...). Also, keep in mind that 87% of statistics are made up on the spot, so don't spend a lot of effort on the research.

What they know:
The report itself matters - You feel like you spend the entire month of April taking high-stakes tests. In truth, you spend three days in April taking high-stakes tests. You spend the rest of the month taking relatively low-stakes tests. Your school knows that only the Math and Reading PSSAs count toward AYP, and consequently get reported in the newspaper. We pretend to take the other tests seriously, but we've already shot our collective wad on the Math and Reading and basically just coast through the others.
How you can use this:
Kids with good grades do this sort of thing all the time. In your syllabus (if you've lost it, you can probably find the same stuff in the online grade program) is some kind of formula that tells you how much everything is worth. In there you will find the high-value stuff, and some general padding to make things look more legit. Many teachers communicate this by dong a "total points" calculation--10 point quizzes are worth exactly 10% as much as 100 point tests. Others use the "weighting" option to tweak the points so that quizzes have a certain value regardless of how many questions are on them. Either way, figure out what something's worth before you put effort into it, or not.

What they know:
Take care of yourself - Schools have studied data that says students do better on testing if they've had breakfast. So, we feed you breakfast on days of testing that we care about. We can't control how well-rested you are, but we'd take control over that too if we could.
How you can use this:
Take care of yourself. There are limits to what this can do for you, and I've probably tested those limits as much as you have. Skipping the studying entirely in favor of a good night's sleep means that you will ready for peak performance, but will have no information to write down on the paper. Still, it's a handy excuse for giving up and going to play video games.

What they know:
Demographics matter - However, not in the way you may think. They matter in a sort of Bizzaro, bass-akwards way. It probably started with the fact that standardized testing has a long history of problems with racial bias, which you can start checking out here. Rather than deal with the underlying problem, NLCB (we don't say "No Child Left Behind," or even "so-called No Child Left Behind" anymore because everyone knows the name doesn't match what the legislation has actually done to education) pretended to care through an unfunded mandate. For AYP calculation, if your school has a high enough number of minorities, or economically disadvantaged, or even special education (see below) students, these subgroups must meet the higher standard on their own. In other words, if your demographic is well-represented in a school, you no longer only belong to the student body as a whole, but to your a sub-group as well.
How you can use this:
You can't really. If you're a minority, you're entitled to any advantages you can get. If you're not, it's too late to start.

What they know:
IEPs matter - IEPs matter in standardized testing in much the same way demographics matter, that is: kind of backward. There are some adaptations made for the testing experience itself, but Pennsylvania has exempted itself from adapting the content of the PSSA. This means that you may be in 11th grade, taking the equivalent of 8th grade math, but you will still take the 11th grade math PSSA.
How you can use this:
You can't. Not here anyway. We'll do a whole thing on what IEPs can do for you, and how to get yourself one, but some other time. For this, it doesn't really matter.

What they know:
Slow and steady - The best way to prepare for any large and important task is to prepare continually and non-specifically. If your school has a "daily vocabulary word" or a "math problem of the day" or a "Drop Everything And Read" program, it may be something based on this theory. The College Board even publishes a Question of the Day for the SAT, probably to prove that they are our benevolent overlords.
How you can use this:
My father has a theory that if he would lift a small child every week at church that his strength would increase in line with the child's growth, so that eventually he'd be able to lift a full-grown person. He's never had the attention span to find out if this works, so he can just assume that it would. The merit of this theory is that a drop of water can level a mountain, given enough time and enough of its little friends. We have a similar axiom in music: ten 6-minute work sessions are much more likely to stick with you than an hour straight. It's a tough one for Slackers to make use of, especially since our efforts tend to come in the form of furious sprints from behind, rather than slow and steady progress. It is worth noticing, however, that if more benefit can be gained from the same amount of work, this is probably something we'd be interested in.

What they know:
Environment matters - If you take the SAT in a room with a marching band practicing right outside your window, there's going to be an incident report filed. For the PSSAs (the ones that that matter, anyway), our school converts the cafeteria to testing central. We adjust the bells so that there are no distractions during the tests. We provides nice long breaks, complete with thumping pop music to get the blood going. Even the cafeteria workers prepare lunch on those days in total silence so that nothing interferes with the testes' (testers'? testingers's??) efforts.
How you can use this:
Again, Slackers are rather sunk on this one. If we do our homework, it's going to be on the bus, or in front of the TV, or during chorus. These, as you know, are terrible places to try to concentrate, but they are better than skipping the work entirely (except for chorus. That should just never happen.) Still, if something is of high point value (see above), requires concentration (even from you), and your girlfriend isn't taking the same course that semester, you really should try to find a work-space that gives you a fighting chance.

What they know:
Cut everything else out - This sounds ridiculous, but it's more common than you think. A friend of mine was interviewing for an administrative position in a school district and was marveling to the interviewer about his school's test scores. He said, "Well, our kids don't play the flute." The truth is, we can make the test scores whatever you want, its just a matter of how much of the rest of it you're willing to cut away.
How you can use this:
Don't use this. Okay, maybe it's good to have a thought about how many activities you can add to your schedule and still contribute well to all of them, plus function in school. On the whole, however, don't do this. School is important, but the other stuff you do may prepare you for life just as well.

It's tough for Slackers to care about grades. In fact, if you want an excellent rant on getting rid of grades entirely, check out this exceprt from Robert Pirsig's book Zen and the Art of Motorcycle Maintenance: An Inquiry into Values (more on this another time as well). It's tough for real educators to care about standardized tests. The problem with grades and standardized tests is that our well-developed rants about them fail so badly at making them go away. For now, it's probably worth it to game the numbers so that they come out more in our favor. In fact, proving that we can do this so easily may be more effective than the rants ever could be.





Thursday, October 20, 2011

And What are You Going to Be When You Grow Up?


At some point people stopped asking me what I was going to be, and started asking what I was going to do. I don't know if this was because the '70s ended and some kind of fad ended along with it, or because I wasn't a little kid anymore. Either way, it sets off my political correctness alarm. I guess it's good to acknowledge that what a person does every day for money shouldn't entirely define who he is; however, I'm not so sure it's works as well for kids.

I think too many people choose what to do without taking inventory of who they are, what they like doing, and how they would spend their time if they had a real a choice in the matter. Some of this may be due to the remarkable achievements that young people make so early in life. Kids are already soccer players at 4, science majors in 9th grade, and have selected their dream colleges before they're out of middle school. Then, a few years later, they get a degree in German Renaissance Literature and a graduate degree in Ancient Runes, followed by a job in a coffee shop and a move back home with their parents. Much of this is due to the fact that we don't take real vocational training very seriously anymore, but that will have to wait for another post.

Three things before I go on:
  • This is not just when-I-was-a-kid-things-were-different curmudgeon crap. I know that my own generation was a part of making things this way. Even if things were a little different when I was younger, many people my own age fell victim to the same wrong-headed thinking anyway.
  • This is not to blame school counselors. I say this not only to shield myself from the very lovely; very, very smart; and very, very, very powerful women who work in the guidance office at my school. I say this because I think a lot of the shoddy career discernment is happening before the professionals are even involved. By the time you're taking the SATs and applying for colleges, you've already got some pretty firm ideas in place. Besides, school counselors are more in the business of helping you to achieve your dreams than changing them.
  • This could be like one of those weight-loss methods, or get rich in real estate things--just because something worked for me doesn't mean that it's a good method. In fact, that axiom is probably a Slacker Guide topic all on its own.

The problem with the way we choose a career these days is that students tend to respond to the wrong set of data. "I like science, therefore I should be a scientist." Well, what is it you like about science? Is it being introduced to a novel, thrilling, and vast concept every few weeks? Is it engaging in something mentally taxing and working your way through it? Is it working with a teacher you really admire for her intelligence and enthusiasm?

If so, you may find that the daily work of many working scientists--e.g. 1. Distribute a solution to a reagent with a micro-pipet. 2. Repeat. 3. Repeat...--isn't quite the same as what attracted you to the sciences in the first place.

Choosing what you're going to do every day for forty years or so should involve some measure of looking at what these people do every day. It's a complicated set of variables to manipulate, but try this simple flowchart (this one is for music careers, but it can easily be adapted to your own interests):



If it's not entirely obvious how this works, take yourself through an example. For instance, if you would like to work outside, be your own boss, and get paid for what you produce (rather than punching a clock) you could either be a rock star, or a bum--depending on earnings and fringe benefits. Notice also, if you end up in music teaching and find it lacking, you can move over to retail with similar schedules and earning potential. Similarly, if you enjoy the perks of recording engineers, but would like to avoid all of the actual work, you can slide over to record executive...or, once again, bum.

If the above flowchart isn't as user-friendly and obvious as I intended, try these tips:
  1. Pay special attention to what people in your potential career actually do at work. Case in point: I very seriously considered becoming a priest. Aside from the fact that I really liked girls (more, perhaps, than the average guy did--especially when you factor in the guys who turned out to be gay), I see now that what I actually wanted to do was say Mass--probably not even say Mass, but chant it. In Latin. I didn't think about visiting sick people, attending parish council meetings, and writing (not just delivering, but, you know...preparing) homilies.
  2. Don't be suckered into something just because you're good at it. We all get a lift from doing something well and being praised for it. However, eventually the amount of praise goes way down, even if you're actually good. At that point, your motivation will need to come from elsewhere.
  3. Distinguish between people you admire and jobs you want to do. Your favorite teacher, coach, boss, etc. may not be a proper representative of his field and you may find that everyone else in that field is a disappointment.
  4. Test yourself on your beliefs about your own strengths. If you think you could be a writer, try to write something every day and see how long you can keep up with it. If you like working with computers, do it for a whole day sometime--without playing video games.
  5. Consider the money, but don't let that be the only factor. Market forces can change over a period of time, especially in the amount of time it takes to get your degree/certification/credentials. Today's safe bet may be over by then. Your dream job that you consider too impractical may suddenly be booming by the time you're ready. Plus, you could always marry money. What would you want your job to be if that happened? That's probably a good indicator.
A final suggestion: beware bias in all career advice you're given. As Mr. Weasley said, “Never trust anything that can think for itself if you can't see where it keeps its brain." Bias can even come from a pretty good motivation--I like what I do for a living; you're smart and skilled in this field; therefore, you should do this too. This is not enough. No matter how much I seem to like what I do, it isn't enough to make you do it. And it certainly isn't enough to have you try to be who I am.

Tuesday, October 18, 2011

Is There Gonna Be a Test On This?


We in education have basically admitted defeat on Standardized Testing. It's something that (otherwise) liberal Democrats will pander to Republicans on. It's something that (otherwise) reasonable superintendents will lament but won't attack in their (otherwise) very moving and motivational opening-day speeches. Standardized Tests are used to determine teacher quality, school funding , and student futures--even in cases where they are poorly constructed (see also the PSSA) or serve to enrich powerful corporations (see also ETS and the College Board) with public money. Many educators don't like Standardized Testing, but concede that it's here to stay, and that we'll all just need to learn to deal with it.

Did you know that Pennsylvania produces a re-test booklet and a ream of paperwork for every student who scored Basic or Below Basic on the PSSA? That's fine. Did you know that they ship these books to each school and mandate that students have the opportunity to re-take the test? Still fine. Did you know that re-taking the test counts for nothing? Nothing. How a student does on the re-test means nothing for the students, the school, anything--the only educational impact is the loss of another three days of school spent taking the PSSA. Pennsylvania is in a continual budget crisis. Shocking.

Little freak-outs like that notwithstanding, as a Slacker I kinda loved Standardized Tests. Show me a test I can't study for, and I'll show you a level playing field. Observe:

Preparation for school tests (non-Slacker version):
  • read material
  • take notes
  • form study group
  • develop practice test
  • re-read material
  • study notes
  • find tutor
  • pull all-night'r studying
  • cram material on bus and before first period
  • fix wrong answers on corrected test for extra-credit
Preparation for school tests (Slacker version):
  • uhm...preparation?
Preparation for Standardized Tests (non-Slacker version):
  • get a good night's sleep
  • eat a good breakfast
  • bring a pencil (or one will be provided for you)
  • ditto a calculator (or, you know...)
  • don't sweat the questions you don't know
  • don't guess entirely at random
  • go home and wait for scores
  • take it again, and your score will probably go up
Preparation for Standardized Tests (Slacker version):
  • adaptations declined
Standardized Tests are changing and expanding, however and unfortunately some of their changes will lessen our Slacker advantage. "SAT" stands for Scholastic Aptitude Test, yet they're trying to make it more achievement-based with the introduction of Algebra II--known in the industry as "the stuff you can't just think your way through". Final exams are also becoming more standardized (see also Keystone Exams and Regents). Soon, the psychological profiling of your teachers that you've been doing to determine the content of your finals may not be worth the empty out-the-window-gazing you've put into it.

In music, we also have a form of Standardized Testing--called auditions. It's a process by which a person's entire estimated value to a program or ensemble is allegedly predicted by a single performance of some bit of music or set of tasks. Often the people hearing the audition have been doing so all day (or evening, or semester...) and are finding it harder to care. Even when one does manage to care the entire time, at some point you start to face thoughts like "Well, I like that tone, but is it good tone? Is it "8" tone, or more of a "'7". What did I give that kid three hours ago, the one with the strange glasses?" Students' self-worth, and sometimes entire future, can ride on these auditions, and the judges may be spending 65% of their concentration on trying to look interested but non-judgmental. There must be a way to exploit this fact as well, and when I figure it out you'll read about it here.

Since this is a Slacker's Guide, and not just a guide for slackers, here are a few tidbits on test-taking that even non-Slacker types can borrow:
  1. Read the question - Everyone thinks they do this, but if you grade a lot of stuff you find out that they don't. I recommend you spend as much time on reading the question as you usually do to answer. Teachers know that little one-word sentences like "Why?", "Explain.", "Agree?" Disagree?" and "Twice." can double the length of the answer with almost no effort on the part of the questioner.
  2. Mine the test - it's rather difficult to construct a test without providing some of the answers in another portion of the test. Some teachers won't even try, assuming you just won't figure it out.
  3. Answer every question - not true on the SAT, but in most testing situations there is no penalty for guessing. Multiple choice questions are obvious opportunities, and if you ever left a true-false question blank, please just stop reading right now. Still, take a crack at the essays too. In fact, if you can quote your teacher from something that may have been a pointless rant at the time, you may earn credit you don't deserve.
  4. Know your adversary - Teachers are biased. I don't mean that you got a D- because Mr. Rouche hates you--you got a D- because of the wrong answers. Besides, teachers tend to do grading with computers now, so exerting that kind of bias would be more work than its worth. What I mean is that teachers' tests are an expression of their values. You already know what's on the test if you think back to what the teacher seemed particularly animated about when they first presented it. Make an asterisk in your notes--okay, you don't have notes. Make an asterisk on your forehead whenever a teacher seems to really care about something. If they care about it, you should too..for a little while at least.
  5. Hoard - If you're lucky enough to have a teacher who hands back tests and quizzes and doesn't expect them back, hoard them. I know managing paperwork is not really our gift, but line the bottom of your locker with this stuff for now and sort it out later. Teachers re-use questions, and when they don't they tend to ask the same stuff in a different way. If you can look through an old quiz and know all the answers, you're ready for the test. When you can deal with all of the test questions, bring on the exam.
There should really be ten of those, I guess--like commandments. Check back, and I'll add to it as I think of things.