Two days ago I commented on a questionable item in the 10th grade TAKS mathematics test, for which scores were revised. The associated TEA press release refers also to a controversy over some science test items, and states that, upon review, these items were found to be correct. It is fascinating to see the items and to see how they are judged to be correct. The TEA (Texas Education Authority) put out an Additional Information Regarding Released Science Items for the spring 2003 testing cycle. Four controversial items are discussed.
Grade 5 Science, Item 13. Item 13 asked students which two planets are closest to Earth. Among the possible answers: Mercury and Venus, and Mars and Venus. The correct answer varies over time, and the question is plainly wrong or crazy. To add insult to injury: the intended answer was Mars and Venus, but on the day the test was given the correct answer was Mercury and Venus. Nevertheless, the TEA insists that for the purpose of the 5th grade test the question had only one correct answer - to wit, the wrong answer.
Grade 10 Science, Item 50. Item 50 looks crazy to me - they seem to be testing in a most convoluted way that the student knows that the element symbol K stands for Potassium. The TEA discussion indicates that the item is factually wrong to boot, but they insist that it is valid just the same.
Grade 11 Science, Items 11 and 45. Question 11 asks for the force exerted by a jumping frog on a leaf. The force has two components: one due to the weight of the frog and the other due to its acceleration. These are to be added vectorially, but the direction of the jump is not given. The TEA insists that therefore the correct treatment of the question must ignore the weight of the frog. Obviously the question is wrong and the TEA is wrong to insist that it is correct. Question 45 concerns a hypothetical situation in which a force is exerted on an object but no work is done. The question asks what can be concluded, and the intended answer is that the object is and remains at rest. This is wrong; the force may be perpendicular to the direction of motion. The TEA insists in effect that students don't know that, and that therefore the TEA's intended answer is, for the purpose of the test, the unambiguously correct answer.
The TEA has a bit of a quality control problem, obviously. In connection with the earlier 10th grade Math test problem Kimberly Swygert asked if the pre-testing might not have found the error. The same question could be asked for these science test items, but I think that it is too much to ask of the psychometric process that it correct for blunders of this kind.
I suspect that for many patently wrong questions students will nevertheless do what the TEA expects of them. The pernicious effect of the bad test items is indirect. It creates among the students and the public an impression (a correct impression) that the TEA doesn't have its house in order; that questions can't be read to mean what they mean; and that one should always be prepared to second-guess the clear meaning of a question.
A closing remark: the New York State Regents testing division has similar quality control problems. I remind the reader of the earlier discussion about the June 2003 Regents Math A exam, and my related Critique of the New York State Regents Mathematics A Exam
As reported in the Houston Chronicle the Texas Educational Authority yesterday announced a revision of the scores of the 10th grade TAKS exam that was given this spring, because of an error in one of the questions.
Readers with some knowledge of high school trigonometry may find it interesting to see the problem. The question is reproduced in the Houston Chronicle, or one can see the TEA original (look at question 8). The question shows a drawing of a regular octagon, indicating the inscribed radius as being 4.0cm and the circumscribed radius as being 4.6cm. The question is what is the perimeter of the octagon to the nearest cm. The choices are 41cm, 36cm, 27cm, and 18cm.
The data are contradictory: an octagon with inscribed radius 4.0cm has circumscribed radius about 4.33cm. Taking the 4.0cm and 4.6cm at face value a student might reason that the perimeter of the octagon is somewhere between 2*pi*4.0cm and 2*pi*4.6cm, and this leads to the answer 27cm in the multiple choice format. Or the student could apply trigonometry and obtain perimeter 26.5cm by starting from the given inscribed radius or 28.2cm by starting from the given circumscribed radius. A fourth approach is to use Pythagoras's theorem on a right triangle that has hypothenuse 4.6cm and one right side 4.0cm; then one finds that the circumference of the octagon must be 36.3cm. That (or rather, 36cm) was the intended answer.
According to the TEA press release, "item eight on the 10th grade math test could have been read in such a way that the question had more than one correct answer". That is putting a very kind spin on their blunder - there is in fact no reading of the question under which it has just one correct answer. It is amazing that the TEA would have this test composed and reviewed by people that fail to recognize that one cannot arbitrarily specify both the inscribed and the circumscribed radius of a regular polygon. According to the TEA press release: "Each test item goes through a rigorous review process that includes a field test of the items and two separate review sessions by professional educators who have subject-area and grade-level expertise and who are recommended by their district." The TEA didn't mean that as an explanation, but for me the "professional educators" part goes a long way just the same.
[Addendum, Aug 09. Please see the figure accompanying question 8 in the exam. The line segments that I described as inner and outer radii are not, in fact, identified as such in the figure or in the question. They meet at a point that certainly appears to be the center of the octagon, but that is not labelled either. There is, therefore, a reading of the question under which it has a single correct answer. Under that reading the given data are all correct, the special point is not meant to be the center of the octagon, and the figure is simply distorted in what happens to be a highly misleading way.]
Here is a sensible Op-Ed on California's decision to delay the high school exit exam requirement: Why the exit exam got held back instead of failing kids, by Daniel Weintraub (SacBee, Sun Jul 20 2003).
The California Board of Education's recent decision to delay the impact of the state's new high school exit exam was a disappointing but necessary tactical retreat that should ultimately advance the long-term goal of accountability in the public schools. If the ed board hadn't backed off, the legal dogs would have sued the state on behalf of thousands of students in the class of 2004 who would have been denied diplomas after failing the test. Their argument: These kids never got a fair opportunity to learn the material on which they were tested. Unfortunately, they are probably right.
The issue continues to play in New York State as well. The difficulty of the June 17 instance of the Regents Math A exam was misjudged and the exam had an unexpected high failure rate. Commissioner of Education Richard Mills might have responded by lowering the passing score, but instead he tossed the exam entirely. It must have given him great relief to get it off his back and not have anyone fail the Math portion this year, because he went a step further and extended the good news to juniors as well: any junior that sat the exam in June has it waived as a graduation requirement. The Board of Education has now named a panel to investigate the Regents Math A exam and respond to 9 specific questions. My take on the exam is in this critique of the Regents Math A.
In other news on high school exit exams, the Boston Herald has this:
STIRRING STORY: Tracey Newhart, an aspiring chef who has Down syndrome, failed her MCAS test, which could keep her from studying cooking at Johnson and Wales University.
The original is pay-per-view at the Herald. The story is also here in the Cape Cod times. How long ago was it that a grade school diploma, or at most an eighth grade education, would be the normal requirement for studying cooking?
Which brings me to this news item from Australia. Drop-outs can be successful, by Farrah Tomazin (The Age (AU), July 21, 2003).
A study tracking nearly 8000 students has found that many teenagers who do not finish year 12 earn more money and have higher job stability than those who stay but do not go on to university. Geoff Masters, chief executive of the Australian Council for Educational Research, which released the study, said the findings contradict the common belief that early school leavers tended to struggle after dropping out. 'It's often believed that students who leave school before the end of year 12 are at risk of not making a successful transition from school into the workforce,' Professor Masters said. 'But when you compare them with students who finish year 12 but don't go on to university, the early leavers are more likely to be working full-time, have a degree of job stability and be in a job that they like.'
It sounds plausible to me. The trend to demand a high school diploma for everything and look down on those that leave high school without a degree mainly reflects a steady lowering of standards for the eighth grade education.
New York State Education Commissioner Richard Mills is under much pressure because of the high failure rate on the recent (June 17) New York Regents Math A exam. The latest and earlier instances of the exam and the associated scoring keys and conversion tables are posted on the Regents Examinations Web site, under the link to Mathematics A. Procedural information related to the exam is posted at the State Assessment site under High School General Information. I know of two reviews of the exam on the Web. There is my own Critique of the New York State Regents Mathematics A Exam, and there is an Analysis of the June, 2003, Administration of Physics and Math A Regents done by the New York State Council of School Supervisors (NYSCOSS).
In connection with the Math A flack the director of the testing division at the NYS Education Department was reassigned and chose to resign, but this is not presented as a cure for any problem. The State Assembly and Senate will hold hearings, according to a report in the Rochester Democrat and Chronicle:
[State Assemblyman Steven Sanders, D-Manhattan], who chairs the Assembly education committee, and Sen. Stephen M. Saland, R-Poughkeepsie, who chairs the Senate education committee, plan to hold public hearings for people to express their feelings about high-stakes testing in light of an estimated 37 percent passing rate on the June 17 Math A Regents exam. Those hearing dates have not yet been set, but Sanders said Rochester, Albany and New York City might be host cities.
Of course there are plenty of calls on editorial pages for the State to abandon its plan to require students to pass five Regents exams for graduation starting in 2004. However, according to an article in the Albany Times Union, Voided math test said to reveal systemic ills, the Regents and Commissioner Mills remain supportive of that plan.
Asked whether he and other Regents still stood behind Mills, board member Saul Cohen responded, "Sure," but added, "That doesn't mean we can't press him to do certain things as we did. We pressed him to nullify the results of the Math A, and I'm continuing to press him to re-examine the physics exam. But that doesn't mean we're not backing him."
A somewhat different take on the exam trouble is found in an article by Karen Arenson in the New York Times, Math Failures Are Raising Concerns About Curriculum.
But some explanations [of the high failure rate] touch on deeper issues, including whether the Math A curriculum is too broad, how much harder it is for students to solve problems than to manipulate equations, and whether unqualified teachers are even less likely to succeed in preparing students than they were with the old math curriculum.
[...] The shift from rote learning to a greater emphasis on mastery of concepts is welcomed by some college professors in math and science, who have been trying to accomplish the same shift. They say that although mastery of some facts is critical, students who focus on memorization may do well in a course but remember little of it six months later.
[...] Some educators say teaching students to be problem solvers takes more skill on the part of teachers, a challenge when there is a shortage of qualified math teachers.
"Teachers are not really prepared to prepare kids for this test properly," said Alfred S. Posamentier, dean of the School of Education, City College of New York, and the author of books on problem solving. "There is very little training for teachers in problem solving; it's assumed they will get it along the way."
My take on it is different. The low passing rate is a complicated affair in any case, and it isn't entirely clear from the data to what extent the exam was really more difficult than earlier instances. Students can take the exam three times per year over multiple years, in August, January, and June, and the character of the test taking population may vary greatly between the months. Many of last year's seniors may still have graduated on the basis of the easier Mathematics I exam. It is surprising that Commissioner Mills does not have the data to say anything more authoritative about the relative difficulty of the latest exam.
The main thing that can be learned from the low passing rate is that, in many cases, New York State high schools are failing to make up for the failures of elementary and middle school education. An eighth grader in a high performing country, say Singapore, or in a U.S. state that has high level content standards, say California, would be well placed to pass this exam. I find ony two kinds of questions that would probably be unfamiliar to such a student. One are the counting questions that require students to know something about permutations and combinations. The other are the very basic trigonometry questions: students must know the ratios in a right triangle that correspond to the sine, cosine, and tangent.
It does appear to me that the June, 2003, instance of the exam had a somewhat more difficult and less "standard" flavor than earlier instances, and in the open response section the June, 2003, exam tilts a bit more towards a test of aptitude rather than a test of school learning, relative to the previous two instances of the exam. This is not to say, however, that all the questions in August 2002 and January 2003 were of a standard and predictable form, and it is not to say that the June 2003 exam is plainly a test of aptitude and the earlier ones plainly a test of school learning.
The unintended shift in the character of the exam should be seen as a failure of the Regents testing division. In addition there are many mathematical flaws in the questions and dubious points in the scoring keys, as described in more detail in the Critique of the New York State Regents Mathematics A Exam and the Analysis of the June, 2003, Administration of Physics and Math A Regents, both mentioned earlier.