NYC test scores; small and unreliable gains

Yesterday, the state finally released school test scores; for NYC schools they are posted here.   Individual student test scores will only be made available August 17 – through the ARIS system, for which you will need your child’s OSIS number. 
Although the city showed gains of a few percentage points, the results were nothing to write home about: only 43.9 percent of city students in grades 3-8 met the standards in reading and 57.3 percent in math.
Though the  Mayor predictably claimed the city's gains of 1.5 percentage points in reading and 3.3 points in math showed great progress, actually the results are very mixed.  Only 35% of 8th graders were proficient in reading.  Moreover, I believe the results overall are still highly unreliable.  Why? 
  • The high stakes attached to test scores  in city schools will tend to lead to gains, because of excessive test prep, narrowing of the curriculum, and even cheating, rather than real learning.  (For more on this see our blog.)

  • The same testing company is still writing them and the same “experts” are in charge at the NY State Education Department as in previous years, when there was tremendous test score inflation (the state intends to switch vendors next year.)

  • Even if the exams were perfectly constructed and scaled, the city's gains are so small as to be likely statistically meaningless.

How else do we know the tests are still flawed?  Only 3.5% of students statewide received 4’s (or advanced) on the ELA exam, and only 2.7% in NYC. This is clearly a test which cannot distinguish performance at the upper levels.
Howard Everson, consultant to SED, claimed otherwise to GothamSchools, saying that the gains under the new standards were small, they can be viewed as statistically significant because of the sheer number of students tested. He also said he trusted the state’s ability to track score trends even as the tests’ length, composition, and proficiency standards change.”    
Yet Everson told the New York Times in 2009 that the state tests were "about as good as we can build them," right before the test score inflation bubble burst and after it was clear to most objective observers quite the opposite. 
In addition, this year, some city schools, according to the NY Post, saw suspicious gains of up to 25 percentage points in both subjects.   Yet the  Mayor in his press conference said the precautions to prevent and check for cheating that were in place before he took office were too expensive to implement. He also said there was “no evidence of widespread cheating” (actually, in a Freudian slip, he said “no evidence of widespread teaching” !) 
Unfortunately, NYSED is no longer releasing the test questions, which will prevent anyone from discerning whether they were poorly or ambiguously worded as has occurred quite frequently in past exams; the state claims that this change “helps to ensure that preparation for the tests goes much deeper than simply reviewing past exams.”    But other states and reputable testing companies like the College Board still release the questions on their exams; why can’t NY State do the same?  The lack of transparency can only further diminish public confidence in the results.

There's a short clip of my views in last night's Fox-TV news story here.
You have read this article flawed / Howard Everson / NYC test scores / NYSED / reliable with the title NYC test scores; small and unreliable gains. You can bookmark this page URL http://thediariesofalawstudent.blogspot.com/2011/08/nyc-test-scores-small-and-unreliable.html. Thanks!

No comment for "NYC test scores; small and unreliable gains"

Post a Comment