Is the Honesty Gap a Distraction?

On October 22, 2015, education reformers from around the nation converged on Denver, Colorado, for the Foundation for Excellence in Education’s 8th annual Education Summit. Strategy Session VII was focused on “The Proficiency Gap: Why It Matters for Your State.” This session was particularly timely for Florida because, less than a week later, its Board of Education members would be meeting to discuss determining the proficiency levels (or “cut scores”) for the new, 2015 Florida Standards Assessment (FSA). (You can watch Session VII here)

The moderator of the session was Dr. Christy Hovanetz, Senior Policy Fellow of Accountability, Foundation for Excellence in Education. She introduced why she felt it was the right time to have the conversation about proficiency levels (because, having spent so much money on new standards, new tests, new curriculum and professional development, “it would all be for naught” if we didn’t raise the standards). She claimed the current “proficiency gap” (the difference between proficiency levels on state tests and proficiency level on “the Nation’s Report Card” or NAEP assessment) was a result of states not being honest with their students. She suggested that, over the last decade, states had been lowering the proficiency bar, lowering the cut scores required to achieve proficiency on their state tests, in order to “achieve” the Adequate Yearly Progress (AYP) required by No Child Left Behind, which had required “all students be proficient by 2014”. The proficiency gap, she claimed, was really an “honesty gap”, a result of states, essentially, cheating the system, and it needed to be corrected.

Is the proficiency gap a result of manipulation of cut scores on states tests, as Dr. Hovanetz (and the rest of the Session VII panel) suggested? In Florida, at least, it appears not.

No Child Left Behind (NCLB) was signed into law in 2001. What was Florida’s “proficiency gap” in the early years of NCLB? If the panel’s suggestion that states deliberately lowered proficiency levels to achieve AYP is correct, than today’s proficiency gap should be greater than that of a decade ago. Historical data, however, shows this is just not true. The 4th grade “gaps” are essentially identical and while the gap for 8th grade reading is slightly larger (23 vs 18), the gap for 8th grade math is significantly smaller (20 vs 30).


 

2003 FCAT % level 3+ 2003 NAEP % above proficient 2003 proficiency gap 2013 FCAT% level 3+ 2013 NAEP % above proficient 2013 proficiency gap
Reading 4th 55 32 23 60 39 21
Reading 8th 45 27 18 56 33 23
Math 4th 51 31 20 61 41 20
Math 8th 53 23 30 51 31 20

2003 FCAT scores from http://www.fldoe.org/accountability/assessments/k-12-student-assessment/history-of-fls-statewide-assessment/fcat/scores-reports/index.stml

2003 NAEP results from http://nces.ed.gov/nationsreportcard/states/

2013 proficiency gaps from whyproficiencymatters.com/florida


If cheating or dishonesty doesn’t explain the gap, what might?

  • NCLB required all states measure all students for grade level proficiency in Math and Reading. State tests were designed to determine if students were performing at grade level. The NAEP assessment, on the other hand, considers “proficiency” to be “solid academic performance for each grade assessed”. The gap might exist solely because NAEP never defined proficiency as “grade level proficiency” but rather something much more advanced.
  • Florida’s state assessment is a high stakes test where, sadly, entire school years focus on achieving high scores, to avoid retention and remediation, graduate high school, earn teacher bonuses and school grades. The NAEP test is a low (no?) stakes test. It is conceivable that at least part of the “honesty gap” is really an “effort gap” in that our test-weary children will surely “try harder” to score well on a high stakes exam. This alone could explain why more children score proficient on state assessments.

What the “honesty gap” really is a public relations campaign to encourage the raising of the bar to aspirationally high achievement levels. It is not an emergency or a sign of cheating. It is, likely, mostly caused by the fact that during the NCLB era, the goal was grade level performance, which, by definition, means average performance and NAEP always measured proficiency at a higher level. Now, the reformer’s stated goal is to eliminate need for college remediation and make all high school graduates “college and career ready”, a much higher achievement level, and reformers hope that NAEP proficiency levels will be an appropriate measure of that (to date, no one has any proof that current standards will make students more “College or Career Ready” but reformers hope they will).

The truth is, since 2001, NCLB required schools to focus on grade level proficiency and states used these test scores to inform parents as to whether their children were performing at or below grade level. Now, with the focus on “Career and College Readiness,” the state has to decide how to impart that information to parents.  Suggesting that states somehow created the “proficiency gap” by manipulating test data and “lowering expectations”, at least in Florida, appears to be inaccurate. Frankly, if Florida did manipulate cut scores to demonstrate false progress, it would have been done with the full knowledge and consent of the Foundation for Excellence in Education, which heavily influences all education policy in Florida.

So, why are reformers, like Dr. Hovanetz, so focused on this “honesty gap”?  We believe reformers, by focusing on the contrived “honesty gap”, have created a false narrative that they are using to further their reform agenda, encouraging the raising of the bar on state assessments regardless of the potential impacts on students and schools. The honesty gap rhetoric creates an emergency situation requiring immediate action. In Florida, it appears the Board of Education will spend significantly more time discussing cut score placement for the FSA than they did evaluating whether the assessment was even valid, fair or reliable.  Nationally, the honesty gap narrative distracts from the obvious failures of the education reform movement: scores are declining on the ACT and SAT, respected college entrance exams, as well as scores on NAEP, itself. After more than a decade of test-focused education reforms, students are graduating from high school “lacking intellectual curiosity”. Dr. Tommy Bice, Alabama State Superintendent, summed it up during the same proficiency panel discussion: “We have been preparing kids to take a meaningless test rather than to think, solve problems, think analytically or collaborate with peers.”

If reformers truly want to improve public education in America, we suggest they spend more time honestly evaluating the apparent shortcomings of their reform agenda and less time accusing others of dishonesty. We will be further addressing “Why Proficiency Matters” (or doesn’t) in upcoming blogs.  Stay tuned…

Algebra 2 EOC Results: Has Florida Jumped The Shark?

fonz_shark

In 1977, in an episode of the sitcom “Happy Days”, Fonzie displayed his bravery by jumping over a confined shark while on waterskis. Ever since then, the phrase “jumping the shark” has come to mean the moment in a TV series that defines the beginning of the show’s decline; a desperate attempt to retain viewership where it becomes obvious to the audience that the show has strayed irretrievable from its original formula. We believe the administration of the 2015 Algebra 2 FSA EOC (Florida State Assessment End of Course exam) will be remembered as the moment when Florida’s Education Accountability System “jumped the shark.”

Over the last week or so, the preliminary FSA EOC student results have been released to parents. Because the cut score process is still in progress, the reports show a percentile ranking for the child and the score breakdown of questions answered correctly in each “reporting category”. The full “horror” of the Algebra 2 EOC is being revealed.

Red flags were raised regarding the Algebra 2 EOC even before it was administered. Prior to the Algebra 2 EOC, every previous state-created EOC had been treated as a baseline administration during its inaugural year. Students would take the new EOCs in their inaugural year but performance on those test would not affect their course grade. The Algebra 2 EOC, however, from day one was planned to be worth 30% of the student’s course grade. Multiple emailed questions regarding this break from protocol went unanswered by the Florida Department of Education last spring.

Students recognized significant problems with the Algebra 2 EOC while it was being administered last spring. Here is a first person account from a 15 year old, gifted, straight A, honor student from Monroe County.

“I took the Algebra 2 EOC last year and it was absolutely ridiculous. I got my scores back and it said that I did better than 83% of the students that took it, but I’m not sure how they could even score such a horrible test. I walked out of the testing room and knew for sure that I failed it, and that they were not going to count it because of how flawed it was. The format that the test was on had many errors, and was filled with information that the teachers were not asked to teach. For example, the test would say “graph this piece wise function” and it would give you the wrong tools to graph it with, so even if you were totally capable of completing the question successfully, you couldn’t.”

We are certain that almost every Algebra 2 student will tell a similar story. The test was a disaster, bringing usually confident high school students to tears.

Before we discuss the results, understand this: Florida (along with the rest of “Common Core” America) has a “standards based education”. Teachers are asked to teach the standards and if students can demonstrate, on their state assessment, that they have learned enough of those standards, they will be deemed “proficient” and “pass” the exam. Florida State Assessments (like the previous versions, the FCAT) are “Criterion Referenced Tests” or CRTs. A CRT should compare a student’s performance to established expectations; if every student meets those expectations, all students would be deemed proficient. Likewise, if no students met the established criterion, all would fail. On CRTs, it is not only possible, but desirable, for every student to pass the test or earn a perfect score. (Remember that, it is important.)

A common example of a CRT is the written test for your driver’s license: if you answer 8 out of 10 questions correctly, you pass the test… even if thousands before you answered 9 or more correctly. (Learn more about Criterion Referenced Tests and their usage here)

The first problem parents should notice when reading their child’s Algebra 2 EOC report is that the score is reported as a percentile ranking, which is an inappropriate way to present results from a criterion based exam. Remember, if all students achieved the established expectations, ALL students would pass. In a criterion based exam, percentile ranking should be irrelevant.

Why would the Department of Education release the percentile ranking of students for a Criterion Referenced Test? Probably, in part, because they don’t expect parents to understand the difference and they are under time pressure to release some information regarding student performance, given that the exam was taken more than 6 months ago. The main reason, though, is that the established expectations for the new FSA have not yet been established and won’t be established until the January 4th State Board of Education meeting (more on that later).

So, despite having NO established performance standards, the Algebra 2 results were released to parents, highlighting a student’s percentile ranking in the state and their performance in individual “Reporting Categories.” Here is what we have learned from reports parents have shared with us:

ALGEBRA 2 2015 FSA EOC results

#correct (out of 56) Total %correct Student percentile ranking
16 25% 63 %tile
18 27% 67%tile
20 36% 71%tile
29 52% 88%tile
43 77% 99%tile

 

Keep in mind that those who took Algebra 2 are essentially all college bound, often honors and/or gifted math students, and this exam is, by statute, worth 30% of their course grade (the 30% was waived after test administration last spring but, by report, will be back in effect next spring). The State’s most gifted math students (99th percentile) still missed one out of 4 questions and students will have passed (if the Commissioner of Education’s current cut score recommendation is approved) missing ¾ of the questions. Something definitely seems amiss.

Remember: “On criterion-referenced tests, it is not only possible, but desirable, for every student to pass the test or earn a perfect score”. So why has Florida created a CRT where its most gifted math student are unable to answer 25% of the test questions correctly?

Of course, the cut scores/expected performance levels have not yet been set for the Algebra 2 exam, so these student reports, actually, say nothing as to whether a student passed or not.

On September 15-17, 2015, at a Rule Development Workshop, the general public got a first look at the impact of potential cut scores. It wasn’t pretty. You can see the complete presentation here.

FullSizeRender-10

To summarize, the Educator Panel recommended cut scores than would fail 69% of students, including 85% of African American students. Since then, the Commissioner of Education has made her recommendations which would (only) fail 64% of students. The ultimate cut scores will be determined by the Florida Board of Education (BOE) on January 4, 2016. Several members of the board have indicated they wanted cut scores that were even higher than the Commissioner’s, failing even more students. Sadly, the process of establishing cut scores seems to reflect political ideology more than sound education judgement. (Learn about the cut score process here)

Prior to the release of the Algebra 2 test scores, on October 28th, I spoke in front of the BOE, and described why I felt “the most egregious example of Accountabaloney is the state’s unwavering assertion that the Algebra 2 EOC is valid.” You can watch my speech here. In summary:

  • In 2015, Algebra 2 had a new set of standards, which now included Pre-calculus and Statistics (learn about the standards here).
  • The commissioner has admitted that no one knows whether it is even possible to teach these standards in a single school year (details here).
  • The Algebra 2 EOC could not have been field tested because Utah does not test Algebra 2 or Trigonometry standards.
  • The Alpine Validity Study completely ignored Algebra 2 in the interest of time (Report here).
  • Honor students reported being unable to answer any questions on day two of this EOC, so it should have been no surprise that the exam’s results were total outliers with almost half of students failing including 85% of African American students. (Report of results here)
  • At the Keep Florida Learning Committee, the department suggested the problem was “the teachers didn’t teach the standards”. (more info here)
  • Crushing the confidence and destroying the GPA of our best and brightest students does NOT make them college and career ready. Blaming it on their teachers does NOT help recruit and retain high quality STEM educators.

I concluded with “the Algebra 2 exam and its standards deserve a complete review.” Seriously, parents shouldn’t have to ask. In the face of falling national math scores (read here), the entire math sequence deserves a review.

Currently, parents and teachers report concern regarding both the pace and scope of the new Algebra 1 and Algebra 2 standards. When Florida switched to the Common Core-like Florida Standards in 2014, much of what used to be taught in Algebra 1 was moved to Pre-Algebra. Algebra 1 now includes statistics and at least a third of the previous Algebra 2 content. Algebra 2 now includes pre-calculus and statistics. The Florida Standards are now out of alignment with the new CCSS-based textbooks. If a teacher teaches from the new, state approved Algebra 1 or 2 textbooks, they will not cover the required standards. This is a problem. Why would the State approve textbooks that do not cover the required course content?

In addition, the amount of content seems to be too much to complete in a single year, especially since the EOCs are given in late April/early May. One Algebra 2 teacher said:

“The problem is that there are TOO MANY standards in the test item specs. It’s simply TOO MUCH to teach, prior to the Algebra 2 EOC which starts in early April to the first week in May. We have to teach at the speed of light to get everything in, and it’s nearly impossible to do so.”

Parents are concerned that the pace does not allow their kids to fully understand concepts and many kids are dropping out of subsequent advanced math courses. Another math teacher wrote:

“Algebra 1 now looks like Algebra 2 and Algebra 2 now looks like Precalculus/Statistics. Regular kids who are not strong in math are drowning. It’s just so developmentally inappropriate. These new standards and new EOC’s are geared towards honors students who excel in math.”

Clearly, when Florida created its new “rigorous” Math standards, moving a significant amount of Algebra 2 content into the Algebra1 course, it created problems beyond the EOCs. By statute, passing the Algebra 1 EOC is a graduation requirement so Florida now requires a mastery of the higher level Algebra 2 content to graduate high school. Was this the legislature’s intention? Also, in the current A-F grading system, middle schools are being graded based on their number of advanced math students (Algebra 1 and Geometry) encouraging increasing student placement into Algebra 1 (Learn about how School Grades are calculated here). Now these very young math students are being asked to master Algebra 2 standards; is this developmentally appropriate? Also, the middle school math teaching credential is for grades 5-9 and DOES NOT include Algebra 2 content. Now that Algebra 2 standards are placed in Algebra 1 courses, most middle school math teachers are not credentialed to teach the content. This is a serious problem.

The current pace, scope and assessment of Math standards has been significantly disrupted by #accountabaloney. We believe re-evaluation of the entire advanced math sequence and math FSA EOCs is warranted.

Here are the concerns that must be addressed:

  1. The overwhelming evidence suggests there is CLEARLY something wrong with this Algebra 2 exam, yet the DOE is releasing test scores, cut score determinations are underway and the test remains on the Spring 2016 schedule. This defies reason. Is the DOE addressing this? Why would they release the scores from such a disastrous test administration? Why even require a state mandated EOC for a non-required class?
  1. Has either the Algebra 1 or 2 EOC been definitively shown to be fair, reliable and valid for special populations including special education (ESE), English Language Learners (ELL), low income or racial subgroups? Where are those reports? 85% of African Americans failed the Algebra 2 EOC… 85% of the best and brightest African American students failed this test… Something is wrong. Also, since passing the Algebra 1 EOC is a graduation requirement, the Algebra 1 EOC definitely needs to be evaluated for fairness for ESE, ELL, etc, students. Where is the evidence this has been done on these new EOCs? (Please don’t accept the answer “the Alpine Validity Study”; it did NOT address these issues.)
  1. Is there any evidence that an accelerated pace through advanced math concepts is a good idea? Why have Algebra 2 standards in an Algebra class or pre-calculus standards on Algebra 2?  Should Algebra 2 mastery be required for high school graduation. Are middle school teachers properly credentialed to teach the current Florida Algebra 1 standards?
  1. We have significant concerns regarding the entire accountability system, extending beyond the math EOCs, at this point. Clearly the “establishing cut score” process has as much to do with political ideology as it does educationally sound practices. We worry that, with the new tests and the new more “rigorous” cut scores, the initial intent of many of the State’s legislated mandates are no longer aligned to the current assessments. For example, mandatory third grade retention is meant to identify and provide interventions for students reading below grade level; is there any evidence that the current cut scores assess grade level proficiency? Will 3rd grade students reading at or above grade level be marked for retention? Likewise, passing Algebra 1 was a challenging requirement for high school graduation but now it appears the EOC is testing a much higher, more challenging Algebra 2 curriculum. Is that what the original legislation intended? We suspect not.

CALL TO ACTION:

We encourage parents and teachers to share this blog with Governor Rick Scott, Commissioner Pam Stewart, the Florida Board of Education and your legislators. Tell them the Algebra 2 EOC is the most egregious example of the problems with the current accountability system. Ask them how parents are expected to have confidence in a system that creates final exams that are so abusive in nature and then seemingly ignores the obvious issues. If the DOE is not evaluating the math standards and assessments, demand they do so. If they are investigating these issues, remind them it would be wise to inform teachers and parents before we lose all confidence and respect.  We need a DOE that can recognize flaws in the current system and work to fix them. Florida needs to re-evaluate both the advanced math sequence and math FSA EOCs.

With the creation and administration of the Algebra 2 EOC, we believe Florida’s Education Accountability has “jumped the shark.” Without dramatic policy reversals, we fear parent confidence in the system will be irretrievable.

You cannot measure what makes America great. To say otherwise is Accountabaloney.

Kid with jet pack. Child playing at home. Success, leader and winner concept

What if the accountability system is focusing on the wrong questions? What if grade level proficiency is NOT the secret to America’s (or Florida’s) success?  On second thought, why would anyone think that economic success would be the result of grade level proficiency?  What actually makes someone “Career and College Ready”?

To date there has been NO evidence that current standards and testing will increase career and college readiness, let alone get someone “ready for the Global Workforce”.  The Reformers have been unable to provide a clear definition as to what describes someone who is career and college ready, beyond a basic (somewhat low) score on the SAT, ACT or PERT.  If you choose to use those parameters, after a decade of test-focused accountability reforms, it appears our “Career and College Readiness” must be  declining along with our with SAT and ACT scores.

What we do know is that some students with high college readiness scores fail to complete college and/or obtain gainful employment afterwards.  Also, some students, who were never successful in K-12 education go on to create business empires (Dave Thomas of Wendy’s, for example).  The definition of success is as wide as an individual’s career opportunities.  Is it possible to measure likelihood of future success with a few math and reading (or even AP) scores? Should we bother?

Honestly, to define career and college readiness is meaningless when you look at the real world.  Let’s look at Gates himself.  He has poured billions into the current education reform known as Common Core knowing that he himself was able to start up his own business, and be successful at it, without a college degree.  He started a line of business that did not even exist.  How many non-college degree  kids are just like him?  Many.  They might never reach the level of financial success that he has, but few successful individuals ever do.  Some of the greatest businesses or greatest personal success stories come from those who did not perform well on standardized tests.  Some of these notables are listed here.  Why so much focus on these tests today?

In 1965, President Johnson established the ESEA Act in order to combat the war on poverty.  The goal was to  set up a system of federal grants that would help infuse more federal dollars to low income schools throughout the nation.  Since then, we have seen the rise of accountability measures.  Why?  Because people wanted guarantees that tax dollars being spent on education were in fact being used properly.   Since that time, we have seen some rise in funding in education, yet our international scores (where we never excelled) have remained flat.  Ignoring the fact that the U.S. economy flourished, and public schools educated individuals who went on to spawn the Silicon Valley, medical and biotechnological advances and the Space Program,  the lack of improvement in test scores exacerbated a false narrative that public schools were failing U.S. children.

So while the U.S. economy was thriving, education “reformers” remained focused on the low test scores of our students and began adding more accountability measures that eventually micro managed our teachers, schools and districts.  What has our return on investment been since that time?  We have killed the love of learning and have turned our schools into factories that strive to check off certain benchmarks.  So much focus was placed on the test scores, that education best practices have been ignored and actually learning appears to be a secondary goal.

I think everyone can agree that there should be some sort of assessment to gauge how students are progressing.  By measuring all children, public school systems can minimize the number of children who “fall through the cracks.” If individual student data were used to inform instruction, to provide supports where needed, to demonstrate a program’s strengths and weaknesses, and to identify children in need of differentiated instruction, than testing could be quite helpful. Unfortunately, there is no measure that can, or should, show a small child whether they are on the tract to a successful life, whether that includes college and/or a well paying career. And even if there was, once high stakes are attached to those test scores, than the usefulness of the data is diminished. The game changes once there are stakes attached.  The focus no longer becomes about using the scores as a measure, but using them to reward or punish those who have or have not achieved the targeted goals. Students are unique individuals who will grow, mature and learn at their own rates. They cannot all be expected to reach targeted goals at the same time. To attach high stakes to goals attained along a prescribed timeline is absurd, because few children will be appropriately assessed along that one contrived timeline.

Vermont_Quote

To read the entire Vermont letter, click here.

There are just too many other factors that influence success; factors that are not easily defined.  Creativity, ingenuity, entrepreneurship are big ones.  This is what truly makes our nation great.  It is also not something that you can easily teach, you have to inspire.  This is the reason why so many strive to come to the U.S.  It’s because these concepts are synonymous with the U.S. and the concept of the American Dream.  No matter who your parents are, where you live, what difficulties you had in the past, in this country you can change all that and come out ahead if you have the drive, passion and willingness to push forward.  Education is key to any success, there is no denying that.

The role of education in my opinion, is to provide strong foundational knowledge and guidance in order to give kids kids room to grow and to take off from there.  Schools should get out of the way of kids  when they are pushing forward and should provide all kinds of supports for those that can and are willing to learn more and do more.

Over a year ago, I heard a high school student speak in front of his school board.  His name was Ethan Young and he was spot on with his analysis of what is going on.

The task of teaching is not quantifiable.  If everything I learning in high school is a measurable objective, I have not learned anything.  Creativity, appreciation, inquisitiveness, these are impossible to scale, but they are the purpose of education… why our teachers teach and why I choose to learn. Today we find ourselves in a nation that produces workers.  Everything is career and college preparation.  Somewhere our founding fathers are turning in their graves, pleading, screaming and trying to say to us that we teach to free minds.  We teach to inspire. We teach for the career to come naturally.   November 6, 2013, Ethan Young. Click to listen to his full speech.

The gifted child should be as challenged as the struggling child.   In the era of No Child Left Behind, that no longer is the case.  Too often we are seeing gifted children being given more “projects,” supposedly going deeper into the very narrow standards and even narrower curriculum when they should in fact  be accelerated.  At the end of the day, if you have a gifted child, they will take the same standardized test that a general ed or  struggling learner will take.  Is that a fair gauge of career and college readiness for such a diverse group?

In the past, what distinguished our public education system from the rest of the world’s was that ALL children were educated and no one was placed on a path that was not of their choosing. Any child could reach for the stars and achieve greatness. The current system puts up road blocks rather than creating opportunities.  Our current education system is based on putting in all kinds of check points on student progress that, instead of being a measure to help find strengths or weaknesses, are seen as road blocks where if a student doesn’t do well, they might get stuck.  If a student does happen to do well, there is rarely a good system in place to help those kids accelerate and move further faster.  Gifted students remain on the same path as their struggling counterparts.

If the goal is to be career and college ready, then the role and significance of these high stakes tests must change.  We need to look at the right goals and the right problems so that we can come up with better, more appropriate solutions.  Our teachers want their classroom time back.  Returning the classroom time, wasted on testing and test prep, back to the teachers would be a good starting point.  Eliminating the time spent focused on testing could free up teachers to once again differentiate or enrich instruction for more children.

So, instead of focusing on standardized test scores as a predictor of whether a child is career and college ready, let’s talk about what can be done to bring back an environment that encourages problem solving, rewards ingenuity and inspires creative thinking. Work to make schools a place where kids can once again walk through their classroom doors on a daily basis and feel empowered to take on the world.  Is that too much to ask?