What Is the Goal of the Algebra 1 EOC?

It’s that time of the year: all across Florida, the children are crying, stressed and overwhelmed. Why? It is Algebra 1 EOC time.

Since 2010, Florida’s middle and high school students have been administered Algebra 1 End of Course (EOC) exams. We have argued that the Algebra 1 EOC has the highest stakes of all: not only does it count as 30% of a student’s course grade, it is a high school graduation requirement, a requirement for admission to many magnet schools, calculated into school and district grades, etc, but, this year, schools will be fined 1/6 of a student’s per pupil funding (about $1,000) if the child does not pass their EOC. Those are high stakes!

How do students react to the stress? This is what parents have to say:

  • “A 7th grader (earning high school credit) took the Algebra 1 EOC/FSA and didn’t finish the test. She is stressed and terrified of receiving a failing grade (she has always been an outstanding student). She said there was material on the test her teacher never covered and she is concerned because it makes up 30% of her grade and could lower her future high school GPA.”
  • “My 8th grader is set to take her honors Algebra I EOCs tomorrow and Wednesday. She’s so stressed and scared she was bursting into tears all weekend.”
  • “My 8th grader is taking Alg 1 tomorrow and Wednesday. The kids are all afraid to fail the test — Crying, stressed and overwhelmed. The prep work prior to the test was brutal. Sent home with a 40 pg packet, then a 12 page packet trying to help prepare them. This on top of other FL standards tests and EOC. The teacher told them they were all going to fail.” (emphasis mine)

After students take the test, they tell us this:

  • “The questions on the EOC were confusing. I left the test feeling tired, stressed out, and not confident that I did well on it at all.”
  • “This is a test of “gotcha”-type questions. There are a mishmash of questions and they are often more of a function of how to use a computer than how to do math.”
  • “My son took his Algebra 1 EOC yesterday and is feeling the same way. He too has had an A all year and said that he knew nothing on the test. This is his first year in public school and he was so upset. He said that he felt like he was set up for failure and by his teacher. How sad that an 8th grader who works so hard at school now feels this way.”

So, we have a ridiculously high stakes test, students are stressed and overwhelmed, and they describe the questions as confusing and a test of computer skills more than math knowledge. Some children are blaming their teachers, feeling they have been set up for failure. Interestingly, the Florida Department of Education (FLDOE) has responded similarly. Commissioner Pam Stewart said (during the Keep Florida Learning Committee) that if anything on the test doesn’t match what’s being taught, it’s the fault of the teachers who just don’t understand what they’re being asked to do. Is that true? Is the test is fine and the teachers are the problem? What if the problem is not the teachers, but, rather, the test or the standards? During the Keep Florida Learning Committee meetings, Alachua County Superintendent, Dr. Owen Roberts, repeatedly questioned the validity of the instructional standards. While they are busy placing blame on the teachers, what is the FLDOE doing to assure the validity of the EOC, its associated standards and its many high stakes uses? Is the Algebra 1 EOC designed to confirm student proficiency or teacher failure?

2015 was the inaugural administration of the new Florida Standard’s based Algebra 1 EOC, created by American Institutes of Research (AIR).  As we have described previously (read here), the AIR tests were never field tested and they lack completed validity studies. During the summer of 2015, Alpine Testing Solutions’ “Independent” Validity Study did little more than confirm Algebra 1 test questions were aligned to the standards, with no mention of fairness, reliability or validity for at risk sub populations of students; hardly (in our opinion) an appropriate evaluation for a test with such high stakes.

The 2015 FSA Technical Report reveals some interesting finding regarding the Algebra 1 EOC.

On page 4 of Volume 2 of the 2015 FSA Tech report, the number of field tested items on each assessment. Students should know that 10 of the 68 are field test items and will not be counted for or against them.

13164224_1316906234991721_271044146925004399_n

 

This is also from the 2015 FSA technical report and shows the number of technology enhanced items that had to be replaced when writing the paper/pencil version of the test.

13165928_1316897624992582_1324943782525090398_n

Nearly a third of the Algebra 1 test questions are technology enhanced, requiring specific computer skills to answer them correctly. This is important because my district (Monroe) informed the school board that students performed worse on the technology enhanced items than the standard items. This brings up the question “is this test assessing math skills or computer skills?” Are students who do not have computers at home at a disadvantage on this graduation requirement? Neither the technical report nor the Validity Study addressed this. We believe this should be addressed.

There is no report comparing student performance on the FSA Algebra 1 EOC to any established, normed Algebra 1 assessment. This would be vital to assuring the FSA EOC tests what it purports to (an essential ingredient of validity).

This is the most alarming thing we discovered:  The Technical Study confirmed the EOC is ridiculously hard, yet it continues to be administered with the hope that students will, someday, acquire the skill needed to pass it. In the meantime, student self-confidence, emotional well being and GPAs (not to mention school finances) will just have to suffer.  Here is what we found:

The Algebra 1 (and 2) EOC was shown to be more difficult for the test population, with an atypical standard error curve.

From Volume 4 FSA Technical Report:

FSAV4Figure 3 SEM

 

 

Figure 3 show, for grades 3-7, the SEM curves show the expected “U” shape.

 

 

FSAV4Figure4SEMEOCs

 

 

Figure 4 shows the right side of the “U” has flattened out for Algebra 1 and Algebra 2 (and grade 8 above), indicating these exams are more “challenging” for the tested population.

 

From Page 18, Volume 4:

“For most tests, the standard error curves follow the typical expected trends with more test information regarding scores observed near the middle of the score scale. However, there are two general exceptions. In grade 8 Mathematics and for both Algebra EOC tests, the test is maximized at a higher point along the ability scale. This suggests the items comprising these tests are somewhat challenging relative to the tested population for this initial test administration. Because the testing of Algebra 1 is relatively new and the statewide testing of Algebra 2 is entirely new to these populations, this atypical curve is not unexpected. As students continue to learn these required skills, it is probable that this SEM curve will shift to reflect the expected, normally distributed SEM curve over time.”

What does this mean in English? We are not statisticians but we think this tells us that the 2015 FSA 8th grade Math and BOTH Algebra EOCs DID NOT fit the expected SEM curves, demonstrating the items on these tests are more challenging for the tested population. Is this the result of students not knowing the required skills or is this a problem with test design? No one knows because there was no comparison to an established, valid assessment. No one knows because there was no evaluation as to whether the test assesses math skills or computer skills.

The SEM results should surprise no one. The “curve” on the algebra test has always been huge, with students passing while answering less than 50% of the questions correctly. If you add technology enhanced “gotcha” questions to the previous EOC, you should expect even worse performance. The question is whether it is appropriate, or necessary, to test Algebra 1 students this way. If the DOE believes students will adjust to the new exams in time, should the high stakes be waived while allowing students to catch up on the required skills? Is there a deficit in the earlier (Common Core) math standards that contributes to the poor performance on these exams? Has anyone even looked? We hope these questions are being asked.

We believe, in a standards based system, tests could be written to assess essential Algebra knowledge without making students cry.

What is the goal here? Students can be challenged without being crushed. Exams can be rigorous without being punitive. A high school graduation requirement should not be full of “gotcha” questions. Accelerated middle school math students should feel proud of their accomplishments, not demoralized. No test scores are that important.

Is the goal of this EOC to assure student proficiency or to blame teachers for their inadequacies? Is this exam an educational tool or a political weapon?

Destroying children’s confidence should concern the Department of Education.  It is not enough to wait and see if students acquired new skills in the future. There are children taking these tests now. They are stressed and crying now. Waiting a year or two will only assure next year’s students are equally tortured.

We urge you to contact your Florida legislators and demand that these exams, and the high stakes attached to them, undergo review. Creating a test that demoralizes even the most academically talented students and then blaming their teachers for the outcome is accountabaloney.

Just prior to publishing, this was reported in the Tampa Bay Times: “Florida Department of Education leaders will be setting up an online forum this summer to collect ideas about how to revise the state’s educational accountability system.” They will have their work cut out for them because there is A LOT in need of revision. The problems with the Algebra 1 EOC are just the tip of a very large iceberg.

Algebra 1 EOC: Ridiculously High Stakes

5/22/2016, ACCOUNTABALONEY UPDATE:

At the 2016 Spring FOIL Conference (Florida Organization of Instructional Leaders), The FLDOE gave a presentation on the details contained in HB 7029:School Choice, one of the two long “train bills” from the 2016 session. To give you an idea of the bill’s complexity, the FLDOE summary presented was 135 pages long! It included this good news:

Section 27. Amends s. 1011.61, F.S…to “Delete the provisions requiring an FTE adjustment when a student does not pass an end-of-course exam required to earn a high school diploma. The FTE adjustment was scheduled to begin in the 2016-17 school year.”

In other words, the Algebra 1 Performance Based Funding, discussed in this blog, has been repealed. The other high stakes attached to the Algebra 1 EOC remain in place.


Occasionally, reformers will dismiss concerns regarding the high stakes attached to standardized testing in our schools by comparing these state mandated tests to the test you must pass to get your driver’s license. For example, in 2015, current Senate President, Andy Gardiner told an Orange County Legislative Delegation (read about it here):  “My driver’s test was high-stakes, because I didn’t want my parents driving me around anymore.” Yes, being driven around by your parents is terrible…

The stakes attached to the Algebra 1 EOC recently got quite a bit higher, so we decided this was a good time to compare the consequences of the current state mandated Algebra 1 End of Course exam (EOC), which we will argue has the highest stakes of all of Florida’s state mandated assessments, with having your parents drive you around when you are 16.

The current Algebra 1 EOC was created by AIR, the same company that created the Florida Standards Assessment or FSA and first administered in 2015.  Cut scores for the 2015 administration were finally set in January 2016, resulting in an overall FAILURE RATE of 44% (with some schools and districts having worse performance than others).

Passing the Algebra 1 EOC is a high school graduation requirement. Students who fail the 3 hour, computer based assessment must either retake and pass the exam or earn a concordant score on the PERT exam.

A student’s score on the Algebra 1 EOC, like all of Florida’s EOCs, is worth 30% of the students grade. Some Magnet Schools require the completion of Algebra 1 before a student can gain admission to their programs. Students failing Algebra 1 will be ineligible for admission to these highly sought after programs.

Student scores on the Algebra 1 EOC are used to calculate their teacher’s VAM score, which affects their teacher’s effectiveness rating and merit pay. Teachers receiving multiple ineffective ratings may lose their jobs.

Student scores on the Algebra 1 EOC are a major component of the Middle School A-F School Grade calculation, making up at least 11% of the possible points. Middle schools are rated on the percentage of eligible students who pass a high school EOC (most often Algebra 1, though occasionally students will be offered other courses with EOCs) or earn an industry certification.

FullSizeRender-17

The number of eligible students is determined by the number of students in the class AND all 8th graders who scored a 3 or higher on EITHER Reading or Math in their 7th grade state assessment. To be clear, the number of students deemed eligible to take a high school level Algebra 1 course in 8th grade is NOT determined by their performance on any Algebra readiness standards but on their previous year’s FSA Math and Reading scores.

FullSizeRender-15

Performance on all of the State mandated Math EOCs, including the Algebra 1 EOC, along with calculated learning gains, also represent 30% of the calculation of High School A-F grades.

FullSizeRender-13

Schools achieving high letter grades receive extra funding while school’s receiving failing grades can be threatened with take-over or closure.

Performance on the Algebra 1 EOC is also calculated into the School District’s overall grade calculation.

FullSizeRender-20

As if the stakes described above weren’t high enough, there is now a thing called “Algebra 1 Performance Based Funding”, described in this slide from the DOE:

FullSizeRender-16

So, yes, you read that right. Beginning in 2016-17, if a student takes an Algebra 1 class and then, at year’s end, fails the mandated EOC, that student’s school must return 1/6 (representing the cost of one out of 6 classes) of the FTE money (this is the money the school receives to educate each student) back to the state. The money (amounting to over $1000 per student who fails the EOC) must be returned after a student has been taught by a teacher, in a classroom, for an entire school year… Schools in low income areas, with lower passing rates, will be hit particularly hard.

Apparently, schools will not be penalized if the student “subsequently enrolls in a segmented remedial course delivered online.”  I called my daughter’s middle school and, at this point in time, they have been given no direction from the DOE as to what a “segmented remedial course” looks like, but they are scrambling to assure it will be available to their students by the end of next school year.

So, yes, the stakes attached to the Algebra 1 EOC (high school graduation, 30% of course grade, Magnet School eligibility, teacher’s evaluation and job security, school and district grades and school funding) are ridiculously high and, I think we can all agree, are significantly worse than being dropped off at a high school dance by your parents.

The high stakes attached to the Algebra 1 EOC (and the rest of Florida’s state mandated assessments) are real, they lack common sense and, unless they are addressed, Florida will never have a valid accountability system.

In the meantime, all comparisons to getting your driver’s license are baloney.

 

Addendum: All images are from a FLDOE presentation entitled “School Accountability and State Assessments” at the Fall 2105 FOIL conference and can be found, along with lots of other great information, in their archived materials.

 

SB1360: Baloney on Rye ADDENDUM

This is an addendum to our previous blog, “SB1360: Baloney on Rye is Still Full of Baloney” :

It has been brought to our attention that it is unclear whether the SAT score targets described in SB 1360 reflect scores from the current SAT or from the “newly designed” SAT, which will debut later this year (info here and here). The redesigned SAT will have a maximum score of 1600, compared to the current SAT maximum score of 2400. Since the exam is yet to be administered, the percentile ranking of scores on the new SAT can only be predicted. It is estimated that a score of 1200 (that required to be exempt from Florida’s U.S. History EOC) will be closer to the 75th percentile on the new SAT (not the 15% we stated in our blog).

Additional comparisons with SB1360’s target scores for the ACT, suggest that exemptions for Algebra 1, Geometry and Algebra 2 EOCs may be closer to the 50th to 75th percentiles, respectively. So, SB1360’s required scores may be more “rigorous” than we first thought, but will they be appropriate? It turns out neither the old nor the new the SAT assess math skills beyond basic geometry. Why are we allowing scores on an assessment that does not test beyond basic geometry to exempt students from their Algebra 2 EOC, which covers up to Trigonometry concepts? We hope the Senate Education committees will address this.

Since there are dramatic differences between the performance level associated with the same reported score, we feel SB1360 needs to define exactly which SAT exam (old or new) it is referring to. We also question why Florida would put into statute target scores from an exam that is yet to be administered (even if it does have the same name)? Are Florida students expected to field test the new SAT and then have those scores used for accountability purposes? Remember how well that worked out for the 2015 FSA?

Also, there are significant concerns regarding the math portions of the newly designed SAT, especially for low income and English language learner (more here). The new format of math questions will require higher level verbal and reasoning skills and is predicted to put English language learners and low income children at a significant disadvantage. Given the ever increasing population of low income, English language learner, and immigrant students in our public schools (Miami Dade is currently expecting ~8,000 new immigrant students this school year), why is Florida choosing an exam that would put those students, their schools and districts, at a distinct disadvantage. How is that a fair accountability assessment?

Our initial blog may have underestimated the “rigor” of SB1360’s target scores. If they represent scores from the new SAT, they may be more “rigorous” than we thought. Does this make us feel any better about this bill?

No.

Reviewing the new SAT only raises more questions about the fairness of an accountability system that uses these scores as metrics and in this manner. Students with high standardized test scores (even in subjects unrelated to the course they are taking) will be exempt from taking final exams/EOCs. Students with lower standardized test scores (many who will be immigrants, disadvantaged and/or english language learners), will not only be required to take the exams, but they will be worth 30% of their course grades and (for Algebra 1) passing will be required for graduation. “Smart kids” (often wealthier, white students) will no longer need to take the U.S. History or other state EOCs. They will be exempt from the Algebra 2 EOC based on scores that don’t test the course content; their course grades will reflect their classroom performance and will not suffer from poor performance on the EOC. It appears that students and schools with high test scores (like Seminole County, which has been lobbying hard for this bill, originally calling it the “Seminole Solution”) will require significantly less testing than their less advantaged counterparts.

This does not describe a fair, equitable, uniform education system. This describes the misuse of standardized test scores.

This will not “fix” anything.

This is Accountabaloney.