What Is the Goal of the Algebra 1 EOC?

It’s that time of the year: all across Florida, the children are crying, stressed and overwhelmed. Why? It is Algebra 1 EOC time.

Since 2010, Florida’s middle and high school students have been administered Algebra 1 End of Course (EOC) exams. We have argued that the Algebra 1 EOC has the highest stakes of all: not only does it count as 30% of a student’s course grade, it is a high school graduation requirement, a requirement for admission to many magnet schools, calculated into school and district grades, etc, but, this year, schools will be fined 1/6 of a student’s per pupil funding (about $1,000) if the child does not pass their EOC. Those are high stakes!

How do students react to the stress? This is what parents have to say:

  • “A 7th grader (earning high school credit) took the Algebra 1 EOC/FSA and didn’t finish the test. She is stressed and terrified of receiving a failing grade (she has always been an outstanding student). She said there was material on the test her teacher never covered and she is concerned because it makes up 30% of her grade and could lower her future high school GPA.”
  • “My 8th grader is set to take her honors Algebra I EOCs tomorrow and Wednesday. She’s so stressed and scared she was bursting into tears all weekend.”
  • “My 8th grader is taking Alg 1 tomorrow and Wednesday. The kids are all afraid to fail the test — Crying, stressed and overwhelmed. The prep work prior to the test was brutal. Sent home with a 40 pg packet, then a 12 page packet trying to help prepare them. This on top of other FL standards tests and EOC. The teacher told them they were all going to fail.” (emphasis mine)

After students take the test, they tell us this:

  • “The questions on the EOC were confusing. I left the test feeling tired, stressed out, and not confident that I did well on it at all.”
  • “This is a test of “gotcha”-type questions. There are a mishmash of questions and they are often more of a function of how to use a computer than how to do math.”
  • “My son took his Algebra 1 EOC yesterday and is feeling the same way. He too has had an A all year and said that he knew nothing on the test. This is his first year in public school and he was so upset. He said that he felt like he was set up for failure and by his teacher. How sad that an 8th grader who works so hard at school now feels this way.”

So, we have a ridiculously high stakes test, students are stressed and overwhelmed, and they describe the questions as confusing and a test of computer skills more than math knowledge. Some children are blaming their teachers, feeling they have been set up for failure. Interestingly, the Florida Department of Education (FLDOE) has responded similarly. Commissioner Pam Stewart said (during the Keep Florida Learning Committee) that if anything on the test doesn’t match what’s being taught, it’s the fault of the teachers who just don’t understand what they’re being asked to do. Is that true? Is the test is fine and the teachers are the problem? What if the problem is not the teachers, but, rather, the test or the standards? During the Keep Florida Learning Committee meetings, Alachua County Superintendent, Dr. Owen Roberts, repeatedly questioned the validity of the instructional standards. While they are busy placing blame on the teachers, what is the FLDOE doing to assure the validity of the EOC, its associated standards and its many high stakes uses? Is the Algebra 1 EOC designed to confirm student proficiency or teacher failure?

2015 was the inaugural administration of the new Florida Standard’s based Algebra 1 EOC, created by American Institutes of Research (AIR).  As we have described previously (read here), the AIR tests were never field tested and they lack completed validity studies. During the summer of 2015, Alpine Testing Solutions’ “Independent” Validity Study did little more than confirm Algebra 1 test questions were aligned to the standards, with no mention of fairness, reliability or validity for at risk sub populations of students; hardly (in our opinion) an appropriate evaluation for a test with such high stakes.

The 2015 FSA Technical Report reveals some interesting finding regarding the Algebra 1 EOC.

On page 4 of Volume 2 of the 2015 FSA Tech report, the number of field tested items on each assessment. Students should know that 10 of the 68 are field test items and will not be counted for or against them.

13164224_1316906234991721_271044146925004399_n

 

This is also from the 2015 FSA technical report and shows the number of technology enhanced items that had to be replaced when writing the paper/pencil version of the test.

13165928_1316897624992582_1324943782525090398_n

Nearly a third of the Algebra 1 test questions are technology enhanced, requiring specific computer skills to answer them correctly. This is important because my district (Monroe) informed the school board that students performed worse on the technology enhanced items than the standard items. This brings up the question “is this test assessing math skills or computer skills?” Are students who do not have computers at home at a disadvantage on this graduation requirement? Neither the technical report nor the Validity Study addressed this. We believe this should be addressed.

There is no report comparing student performance on the FSA Algebra 1 EOC to any established, normed Algebra 1 assessment. This would be vital to assuring the FSA EOC tests what it purports to (an essential ingredient of validity).

This is the most alarming thing we discovered:  The Technical Study confirmed the EOC is ridiculously hard, yet it continues to be administered with the hope that students will, someday, acquire the skill needed to pass it. In the meantime, student self-confidence, emotional well being and GPAs (not to mention school finances) will just have to suffer.  Here is what we found:

The Algebra 1 (and 2) EOC was shown to be more difficult for the test population, with an atypical standard error curve.

From Volume 4 FSA Technical Report:

FSAV4Figure 3 SEM

 

 

Figure 3 show, for grades 3-7, the SEM curves show the expected “U” shape.

 

 

FSAV4Figure4SEMEOCs

 

 

Figure 4 shows the right side of the “U” has flattened out for Algebra 1 and Algebra 2 (and grade 8 above), indicating these exams are more “challenging” for the tested population.

 

From Page 18, Volume 4:

“For most tests, the standard error curves follow the typical expected trends with more test information regarding scores observed near the middle of the score scale. However, there are two general exceptions. In grade 8 Mathematics and for both Algebra EOC tests, the test is maximized at a higher point along the ability scale. This suggests the items comprising these tests are somewhat challenging relative to the tested population for this initial test administration. Because the testing of Algebra 1 is relatively new and the statewide testing of Algebra 2 is entirely new to these populations, this atypical curve is not unexpected. As students continue to learn these required skills, it is probable that this SEM curve will shift to reflect the expected, normally distributed SEM curve over time.”

What does this mean in English? We are not statisticians but we think this tells us that the 2015 FSA 8th grade Math and BOTH Algebra EOCs DID NOT fit the expected SEM curves, demonstrating the items on these tests are more challenging for the tested population. Is this the result of students not knowing the required skills or is this a problem with test design? No one knows because there was no comparison to an established, valid assessment. No one knows because there was no evaluation as to whether the test assesses math skills or computer skills.

The SEM results should surprise no one. The “curve” on the algebra test has always been huge, with students passing while answering less than 50% of the questions correctly. If you add technology enhanced “gotcha” questions to the previous EOC, you should expect even worse performance. The question is whether it is appropriate, or necessary, to test Algebra 1 students this way. If the DOE believes students will adjust to the new exams in time, should the high stakes be waived while allowing students to catch up on the required skills? Is there a deficit in the earlier (Common Core) math standards that contributes to the poor performance on these exams? Has anyone even looked? We hope these questions are being asked.

We believe, in a standards based system, tests could be written to assess essential Algebra knowledge without making students cry.

What is the goal here? Students can be challenged without being crushed. Exams can be rigorous without being punitive. A high school graduation requirement should not be full of “gotcha” questions. Accelerated middle school math students should feel proud of their accomplishments, not demoralized. No test scores are that important.

Is the goal of this EOC to assure student proficiency or to blame teachers for their inadequacies? Is this exam an educational tool or a political weapon?

Destroying children’s confidence should concern the Department of Education.  It is not enough to wait and see if students acquired new skills in the future. There are children taking these tests now. They are stressed and crying now. Waiting a year or two will only assure next year’s students are equally tortured.

We urge you to contact your Florida legislators and demand that these exams, and the high stakes attached to them, undergo review. Creating a test that demoralizes even the most academically talented students and then blaming their teachers for the outcome is accountabaloney.

Just prior to publishing, this was reported in the Tampa Bay Times: “Florida Department of Education leaders will be setting up an online forum this summer to collect ideas about how to revise the state’s educational accountability system.” They will have their work cut out for them because there is A LOT in need of revision. The problems with the Algebra 1 EOC are just the tip of a very large iceberg.

2016 FSA Administration: Ready or Not, Here it Comes

FSA Picture_thumb

On February 25, 2016, my son’s 3rd grade Florida State Assessment (FSA) score report was delivered to his school, (ironically) by express delivery, and I was finally given a copy demonstrating his percentile performance in last year’s assessment.  For some reason, never explained, his scores were late reported by the state. He is scheduled to take the 4th grade FSA Writing component on March 1st (5 days after we received last year’s results). True story.

Last Spring, the inaugural administration of the Florida State Assessment (FSA) was marred by computer glitches, cyber attack and validity questions.  Round Two begins on February 29, 2016. Has anything changed? Is Florida ready?

Last Spring: Frozen Screens. Service denials. Cyber attacks. Server Crashes. Log-in failures. This year: Low Expectations.

The computer-based FSA had problems from the start. “From the get-go, the FSA test was riddled with technical problems, including a cyber attack from an unknown source and a slow-moving AIR server which left thousands of students with error messages in their language arts and mathematics tests.” (read about it here and here)

The Florida Department of Education did launch an investigation into the server attack that, presumably, had left thousands of students across the state unable to take the new FSA, but 6 months later, the Department still had no answers to who was behind the server attack or why it took place (details here). Is that investigation still ongoing? Are adequate safeguards in place?

In September, State Education Commissioner, Pam Stewart, announced that she would seek liquidated damages from American Institutes for Research, or AIR, the creator of the FSA (damages which are clearly delineated in the State’s executed contract and collection of such is required, “if applicable”, by Florida State Statute). Nevada has successfully recovered $1.3 million in damages from AIR for similar testing disruptions there (details here). Recently, an attorney from the FLDOE refused to comment on the status of this, stating it was “pending litigation,” but we can find no public record of any pending litigation. Is the State still pursuing this?

This year, expectations for FSA computer-based testing are being set low, as reported here.

“The Florida Department of Education, its testing vendor American Institutes for Research, along with districts and schools, have taken several steps to prevent such troubles. Those include expanding bandwidth, upgrading defenses against outside attacks and improving testing software.

Even with such moves, though, the department warned that students still might encounter interruptions beyond their control. And that, said FairTest public education director Bob Schaeffer, could hurt some children.

Imagine the impact when the screen goes blank for a seventh-grader taking a civics test required to get out of middle school, Schaeffer said. “For an emotional adolescent to experience that, it’s a scary situation.”

Yet there’s almost no way to guarantee trouble-free computerized testing on a stage as large as Florida’s, experts said.

So… expect glitches.  Lots of them.

Earlier this month, students across Florida, when asked to participate in district “infrastructure testing,” reported wide-spread issues, in multiple counties, with the system: screens freezing, difficulty loading, bugs in the program, unexpected shut downs.  It appears there is still “work to be done.” Similar situations in Tennessee, led that state to “pull the plug” on this year’s computer testing (read about it here). What would it take for Florida to pull the plug and return to paper and pencil tests, as recommended by many of Florida’s superintendents?

There are more reasons than just technical issues to question continuing with computer testing. Reports (here and here) from Illinois, New Jersey and Maryland have shown students who took the 2014-15 PARCC exams (similar to Florida’s FSA) via computer tended to score lower than those who took the exams with paper and pencil—a discovery that prompts questions about the validity of the test results and poses potentially big problems for state and district leaders. Why evaluate student proficiency with an assessment that under-represents their abilities?

Were similar results found with the Florida assessments?  Did students perform better on paper tests than computer tests? Is the comparison even being made?  No one knows because the 2015 FSA Technical Report, due out in January, has yet to be released.

Technical Report not complete

The 2015 FSA Technical Report, usually published by January following a test’s administration, is still (per personal communication with the FLDOE on 2/25/16)  “undergoing final reviews and will be available shortly.” In the past, FCAT Technical Reports (past examples can be found here) focused on test validity and reliability, “key concerns for establishing the quality of an achievement test such as the FCAT.” Also in the past psychometric analysis was the major focus of these reports. Without a completed report, is Florida proceeding with this year’s FSA administration in the absence of proof of psychometric validity?

Interestingly, the 2014 Florida Statewide Assessments (FCAT 2.0) Technical Report (released in 12/2014), on page 137, suggested that further studies were needed to verify some implication arguments. “This is especially true,” it read, “for the inference that the state’s accountability program is making a positive impact on student proficiency and school accountability without causing unintended negative consequences.” (Emphasis mine.) I am, especially, eager to learn whether Florida completed these “further studies” because it appears there are LOTS of unintended consequences in the current accountabaloney system.

An Incomplete and Not Independent Validity Study

Questions regarding the validity of the FSA began to be asked last March, when Commissioner Stewart testified before a Senate Education Committee that the FSA had been validated in Utah and she promised to provide the senators with those reports.  No such reports were ever delivered.  This prompted legislators to pass a bill requiring the department to hire an independent company to verify whether the FSA was a valid tool to assess the academic achievement of Florida’s students.

Let’s take a little pause, here, to review how to determine whether a test is valid or reliable.  This article outlines the process (you can substitute “FSA” for “SBAC” and the article would still, mostly, hold true):

“With real scientific educational research, a group of independent researchers, not bought off by any billionaire, would select a large group of representative students, such one thousand 8th graders randomly chosen from an urban school district. These students would be randomly assigned to an experimental group and a control group. A survey would be given to each group to make sure that the groups were matched on important characteristics such as free and reduced lunch status and ELL status. Each group would be given a series of tasks such as completing the 8th grade NAEP Math test, the 8th grade SBAC Math test, the 8th Grade MAP test, the 8th Grade MSP test and/or the 8th Grade Iowa Test of Basic Skills. These would then be compared against comparison measures such as teacher grades in their previous and current year math courses. The actual test questions of every test would be published along with student scores for each test question on each test. Objective analysis and conclusions could then be made about the reliability and validity of various measures using these carefully constructed norming studies. This results and conclusions would be peer reviewed and quite often the entire study could and would be replicated by other independent researchers at other major Universities. “

Suffice it to say, Florida “independent” validity study didn’t do that.

Partial, hardly independent, FSA Validity Study

Florida hired Alpine Testing Solutions to perform the mandated validity study.  Alpine partnered with EdCount, a partner of AIR, the test creator.  The project team contained previous AIR employees.  So much for the independent part…

The Alpine Report was presented to the FLDOE on August 31, 2015, after allowing the DOE to review and make suggestions regarding two earlier drafts of the report (more here). The final report was released to the public on September 1st. The FLDOE announced the study showed the FSA to be “valid,” a claim that was challenged by educators (more here and here). The report recommended against using the results from the computer-based assessments as the sole factor in determining individual consequences, such as whether students should be promoted, retained or remediated, but felt those same scores COULD be used to evaluate teachers and schools. Not surprising, many wondered how a test, not found to be accurate in measuring student achievement, could be used to rank teachers, schools and districts.

While many debated the contents of the Alpine report, what was NOT in the report was equally interesting.  Alpine was charged with reviewing the grade 3-10 English Language Arts (ELA) exams, the grades 3-8 Math assessments and the Algebra 1, Algebra 2 and Geometry End of Course (EOC) exams. Because of time constraints, however, the report ONLY evaluated ELA exams for grades 3, 6, and 10, Math exams for grades 4 and 7 and the Algebra 1 EOC. This leaves 11 of the 17 new FSA exams (ELA grades 4, 5, 8, and 9, Math grades 3, 5, 6, and 8, and Algebra 2 and Geometry EOCs) UNEVALUATED and possibly invalid. When will those tests be evaluated? Before the next FSA administration? That seems unlikely, since the next administration begins on Monday (2/29/16).

Also missing from the Alpine Report is any evaluation regarding whether the FSA is valid, fair or reliable for vulnerable populations, such as special needs students, english language learners and other at-risk populations, as outlined by Dr. Gary Thompson here.

“…Due to the limited time frame for developing the FSA, item reviews related to content, cognitive complexity, bias/sensitivity, etc. were not conducted by Florida stakeholders.” (Alpine Testing Solution, Inc. Validity Report P.35)

Per Dr. Thompson, “Neither Utah nor Florida has produced validity documents suggesting that either the SAGE or FSA high stakes academic achievement tests can validly measure achievement in vulnerable student populations, or that the current testing accommodations allowed or banned, are appropriate or fair.”

Without determining fairness for vulnerable sub-populations of students, continuing to use FSA scores to retain, remediate or prevent students from graduating is unconscionable. How can an accountability system be based on an invalid or unfair measurement?

So, in summary:

  • Expect computer glitches. Keep your FSA expectations LOW.
  • Don’t hold your breath for any liquidated damages from last year’s FSA fiasco, despite being spelled out in the AIR contract and mandated by statute.
  • State assessments, in general, are not appropriately validated. Will Florida ever evaluate whether, like the PARCC assessment,  FSA scores are worse on computer then pencil and paper?
  • The validity of the FSA is, at best, incomplete; the Alpine Study failed to evaluate 11 of 17 new assessments.
  • There are no documents suggesting the FSA can validly or fairly measure achievement in vulnerable student populations, yet those students suffer the same consequences as their more advantaged classmates.
  • Don’t expect the FSA Technical Report to address the unintended negative consequences of the current system.
  • My son’s test scores, delivered just 5 days before this year’s test, will not inform his instruction.
  • The Accountabaloney will continue until further notice.

Or, in other words, “Same song, second verse, could get better but it just gets worse.”

FSA testing starts Monday, whether we are ready or not.

 

 

 

 

What Do Florida’s School Grades REALLY tell you?

 

“What was educationally significant and hard to measure has been replaced by what is educationally insignificant and easy to measure. So now we measure how well we taught what isn’t worth learning.”

Arthur Costa, Emeritus Professor at California State University

Take a moment and think about the best moments your children have had at their public school… you know, those times where they came home so excited, talking a mile a minute, about what they have seen and learned. For my daughter, this might have been when the whole third grade spent the day celebrating “Early Man,” with art, music and skits, all dressed in fake fur, dirt smeared on their faces, hair all teased up into “Early Man Style,” culminated with a “eat with your hands” feast (Thank you, Mrs. Carter). For my son, it is probably a tie between “getting to do science experiments ALL DAY” and having a Beatnik Poetry Slam that was “Oh too cool” (Thank you Ms. Osborne). I remember similar moments from my childhood and they still make me smile. These are the moments that will make or break your child’s education experience. These are when the real learning, the kind that inspires, really happens. To everyone who has provided these moments for my children, I thank you.

This week, the Florida Department of Education will finally be releasing school grades from the 2014-15 school year, six months late, after the flawed rollout of the new Florida State Assessment (FSA). Superintendents have warned “these grades hold little value for school districts and should be viewed as such by the public.” In a recent op-ed, Orange County Superintendent and current President of the Florida Association of District School Superintendents, Barbara Jenkins explained the superintendents’ opposition to the publishing of school grades, asking for “incompletes” to be issued instead. In addition, she repeats that “superintendents in the 67 Florida districts have unanimously called for a comprehensive review of our accountability system in order to better inform students, parents, educators and the public.”

“Now is the time to admit that the current system is no longer sufficient, and simply labeling schools A through F provides an inaccurate picture of what is occurring in every school.”

Here’s a News Flash: Florida’s A-F School Grading System has NEVER been sufficient, it has ALWAYS provided an inaccurate picture of what is occurring is schools. Why? Because it focuses almost entirely on standardized test scores and they are a poor reflection of real learning. We hope Superintendent Jenkins understands that even with the addition of measured “learning gains” calculated from this year’s FSA scores, the A-F grading system will still be completely flawed.

According to the 2015 PDK/Gallup Poll, these are the factors most important to parents when choosing a school.  The primary factors being quality of teaching staff, curriculum, student discipline, class size, variety of extra-curricular activities and reputation of school. Only 15% of those surveyed felt “student achievement on standardized tests” was “very important” when choosing a local public school.

FullSizeRender-12

Florida School Grades do not reflect what parents deem important when choosing schools but, rather focus almost entirely on standardized test scores.  Here is the Elementary School Grades Model. Elementary schools are graded based on how well their students, in 3rd grade and up, perform on the grade level, standardized Math and English Language Arts (ELA) FSA and on the 5th grade Science FCAT. By testing students on a grade level assessment, they are looking for how many student are performing at or above average (because that is what “grade level” represents, the average performance of a child in that grade). Learning gains are calculated for all students and the learning gains of the lowest quartile of students count twice in the calculation.
FullSizeRender-21

This year, in all grade levels, English Language Learners will be required to sit for hours of FSA testing, even if they arrived in the country yesterday and do not speak a word of english. Why? To provide a baseline for a learning gains calculation when they are again asked to take the assessment next year.  After two years, English Language Learners’ scores will be counted in with the rest of the student body.

Also, it is the focus on preparation for the third grade math and ELA FSA (third graders must pass the reading/ELA portion for promotion to 4th grade) that drives much of the progress monitoring and test preparation in the younger grades. Children as young as kindergarten are required to have 90 minutes of uninterrupted reading instruction, leaving little time for creative endeavors, art, music and, in some schools, even recess. (Read here to learn about the Florida moms fighting to restore recess in our public schools, after being told there wasn’t time for daily recess “given the growing demands on schools to raise test scores”).

Middle schools, in addition to grade level performance and calculated learning gains on the Math and ELA, are rated on how well their 8th graders perform on the 8th grade Science FCAT, how many students pass the state created Civics End of Course exam (EOC) and how many students pass accelerated math (like Algebra 1) EOCs or qualifying industry certifications.

FullSizeRender-17

The number of students eligible for the “Acceleration Success” calculation in the Middle School Grade Model, is determined by the number of students in the class AND all 8th graders who scored a 3 or higher on EITHER Reading or Math in their 7th grade state assessment. To be clear, the number of students deemed eligible to take a high school level Algebra 1 course in 8th grade is NOT determined by their performance on any Algebra readiness standards but on their previous year’s FSA Math and Reading scores.

The High School Grade Model includes a calculation of overall 4-year graduation rate, but otherwise is entirely determined by standardized test scores. English Language Arts scores are calculated from performance on the 9th and 10 grade FSA ELA and Math scores are determined by performance on state mandated and created EOCs. Science achievement depends on scores for the state mandated Biology 1 EOC and Social Studies achievement depends on the state mandated U.S. History EOC.

FullSizeRender-13

In high schools, “Acceleration Success” is calculated as the percentage of graduates who, while in high school passed an AP, IB, or AICE examinations, earned a C or better in dual enrollment or earned a CAPE industry certification. In addition, high schools receive performance funding for enrolling students into these programs so there is a lot of “encouragement” to enroll students in these courses. Students, sometimes, find themselves enrolled in advanced classes they didn’t ask for and may have difficulty getting out of these classes if that is their desire. With the pressure to pass  AP exams, Monroe County has paid the National Math and Science Initiative, a non-profit, almost $2 million over three years to help improve participation and passing rate on AP exams, and is now requiring AP students to attend extra Saturday sessions during the school year.

What can we learn about our schools from their A-F letter grade?

School grades may be a reflection of the socioeconomic status of the student body. Researchers have been able to predict school grades based on US census data alone. Chris Tienken and colleagues “predicted the percentage of students at the district and school levels who score proficient or above on their state’s mandated standardized tests, without using any school-specific information” making the need for testing obsolete!  For more information, click here.

More concerning, when considering a local school’s grade, is whether the school has gone through any extraordinary measures to ensure high test scores from their students. Are the students engaging in creative engaging lessons that inspire a love for learning or does the curriculum resemble test prep? Have non-tested subjects, like art, music, physical education, etc, been minimized or marginalized to preserve more time for tested subjects? Is there time in the day to read fiction or fairy tales? For the youngest students, has seat work replaced time spent in free play or exploration?  For high school students, has the push towards AP classes eliminated Honors classes and/or inappropriately placed more and more students, ill prepared for the rigors of AP classes, into these classes, setting these students up for failure.  Are today’s AP courses really representative of college level courses or are they little more than test preparation for the required AP exams? Parents should be asking these questions of their highly rated schools. How many hoops have been jumped through in order to earn that coveted “A” rating? What have the students missed out on while practicing “hoop jumping”? Perhaps it is not surprising that many of Florida’s voucher students leave “A” rated public schools; are they escaping from the constant test pressure?

Likewise, some parents of children at schools that earn lower school grades must recognize the many positive experiences their school provides their children. Perhaps they have a wonderful chorus, an active theater group, a school garden… Maybe they focus more on the whole child and less on the test prep… Is the teaching staff loving and devoted to your children? These things are important and should be celebrated! Sadly, they are not even considered in the current school grade formula.

Any straight A student can tell you their final grade did not always reflect the quality of the course. Likewise, school grades may only reflect the test scores, not the quality of the education provided. In fact, parents of children in “A” schools, or in “A” districts, should be questioning whether their school focuses too much on testing and forgets the whole child. Parents need to start recognizing Florida’s A-F School grade system for what it really is: a reflection of a test focused system that has gone too far.

12631499_1259717524043938_5958608282608271154_n