ICYMI: The Florida Charter Study is Baloney

This post has been modified to included more specific information regarding Dr. Jay Greene’s testimony during the Citizens for Strong School v BOE trial.

Full disclosure: my son attends a charter school that currently satisfies his specific needs, so I understand why some parents choose charter schools for their children. As a former board member, I also understand the limitations of charter schools. More importantly, perhaps, as a physician and scientist, I am skeptical of research that finds results beneficial to those funding the study.

During the recent Citizens for Strong Schools vs Board of Education trial, defense witness Dr. Jay Greene presented the results of a new Florida charter school study, which was a follow up to an original study by Booker, et al.  Dr. Greene claimed this new study showed that charter school students in Florida have better adult outcomes; earning higher salaries and completing more years of higher education. (You can watch his testimony regarding this at 1:15:00 here.)

Dr. Greene is the endowed Chair of the Department of Education Reform at the University of Arkansas. I don’t believe he disclosed that his position, and indeed his entire department, was endowed by a $10 million gift from the Walton Family, a family outspoken in its support of charter schools at the expense of traditional public schools, vouchers and in education “reform” in general. (You can read more about Dr. Greene’s association with the Walton Family here and here. You can decide whether his perspective is that of an unbiased researcher or not.)

During his testimony, Dr. Greene explained the Booker, et al, study looked at charter middle schoolers and compared outcomes based on whether they attended a traditional public high school or a charter high school (this group he called “lucky” to have a charter high school nearby). He called the study a “particularly good study because it is a comparison of apples to apples.” He suggested that its experimental design approximated the quality of a placebo controlled medical trial. He suggested that every child who attended a charter school in middle school would choose to attend a charter school high school, if they were lucky enough to live near one. Thus, in Dr. Greene’s opinion, comparisons of groups of charter school 8th graders who chose to attend a charter rather than a traditional public school would be equivalent to a randomized controlled trial. He spent more than 10 minutes explaining what a high quality study this was and how it confirmed his belief that “charter schools tend to contribute to a high quality, efficient education system.”.

Dr. Greene’s mention of this new data  was of interest to me, as I had previously heard that most research had not been able to demonstrate significant benefits from charter school attendance.  The research I had seen appeared to confirm that, like all schools (public and private), some charter schools are good, some are particularly bad and most would be considered average.

For example, in 2009, charter schools in eight states, including Florida, were studied (read report here) “there is little evidence that charter schools, on average, are producing test-score impacts that are substantially better or worse than those of conventional public schools.” This, surprisingly, is the conclusion of the first Booker et al study that Dr. Greene discussed in his testimony.  The authors did see some small differences in some non-standardized test based outcomes and sought to pursue those ideas in the followup study.

In 2013, Stanford University’s Center for Research on Education Outcomes (CREDO) published the National Charter School Study, which looked at charter schools in 26 states and concluded that kids in most charter schools are doing worse or no better than students in traditional public schools. The study did show charters had made some improvements in the four years since the previous CREDO study. When looking specifically at Florida, the CREDO study showed students in traditional public schools, on average, read at a higher level than those in charter schools and did just as well in math. (Read more here).

Here at Accountabaloney, we have argued that test scores may not be the best measurement of a quality education (read more here and here), so I was intrigued by the report of this new charter school study that apparently showed education outcome measurements beyond mere test scores. Measuring quality education by a metric other than test scores would be a significant step in the right direction. I wanted to learn more about this study, so I turned to Google..

On April 11, 2016, the Hechinger Report reported that a new study in the Journal of Policy Analysis and Management (JPAM) showed “something startling: the charter students might not have produced higher test scores when they were in school, but years later, when they were in their mid-twenties, the charter school students earned more money, and were more likely to have attended at least two years of college (although still only half of them did so).

The JPAM study only looked at students who attended charter high schools and the authors suggested that, since the studied high schools were all small, the size of the high school (rather than whether it was charter or traditional) might be responsible for the positive results. The Hechinger Reort asked the question:

“Still, what’s causing kids to stay in college and later earn more money is a mystery. The authors wonder if these charter schools are doing more than providing better college counseling, and might be “particularly good at promoting skills such as grit, persistence, self-control, and conscientiousness.”

But there’s no data to prove that.”

A closer look at the study design suggests that there may be problems with the study. These are well documented by education blogger, Mercedes Schneider, in her April 10, 2016 blog, titled Charter School Grads Earn More? Read On…”, which we have reproduced here in its entirety:

On April 06, 2016, in its “Charters and Choice” section, EdWeek published an article entitled, “Charter School Graduates More Likely to Stay in College, Earn Higher Salaries.” The article concerns this study of Florida charter schools as published on April 04, 2016, in the Journal of Policy Analysis and Management (JPAM).

 

The EdWeek article notes that the JPAM study was funded by the Joyce Foundation (which is in Chicago; President Obama was once on the Joyce Foundation board). However, EdWeek does not mention that the Gates Foundation also funded the study.

 

In short, after offering quite a list of limitations, including the fact that no state colleges in Florida were part of the study– only private colleges and universities in Florida as well as colleges and universities in other states– the JPAM study finds in favor of charter high school grads.

The EdWeek article fails to mention this critical limitation.

Also, the study only involves comparison of two limited groups of students: those who attended Florida charter schools in the eighth grade and through high school versus those who attended charter schools in eighth grade and left to attend traditional public high schools.

 

Thus, the JPAM study omits all students who only attended traditional public schools. The EdWeek article does not clearly outline this key limitation in its opening statement– and it should have:
Charter school graduates in Florida were more likely stay in college and earn higher salaries than their district school peers.
That opener only confuses and misrepresents.

 

As for the finding that charter high school students are more likely to stay in college: The Florida high school graduates who attended charter schools from eighth grade through high school and who attended a private Florida college/university or an out-of-state college/university (remember, in-state postsecondary institutions were not tracked in this study) for two consecutive years (there’s your “stay in college”– two years in a row) is 9 percent higher for this specific group than for the “matched on key characteristics” control group of students who attended charters in eighth grade and then attended traditional public high schools.

 

Also, the very specific, charter high school attendee sample was 6 percent more likely to graduate from high school within five years.
Though the findings are statistically significant, practical significance (i.e., the difference this makes in the real world) is scant.

 

As for the charter sample’s higher earnings: This, too, is a statistically significant finding that is of questionable practical significance: An increase in maximum earnings of $2,318 between ages 23 and 25 years old, or 12 percent more than their traditional-public-high-school control group.

 

The EdWeek article mentions the 12 percent increase, but it does not include the actual dollar value of $2,318 per year for 23- to 25-year-olds. The actual dollar amount removes some of the shine from that percentage.

 

Consider this: Average maximum earnings for the traditional-public-high-school control group was $19,366 per year– not much to live on. So, this limited study found that students who attended Florida charters from eighth grade through high school and who attended in-state private colleges/universities or out-of-state colleges/universities (but state-funded colleges/universities in Florida) were earning an average of $21,684 per year between the ages of 23 and 25– also not much to live on.
Truth is, both $19k and $22k annually in 2016 would likely land these young people back in their parents’ basements if their only other option were to pay for housing without the assistance of one or more roommates.
So, yes, there might be statistical significance in this study, but it is overshadowed by both practical significance and the limits on the study sample as noted in this post.

Why would the authors choose to omit students who attended Florida’s public colleges and universities? Don’t most of Florida’s graduate attend public institutions? How can you trust results that eliminate data for most of Florida’s college bound students? Without the public university data, to most parents, the comparisons seem  meaningless. Why did Dr. Greene’s testimony fail to mention this?

We also agree with Ms. Schneider that, despite the statistical significance of the results, the real world significance of the results is debatable. The study did follow students who attended out-of-state universities; could this have affected that salary data?  According to the 2014 census, the median per capita income for Florida was $26,582 in 2014. Compared to the median US per capita income (of $28,889), Florida per capita income is $2,307 lower, almost the same difference as between the two study groups. Without controlling for this, the salary data is not impressive. Just moving out of state appears to give Floridians a $2,000 raise, whether they attended a charter high school or not.

We applaud the authors attempt to look beyond test scores as the measure of a quality education. That is about where the applause ends. Given the study limitations, we do not think the results support the conclusions testified to in the CSS v BOE trial. Indeed, Dr. Greene spent more than 10 minutes describing the “fine” attributes of this trial yet failed to mention that the college performance data disregarded data from any students attending Florida’s public universities or colleges. Was he unaware of this limitation or did he just choose not to mention it? Claiming charter school students have better adult outcomes, based on this study, is #accountabaloney.

Advertisements

Funding & Other Baloney Part II: How You Spend It Matters A Lot

In Part I of this series, we questioned whether Florida’s embarrassingly low per pupil spending, in the face of questionable measures of quality, really meant our education system was “efficient” or just cheap. We also asked that the Florida Department of Education stop celebrating high 4th grade reading scores as a measure of high quality, when those scores are clearly the result of mandated 3rd grade retention and any benefit noticed evaporates by the 8th grade assessments. In Part II, we wonder if increased funding might lead to improved outcomes in our public schools.

Part II: How You Spend It Matters A Lot

When parents learn that Florida spends well below the national average in per pupil spending (learn more here and here), many advocate for increased education spending as a way to improve the quality of our public schools. It seems to make intrinsic sense that schools spending significantly more educating their students would, naturally, be able to provide a higher quality education. Indeed, many districts rely on that belief to get voter approval for local sales tax initiatives or increased millages to help fund local schools.

On the other hand, education “reformers” frequently claim there is no relationship between per pupil spending and student outcomes on standardized test scores and they use that idea, frequently, to advocate for low levels of state education funding in the name of “efficiency.”  This appears to be a commonly held belief amongst Florida Board of Education members, the Department of Education and many, if not most, of our legislators.  The idea seems to defy common sense but (much to my surprise) the research on the subject, until recently, actually appeared to support the position.

The idea dates back to 1966, when James Coleman lead a group of sociologists to write a report on educational equality in the United States, commissioned by the U.S. Department of Education. It was a massive study surveying more than 150,000 students, one of the largest in history. The finished report was titled “Equality of Educational Opportunity,” and is common referred to as The Coleman Report.

The results of the Coleman Report are widely reported to show that variation in school resources (as measured by per-pupil spending and student-to-teacher ratios) was unrelated to variation in student achievement on standardized tests. These results have been upheld by further research leading to reformers to report this as a “commonly held” belief. Indeed, Dr. Jay Greene, testified to as much during the recent CSS v BOE trial, pointing out that variations in state funding did not correlate with proficiency levels in Florida’s districts (you can see his testimony here).

Interestingly enough, the Coleman Report didn’t really deny that funding or other school effects matter, but it did argue that other factors are more important (learn more here). “Specifically, the report found that student background and socioeconomic status are much more important in determining educational outcomes than are measured differences in school resources. It also affirmed that differences in schools—and particularly teachers—have a very significant impact on student outcomes.” The study, also, concluded that disadvantaged black children learn better in well-integrated classrooms, leading to the busing of students in attempts to desegregate public schools. Additionally, the Coleman Report, suggested that standardized tests measured cultural knowledge, not intelligence, putting minority students at a disadvantage.

Regardless, the belief became “commonly held” amongst education policy makers that increased funding will not improve education, pointing at high poverty schools with many special needs children requiring expensive programs, yet scoring low on standardized tests, as proof. Many education reformers equate increased funding for public schools, in the face of low standardized test scores, as “throwing good money after bad.”

What if the lack of a connection between improved funding and a better educational outcome is simply because standardized tests scores are not the best measurement of learning?

A recent study addressed the limitations of standardized testing and the results suggest that increased education funding may result in significantly improved outcomes, especially for children of low income families. The 2105 report by C. Kirabo Jackson, Rucker C. Johnson and Claudia Persico is titled The Effects of School Spending on Educational and Economic Outcomes: Evidence from School Finance Reforms.The report asks the question, “Does school spending matter after all?” (You can read more about it here and here.)

“Tracking students born between 1955 and 1985, the researchers isolated the districts where court-mandated reforms would affect spending at those students’ schools; next, they compared those students’ achievement to similar students’ progress in districts that did not receive greater funding. The effects they found were huge and likely causal: A 10 percent increase in per-pupil spending led to higher graduation rates, nearly 10 percent higher earnings in adulthood, and fewer incidents of poverty later in life. The researchers found small effects on students from affluent families but huge effects on poor students.”

So, increased periods of spending had a significant effect on graduation rates AND  narrowed the wage gap between low-income and non-poor students when they reached adulthood:

highergradratenarrowingwagegap

This study showed that the “commonly held belief” may be wrong, demonstrating, instead, that increased education funding can have significant impacts on educational and economic outcomes, particularly for children in low income families.  This suggests that increased funding, whether it results in improved test scores or not, may lead to improved lives. Perhaps standardized test scores are NOT the best measure of the impact of education funding? Maybe narrowing the achievement gap on standardized test scores is not nearly so important as narrowing the economic gap? Maybe a better metric of quality is needed? We think Florida should consider this.

Public education advocates are often accused of wanting to “throw good money after bad.” Rest assured, we are not asking for that, nor were the authors of the 2015 study. The 2015 study authors cautioned that how additional funds were spent “matters a lot.

The authors evaluated the effects of court-mandated spending increases on spending for school support services, physical capital, and instruction. They also estimated the effects on student-to-teacher ratios, student-to-guidance-counselor ratios, teacher salaries, and the length of the school year. What they found was:

“… exogenous increases in school spending are more likely than other forms of school spending to go to instruction and support services. The increases for instruction and for support services (which include expenditures to hire more teachers and/or increase teacher salaries along with funds to hire more guidance counselors and social workers) may help explain the large, positive effects for students from low-income families.”

“While there may be other mechanisms through which increased school spending improves student outcomes, these results suggest that the positive effects are driven, at least in part, by some combination of reductions in class size, having more adults per student in schools, increases in instructional time, and increases in teacher salaries that may help to attract and retain a more highly qualified teaching workforce.”

So, when appropriately used, increased education funding CAN lead to improved outcomes, especially for our most at-risk public school children. Sadly, in recent years in Florida, rather than focusing funding towards instruction and support services, state and federal mandates have directed large portions of school budgets towards things like technology infrastructure, progress monitoring and assessments. In the face of nearly flat education spending, the net result is less funding for the very things shown to improve student outcomes: class size reduction, guidance counsellors, social workers and the increased salaries necessary to attract and retain a high quality workforce.

We urge the Florida DOE to expand its definition of educational outcomes beyond standardized test scores and reconsider funding priorities. Increased funding, when spent appropriately, can have significant, lasting effects for our students, especially those most at-risk. Continuing to ignore the positive effects of appropriate funding is #accountabaloney.

Funding & Other Baloney, Part I: 4th Grade Reading Scores and “Efficiency”

I will venture a guess that no one truly believes the measure of a high quality education system should rest on reading proficiency levels in 4th grade. Yet, I just spent weeks following the Citizens for Strong Schools v Board of Education (CSS v BOE) trial, where the defense repeatedly argued just that: Florida has high 4th grade reading scores on the NAEP (National Assessment of Educational Progress, the “Nation’s Report Card”) and, therefore, our pathetically low level of K-12 per pupil education spending simply means our education system is “efficient.”

That, my friends, is Accountabaloney.

First, Florida’s high 4th grade reading scores are, clearly, a direct result of our mandatory 3rd grade retention policy. We retain our poorest readers in 3rd grade, allowing them an extra year of “practice” (i.e. development) before they are measured on the 4th grade NAEP assessment. In fact, Dr. Jay Greene, a witness for the defense (watch him here at around 1:35) testified that Florida has retained so many third graders that there is no longer any “stigma” attached to grade retention (tell that to the mother of a child threatened with mandated retention, I dare you). Regardless, Florida’s 3rd grade retention policy disproportionately retains low income, Hispanic and African American children and likely explains Florida’s “success” on the 4th grade NAEP for these groups. (The well documented, long lasting negative effect of grade retention will be a subject of an upcoming blog, in the meantime you can read this or this). Even so, how well a child reads when they are 10 years old remains a strange metric of the overall quality of an education system.

Additionally, when you hear someone celebrate Florida’s 4th grade NAEP scores you should ask “How do we do in 8th grade?” Florida’s 8th grade results are NOT stellar (learn more here). Any so-called “gains” our retained 4th graders made have been lost by 8th grade. Our 8th grade math results are especially dismal and significantly falling off the national average.

It is certainly true that Florida underspends on education, as demonstrated on this latest study from Education Law Center/Rutgers, “Is School Funding Fair? America’s Most Fiscally Disadvantaged School Districts“, which examined education spending 2008-2013 and  gave Florida an “F”, placing it near the bottom in public education spending. As reported in The Times Union:

  • Florida ranked 42nd for education funding per student ($7,033 compared to the average national funding level per student of $9,766.80)
  • Florida ranked 49th for the number of teachers per 100 students in public schools.
  • Florida showed the second steepest decline in education funding, behind Hawaii, between 2008 to 2013, the years the study measured.
  •  while Florida’s was , after adjusting for regional differences and district sizes.
  • Florida spends nearly the same on high-income schools as it spends on poor schools, earning Florida a “C” in that category.
  • Florida ranked 47th among states for how it apportions teachers: low-poverty schools here have more teachers per 100 students than high-poverty schools.

DOE representative responded to The Times Union:

“Unfortunately, the Education Law Center’s report focused solely on funding levels without considering the state’s return on its investment in education,” said Meghan Collins, Director of Communications.

“Florida’s public education system is being funded at historic levels, and is ranked 11th nationally for k-12 achievement. Furthermore, our fourth grade students are among the best readers in the world and our high school graduates rank third in the nation for performance on advanced placement exams. We are proud of these accomplishments and believe Florida should be commended for its excellent student outcomes, which is our top priority.”

Ah, the “efficiency” argument and those 4th grade reading scores again…

During his testimony in the CSS v BOE trial, Dr. Greene suggested that Florida’s education funding must be adequate because proficiency levels were slowly improving.  He compared this to knowing a child was getting enough food because he was growing.  He was unable to define what minimal amount of funding is required to provide a high quality education because there is no universal standard for “proficiency”. He explained that proficiency standards set by the state are NOT objective numbers but are “simply based on a judgement by the state about performance and what performance it’s expecting from students and schools.” He compared the lack of an objective standard for proficiency to the inability of people of different heights to agree on “what is tall.”

So, let me get this straight. The state sets the budget AND the state sets the proficiency standards based on its own judgement about performance expectations. Hmmm… The state creates the test and then sets the bar that determines proficiency levels so, essentially, they could manipulate the system to assure the appearance of adequate funding. Hmmm… How can such a fluid measure be used as evidence of a quality system over time? What is tall?

As a parent whose children have attended public school in Florida, I have seen what my district has been able to do with the limited amount of funding from the state; a level of funding the state, apparently, believes is adequate to obtain the proficiency rates they “expect from students and schools.”  While this funding allows my local schools to get enough students to minimal proficiency standards to earn stellar school grades (a grading system ALSO manipulated by state policy), can the overall education they receive really be described as “high quality”?

An estimated 25% of the school year (or more) is spent practicing, prepping and administering state mandated assessments. That is expensive, but it hardly qualifies as high quality. In many schools, recess has been eliminated for the youngest learners, social studies and science courses have been incorporated into reading lessons, and art and music instruction has all but disappeared.  Is that high quality? Teachers lament that the required pacing guides no longer allows them the time to delve deeper into topics, to follow the class’ interests with projects, take field trips, and engage in in-depth study of topics of interest. There is barely funding for field trips, anyway. Gifted and Special Education programs have been decimated. For half of the year, computer labs are used primarily for testing and test prep. School counsellors have become testing coordinators and teachers spend their scarce prep time managing data…  Should we still call such a system high quality, just because of some standardized test scores? We think not.

For decades, districts have been notified of meager state funding and asked to make do with the funds offered, without regard to the cost of quality programming. At the same time, unfunded state mandates like requiring the technology infrastructure necessary for computer based state testing has effectively decreased available classroom funds. Districts have been forced to cut programs once felt to be essential to a quality education. Parents are asked to bring in reams of copy paper and other school supplies. Schools run out of pencils. Teachers resort to “Go-Fund ME” campaigns to obtain basic supplies. Low income schools, often with the lowest test scores, have been hardest hit.

Let me simply suggest that, in the absence of quality programming, 4th grade reading scores can not be the measurement of high quality and the amount of per pupil funding required to give ALL children the high quality education they deserve is “something more than what we get now.”  This is not to say that more money spent improperly is warranted but, clearly, there could be less money spent on testing and test prep and more money spent in the classrooms, on authentic learning experiences.

Celebrating how little we can spend to get impressive reading scores from our retained 4th students doesn’t mean we’re efficient, it means we’re either cheap or delusional. As long as the State’s accountability metric is grade level proficiency on a standardized test, one where THEY get to manipulate the proficiency bar, the state will continue to be able to game the system.

The Florida Department of Education needs to stop celebrating the 4th grade NAEP scores. A high quality education cannot be adequately measured by grade level test scores, especially those of 10 year olds. Seriously, no one believes that. Celebrating our “efficiency” without a legitimate measure of quality is Accountabaloney.