ICYMI: The Florida Charter Study is Baloney

This post has been modified to included more specific information regarding Dr. Jay Greene’s testimony during the Citizens for Strong School v BOE trial.

Full disclosure: my son attends a charter school that currently satisfies his specific needs, so I understand why some parents choose charter schools for their children. As a former board member, I also understand the limitations of charter schools. More importantly, perhaps, as a physician and scientist, I am skeptical of research that finds results beneficial to those funding the study.

During the recent Citizens for Strong Schools vs Board of Education trial, defense witness Dr. Jay Greene presented the results of a new Florida charter school study, which was a follow up to an original study by Booker, et al.  Dr. Greene claimed this new study showed that charter school students in Florida have better adult outcomes; earning higher salaries and completing more years of higher education. (You can watch his testimony regarding this at 1:15:00 here.)

Dr. Greene is the endowed Chair of the Department of Education Reform at the University of Arkansas. I don’t believe he disclosed that his position, and indeed his entire department, was endowed by a $10 million gift from the Walton Family, a family outspoken in its support of charter schools at the expense of traditional public schools, vouchers and in education “reform” in general. (You can read more about Dr. Greene’s association with the Walton Family here and here. You can decide whether his perspective is that of an unbiased researcher or not.)

During his testimony, Dr. Greene explained the Booker, et al, study looked at charter middle schoolers and compared outcomes based on whether they attended a traditional public high school or a charter high school (this group he called “lucky” to have a charter high school nearby). He called the study a “particularly good study because it is a comparison of apples to apples.” He suggested that its experimental design approximated the quality of a placebo controlled medical trial. He suggested that every child who attended a charter school in middle school would choose to attend a charter school high school, if they were lucky enough to live near one. Thus, in Dr. Greene’s opinion, comparisons of groups of charter school 8th graders who chose to attend a charter rather than a traditional public school would be equivalent to a randomized controlled trial. He spent more than 10 minutes explaining what a high quality study this was and how it confirmed his belief that “charter schools tend to contribute to a high quality, efficient education system.”.

Dr. Greene’s mention of this new data  was of interest to me, as I had previously heard that most research had not been able to demonstrate significant benefits from charter school attendance.  The research I had seen appeared to confirm that, like all schools (public and private), some charter schools are good, some are particularly bad and most would be considered average.

For example, in 2009, charter schools in eight states, including Florida, were studied (read report here) “there is little evidence that charter schools, on average, are producing test-score impacts that are substantially better or worse than those of conventional public schools.” This, surprisingly, is the conclusion of the first Booker et al study that Dr. Greene discussed in his testimony.  The authors did see some small differences in some non-standardized test based outcomes and sought to pursue those ideas in the followup study.

In 2013, Stanford University’s Center for Research on Education Outcomes (CREDO) published the National Charter School Study, which looked at charter schools in 26 states and concluded that kids in most charter schools are doing worse or no better than students in traditional public schools. The study did show charters had made some improvements in the four years since the previous CREDO study. When looking specifically at Florida, the CREDO study showed students in traditional public schools, on average, read at a higher level than those in charter schools and did just as well in math. (Read more here).

Here at Accountabaloney, we have argued that test scores may not be the best measurement of a quality education (read more here and here), so I was intrigued by the report of this new charter school study that apparently showed education outcome measurements beyond mere test scores. Measuring quality education by a metric other than test scores would be a significant step in the right direction. I wanted to learn more about this study, so I turned to Google..

On April 11, 2016, the Hechinger Report reported that a new study in the Journal of Policy Analysis and Management (JPAM) showed “something startling: the charter students might not have produced higher test scores when they were in school, but years later, when they were in their mid-twenties, the charter school students earned more money, and were more likely to have attended at least two years of college (although still only half of them did so).

The JPAM study only looked at students who attended charter high schools and the authors suggested that, since the studied high schools were all small, the size of the high school (rather than whether it was charter or traditional) might be responsible for the positive results. The Hechinger Reort asked the question:

“Still, what’s causing kids to stay in college and later earn more money is a mystery. The authors wonder if these charter schools are doing more than providing better college counseling, and might be “particularly good at promoting skills such as grit, persistence, self-control, and conscientiousness.”

But there’s no data to prove that.”

A closer look at the study design suggests that there may be problems with the study. These are well documented by education blogger, Mercedes Schneider, in her April 10, 2016 blog, titled Charter School Grads Earn More? Read On…”, which we have reproduced here in its entirety:

On April 06, 2016, in its “Charters and Choice” section, EdWeek published an article entitled, “Charter School Graduates More Likely to Stay in College, Earn Higher Salaries.” The article concerns this study of Florida charter schools as published on April 04, 2016, in the Journal of Policy Analysis and Management (JPAM).

 

The EdWeek article notes that the JPAM study was funded by the Joyce Foundation (which is in Chicago; President Obama was once on the Joyce Foundation board). However, EdWeek does not mention that the Gates Foundation also funded the study.

 

In short, after offering quite a list of limitations, including the fact that no state colleges in Florida were part of the study– only private colleges and universities in Florida as well as colleges and universities in other states– the JPAM study finds in favor of charter high school grads.

The EdWeek article fails to mention this critical limitation.

Also, the study only involves comparison of two limited groups of students: those who attended Florida charter schools in the eighth grade and through high school versus those who attended charter schools in eighth grade and left to attend traditional public high schools.

 

Thus, the JPAM study omits all students who only attended traditional public schools. The EdWeek article does not clearly outline this key limitation in its opening statement– and it should have:
Charter school graduates in Florida were more likely stay in college and earn higher salaries than their district school peers.
That opener only confuses and misrepresents.

 

As for the finding that charter high school students are more likely to stay in college: The Florida high school graduates who attended charter schools from eighth grade through high school and who attended a private Florida college/university or an out-of-state college/university (remember, in-state postsecondary institutions were not tracked in this study) for two consecutive years (there’s your “stay in college”– two years in a row) is 9 percent higher for this specific group than for the “matched on key characteristics” control group of students who attended charters in eighth grade and then attended traditional public high schools.

 

Also, the very specific, charter high school attendee sample was 6 percent more likely to graduate from high school within five years.
Though the findings are statistically significant, practical significance (i.e., the difference this makes in the real world) is scant.

 

As for the charter sample’s higher earnings: This, too, is a statistically significant finding that is of questionable practical significance: An increase in maximum earnings of $2,318 between ages 23 and 25 years old, or 12 percent more than their traditional-public-high-school control group.

 

The EdWeek article mentions the 12 percent increase, but it does not include the actual dollar value of $2,318 per year for 23- to 25-year-olds. The actual dollar amount removes some of the shine from that percentage.

 

Consider this: Average maximum earnings for the traditional-public-high-school control group was $19,366 per year– not much to live on. So, this limited study found that students who attended Florida charters from eighth grade through high school and who attended in-state private colleges/universities or out-of-state colleges/universities (but state-funded colleges/universities in Florida) were earning an average of $21,684 per year between the ages of 23 and 25– also not much to live on.
Truth is, both $19k and $22k annually in 2016 would likely land these young people back in their parents’ basements if their only other option were to pay for housing without the assistance of one or more roommates.
So, yes, there might be statistical significance in this study, but it is overshadowed by both practical significance and the limits on the study sample as noted in this post.

Why would the authors choose to omit students who attended Florida’s public colleges and universities? Don’t most of Florida’s graduate attend public institutions? How can you trust results that eliminate data for most of Florida’s college bound students? Without the public university data, to most parents, the comparisons seem  meaningless. Why did Dr. Greene’s testimony fail to mention this?

We also agree with Ms. Schneider that, despite the statistical significance of the results, the real world significance of the results is debatable. The study did follow students who attended out-of-state universities; could this have affected that salary data?  According to the 2014 census, the median per capita income for Florida was $26,582 in 2014. Compared to the median US per capita income (of $28,889), Florida per capita income is $2,307 lower, almost the same difference as between the two study groups. Without controlling for this, the salary data is not impressive. Just moving out of state appears to give Floridians a $2,000 raise, whether they attended a charter high school or not.

We applaud the authors attempt to look beyond test scores as the measure of a quality education. That is about where the applause ends. Given the study limitations, we do not think the results support the conclusions testified to in the CSS v BOE trial. Indeed, Dr. Greene spent more than 10 minutes describing the “fine” attributes of this trial yet failed to mention that the college performance data disregarded data from any students attending Florida’s public universities or colleges. Was he unaware of this limitation or did he just choose not to mention it? Claiming charter school students have better adult outcomes, based on this study, is #accountabaloney.

Advertisements

2 thoughts on “ICYMI: The Florida Charter Study is Baloney

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s