Tuesday, December 17, 2019

Detailed Analysis Of State Milestones Data Documents Overall Improvements Of Oconee County Schools On Standardized Test Scores

***Also Shows Some Area Of Relative Weakness***

Oconee County School administrators have gone out of their way to make sure the Board of Education, parents and the general public know how highly ranked Oconee County Schools are based on the state’s Milestones assessments, released at the end of the summer.

Anisa Sullivan Jimenez, director of Communications for Oconee County Schools, sent out a news release on July 29 stating that “Oconee County Schools’ Milestones, End of Grade, and End of Course assessment results have placed OCS as the #1 school system in the state of Georgia.”

Jimenez congratulated the Board of Education for the ranking at a meeting later that day, and School System Chief Operating Officer Claire Buck gave a detailed report to the Board a week later.

Buck told the Board that “Oconee County Schools overall was ranked the number one school system in the state of Georgia.”

School Superintendent Jason Branch got into the act at the School Board meeting on Sept. 9, when he told the Board that Oconee County Schools were rated as the number 1 county school system in the state.

The entrance to the school system Central Office, 34 School Street in Watkinsville, through which Board members and the public pass to get to the Central Office Meeting room, where the Board holds its regular meeting, has a sign proclaiming the system is Number 1 in the state of Georgia.

The rating is a fact, as SchoolDigger.com gave Oconee County the top ranking of school districts in the state.

The focus on the rating by SchoolDigger.com, however, overshadows real achievements by Oconee County Schools, which improved in 22 of 29 comparisons with Milestone data of a year ago, and glosses over some areas where Oconee County Schools underperform other elite school districts.

SchoolDigger.com Ranking

SchoolDigger.com reported in July that Oconee County School scored 1 point higher than Buford City Schools in Gwinett County in its 2019 ratings, which it published online after the state released the Milestones data in late July.

Sign At School System Central Office

SchoolDigger.com weights and interprets the data from the state and other sources in computing its ratings score and making it rankings.

The Georgia Department of Education does not rank school systems.

Independent of the School.Digger assessment, Oconee County Schools made its own comparisons based on the Milestones Data.

In her news release on July 29, Jimenez said that “On each of the 23 assessments for the 2018-19 school year, OCS had an average well above that of the state of Georgia.”

“Of the 23 assessments given, the OCS average was in the top 10 out of all districts in the state on 20,” Jimenez wrote.

“OCS was the #1 district in the state on the following assessments: 4th grade English/Language Arts and Algebra I,” Jimenez reported.

Analysis Of Raw Data

For reasons that I will explain below, over the last months I conducted my own analysis of the raw data released by the state as part of the Milestones assessment.

That analysis confirms the high level of performance of Oconee County Schools.

The analysis is based on the same 23 comparisons Oconee County Schools used when it released its report on the Milestones data to the public and the Board of Education, plus six others.

Oconee County Schools perform particularly well in comparison with the vast majority of the schools systems in counties surrounding Oconee.

This is a comparison families looking to move to the area are likely to find most relevant. These families are not as likely to be concerned with the performance of Buford City Schools as with those in Clarke, Jackson, Barrow, Madison or even Oglethorpe counties.

The analysis of the raw data from the Georgia Department of Education documents more than does the Oconee County Schools analysis that the school system improved its performance in 2019 versus a year earlier on most measures.

It also documents areas of the Oconee County School system that are weaker overall than the system as a whole.

Focus On Number 1 Ranking

Neither Jimenez nor Buck noted in their reports to the Board that the number 1 ranking was not by the state, but by SchoolDigger.com.

Branch At BOE Meeting 

A news release on the school system website repeats the comments of Jimenez and Buck about the number 1 ranking, this time with a note at the bottom that the ranking was by SchoolDigger.com.

The news release also has a note indicating that Oconee County Schools were ranked number 1 in the state among county school systems by Niche.com, as Branch had stated in his comments to the Board.

The news release does not state that Oconee County Schools were number 2 among all school districts in the state, with Buford City Schools rated number 1.

SchoolDigger.com

SchoolDigger.com, according to its website, is a service of Claarware LLC. The web site states that Claarware LLC is a one-person software development shop, founded in 2006 by Pete Claar.

SchoolDigger.com says its purpose is “to empower parents like you to make informed choices about choosing a school for your child,” according to its web site.

“Our database contains detailed profiles for over 136,000 schools in every state in the US, including 20 years of enrollment data, several years of test scores, crime data, real estate data, and everything our team hopes is useful in helping you make better school choices,” the web site states.

SchoolDigger.com has a rating score with a theoretical range of 0 to 1, and Oconee County rated .946 in 2019. Buford City School had a rating score of .945.

Bremen City Schools in Haralson County, west of Atlanta, had a rating score of .933, followed by Jefferson City Schools in Jackson County with a rating of .929.

In 2018, Buford City Schools was ranked Number 1, Oconee County Schools was ranked Number 2, Bremen City Schools was ranked Number 5, and Jackson City Schools was ranked number 3.

Forsyth County had been Number 4 in 2018 and fell to Number 5 in 2019.

SchoolDigger states that it updated its 2019 ratings on “July 26, 2019, based on the 2018-19 school year test scores.”

Niche.com

Niche.com, another evaluator of schools systems around the country, released it 2019 rankings for Georgia School systems shortly after the state released the 2018-2019 Milestones data.

Niche.com, based in Pittsburg, Pa., says its 2020 Best School Districts ranking is “based on rigorous analysis of academic and student life data from the U.S. Department of Education along with test scores, college data, and ratings collected from millions of Niche users.”

Niche gave Oconee County Schools an A+ rating. It gave Buford City Schools the same A+ rating.

Niche ranked Oconee County Schools as “Number 2 Best School Districts in Georgia.” Buford City Schools ranked Number 1.

Oconee County schools have said that Oconee County Schools are the top ranked “county” school district in the state, and that is true because Buford City Schools is a city district.

Niche.com ranks Forsyth County Schools as the Number 3 district, Decatur City Schools as Number 4, and Jefferson City Schools as Number 5.

The Georgia Department of Education says that there are 181 school districts in Georgia, including the 159 county school districts and 22 city school districts.

Milestones Assessment System

Georgia Milestones is a Georgia Department of Education initiative designed to measure how well students have learned the knowledge and skills outlined in the state-adopted content standards in English Language Arts (ELA), mathematics, science, and social studies.

Parents and school administrators locally have been critical of the tests, which take up a lot of educational time and can lead to teaching designed to improve test performance rather than knowledge.

Both SchoolDigger.com and Niche.com use data from the Georgia Department of Education’s Milestones Assessment System, which provides assessments for grades three through high school.

SchoolDigger.com and Niche.com weight and interpret the data according to their own techniques, which are not precisely detailed.

The July 29 news release from the Oconee County School System did not say who did the ranking that placed Oconee County Schools as Number 1, and Buck, in her presentation, also did not indicate that SchoolDigger.com was the source of the Number 1 ranking.

The Oconee Enterprise, in its top story in its Back-to-School section of the paper on Aug. 1, incorrectly attributed the ranking to the state, which does not rank school systems based on the Milestones data, rather than to SchoolDigger.com.

Data Available

Buck produced a two-page report with a 10-page supplement for her Aug. 5 presentation to the Board of Education regarding the release of the Milestones data by the state.

The 10-page supplement provided tables showing comparisons of Oconee County Schools with other school systems in the state on 29 different areas of performance.

In response to the July 29 news release, I downloaded the data from the state Department of Education web site so I could examine in detail the state results.

When I attempted to replicate the data in the charts used by Buck, I ran into two problems, which I discussed with her on Aug. 12 after a Board of Education meeting.

Three of the scores for Oconee County--for fifth grade science, fifth grade social studies, and eighth grade social studies--did not match the data for Oconee County in the state date file. The Oconee data were subsequently corrected by the school system.

In addition, the school system at one point in reporting the data used spring 2019 test scores for Oconee County, but at another point combined some of the Oconee County fall 2018 and spring 2019 scores.

When the Oconee fall and spring scores were combined, they were compared with data from other schools systems based solely on spring data, making the comparisons of uncertain utility.

My Interests

I am a trained social scientist and have considerable experience in evaluation research.

Buck At BOE Meeting

Much of my research in my 43-year academic career focused on assessment of journalism and mass communication education. I also have analyzed rankings systems used in education and have assessed training programs for journalists around the world.

As a result of this experience and interest, I decided to expand on the analysis provided by Buck and to make the comparisons of the Oconee data for the spring of 2019 with spring 2019 data for school systems across the state.

I also downloaded the 2018 Milestones data and compared the performance of the School System that year with the performance as reflected in the 2019 data.

It was a tedious process, but similar to others I’ve done over the years. And it is a process and an analysis I enjoy.

I am doing this post during the holiday period when few government bodies in the county are meeting and I have had the opportunity to complete the work I started in July.

Summary Assessment

I focused on how Oconee County Schools performed on 29 comparisons.

This included the 23 comparisons Buck used in part of her report to the Board plus six others.

Buck had not included in all of her comparisons of how Oconee County stacked up against districts around the state six measures of reading performance.

The analysis I performed shows that Oconee County schools perform at a very high level, placing the system in an elite class of school systems around the state.

The elite group of school systems also includes Bremen City Schools in Haralson County, west of Atlanta near the Alabama border, Forsyth County Schools north of Atlanta, City of Decatur Schools, Buford City Schools and Jefferson City Schools in Jackson County.

On two simple comparisons–the number of times a school system scored as the top ranked district across 29 comparisons and the number of times a school system ranked in the top 10 districts on those same 29 comparisons–Oconee County Schools rank high.

Oconee County Schools tied for second place in terms of the number of times it had the top score in the state on those 29 comparisons and was the third system in the state in terms of the number of times it is ranked in the top 10 districts on the 29 comparisons.

Bremen City Schools had more number one ranks out of 29 possible comparisons (10) than did Oconee County Schools, which had 4. Oconee County Schools was tied in the second position with Forsyth County, also with 4 first place ranks.

Forsyth County was ranked in the top 10 on 29 comparisons, giving it the number 1 rank using that assessment standard.

City of Decatur School was in the second position, with 25 top-10 rankings, followed by Oconee County with 24 top-10 rankings, and Bremen City Schools and Buford City Schools, both with 23 top-10 rankings.

The analysis also shows that Oconee County Schools performed better in 2019 than in 2018 on 22 of the 29 comparisons.

The analysis also shows that no school district in the counties surrounding Oconee County other than Jefferson City Schools (0 top rankings among 29 comparisons but 21 rankings in the top-10 among the 29 comparisons, for a rank of number 6) comes close to Oconee County Schools on these criteria.

Comparisons Explained

The Milestones tests measured end of grade reading status for grades 3 through 8 (6 comparisons), English Language Arts for grades 3 through 8 (6 comparisons), mathematics for grades 3 through 8 (6 comparisons), Science for Grade 5 (1 comparison), and social studies for grades 5 and 8 (2 comparisons).

The tests also measured end of class performance for the spring of 2019 for 9th Grade Literature, American Literature, Algebra 1, Geometry, Biology, Physical Science, U.S. History, and Economics.

The comparisons are based on 21 End of Grade comparisons and eight End of Class comparisons, for the total of 29.

In ranking the schools, I eliminated the charter schools and other schools included separately in the state report and compared only city and county school systems. This is the same as what Buck had done in her report.

Buck focused her attention in her written comments on 23 of those 29 comparisons. She eliminated the Reading Level comparisons for grades 3 through 8. I included them in my analysis.

Criteria Identified

Anisa Sullivan Jimenez, director of communications for Oconee County Schools, directed me to two guides to help me understand the comparisons the school district had made when it released the reports.

The first is the End of Grade Interpretive Guide, and the second is the End of Course Interpretive Guide.

I used the same criteria used by the county in its comparisons in making my comparisons.

For the Reading Status comparison, that criterion was the percent of students at grade level or above.

For the other comparisons, I used percent classified by the state as “Proficient Learner & Above.”

Terms Explained

The Interpretive Guide for EOG says the reading scores match a student’s reading ability with the difficulty of textual material appropriate at the respective grade level.

For the other tests, the EOG Guide says “Proficient Learners demonstrate proficiency in the knowledge and skills necessary at this grade level and content area of learning, as specified in Georgia’s content standards. The students are prepared for the next grade level and are on track for college and career readiness.”

The Guide says “Distinguished Learners demonstrate advanced proficiency in the knowledge and skills necessary at this grade level and content area of learning, as specified in Georgia’s content standards. The students are well prepared for the next grade level and are well prepared for college and career readiness.”

The Guide says that “The achievement level classification for a student is determined by the scale score cuts. EOG scores are reported on a scale that can range from 140 to 830. The minimum and maximum scale scores for the different EOG assessments differ because the tests vary in length and their relative difficulty.”

On all of the tests, a proficient learner and above would have had a score of 525 or above.

The Interpretive Guide for End of Class says “Proficient Learners demonstrate proficiency in the knowledge and skills necessary at this grade level/ course of learning, as specified in Georgia’s content standards. The students are prepared for the next grade level or course and are on track for college and career readiness.”

The Guide says “Distinguished Learners demonstrate advanced proficiency in the knowledge and skills necessary at this grade level/course of learning, as specified in Georgia’s content standards. The students are well prepared for the next grade level or course and are well prepared for college and career readiness.”

As with the End of Grade scores, End of Class scores are reported on a scale that can range from 140 to 830, and on all of the tests, a proficient learner and above would have had a score of 525 or above.

Rankings Compared

Oconee County’s four Number 1 rankings in the state come from one End Of Grade comparison and three End of Class comparisons.

Oconee County Schools had the highest score on 4th Grade English Language Arts.

Oconee County Schools also had the highest score on Algebra 1, U.S. History and Economics.

Oconee County Schools had only five of the 29 test areas not in the top ten ranks in the state, and the lowest rank was only 17.

Oconee County Schools had a rank of 12 in 5th Grade Science, 12 in 8th Grade Reading Status, and 13 in 8th Grade Mathematics.

In End of Class comparisons, Oconee County Schools ranked 11 in Biology and 17 in Physical Science.

Comparisons 2019 To 2018

Comparisons of the 2019 Milestones scores with scores from 2018 show that Oconee County Schools not only did well in 2019 but it largely improved the scores in 2019 over those from 2018.

(Click on any of the charts in this post to enlarge it.)

Results were more mixed in terms of Reading Level scores than for others, but Chart 3 below shows that scores increased from 2018 to 2019 for grades 4, 6, and 7 and declined for grades 3, 5, and 8.

The highest score is for 2019 7th Grade Reading, with 89.4 percent of the students performing at or above grade level.

Eighth Grade Reading status was a relatively low 85.9 percent at grade level or above, down from 2018. The county ranked 12 in the state on the 2019 measure.

The number of test takers in this and the following charts is shown in parentheses with the test label.

End of Grade scores for 3rd graders are in Chart 4, which shows that scores in 2019 were higher than scores in 2018 for both English Language Arts and Mathematics.

In 2019, 71.0 percent of the Oconee County Schools test takers were performing at the level of proficient learning or distinguished learner in English Language Arts, and 77.0 percent were performing at that level in Mathematics.

End of Grade scores for the 4th grade are in Chart 5. Once again, scores increased in both English Language Arts and Mathematics from 2018 to 2019.

Fifth Grade scores are in Chart 6.

Scores increased from 2018 to 2019 in English Language Arts and Mathematics but declined in Science and Social Studies.

Oconee County Schools ranked 12 in the state in terms of Science in 2019.

Chart 7 shows large increases in scores for 6th Grade from 2018 to 2019 in both English Language Arts and Mathematics.

Milestone scores for English Language Arts and Mathematics increased modestly for Oconee County Schools for the 7th grade from 2018 to 2019.

Eighth grade scores increased from 2018 to 2019 in English Language Arts, Mathematics and Social Studies.

Even with the slight increased in the Mathematics scores, Oconee County Schools ranked 12th in the state on this test result.

End of Class scores increased from 2018 to 2019 in 9th Grade Literature, Algebra 1, Geometry, Physical Science and Economics.

Scores dropped in American Literature, Biology and U.S. History.

Oconee County Schools ranked 11th in the state in Biology scores and 17th in Physical Science stores.

Scores in Biology were lower in 2019 than in 2018, but they were higher in Physical Science in 2019 than a year earlier.

Other Assessments

The Milestones assessments are only one of many released each year, and Oconee County Schools performs highly on most assessments.

In October, Oconee County Schools Chief Academic Officer Buck reported that Oconee County’s 2019 composite mean score on the SAT college entrance exam placed it fourth in the state.

Oconee County ranked seventh in the state this year in terms of four-year high school graduation rates, up from a rank of 23 a year ago, Buck said.

Buck subsequently reported to the Board on the release of data from the Georgia Department of Education for the College and Career Ready Performance Index and for the five year Graduation Rate.

The 2019 score was the third highest among county systems in the state, Buck said.

The five-year graduation rate for Oconee County Schools placed the system at the number seven spot among school districts in the state, Buck reported.

Buck also reported that the results of the ACT college entrance exam scores for 2019 showed that the average Composite Score for Oconee County test takers put the system in the number four rank in the state.

The release of the Milestones data in July generated considerably more attention than these other indices because of the subsequent ranking of the county system as No. 1 in the state by SchoolDigger.com.

4 comments:

Karen Miller Russell said...

Thank you for doing this nuanced and carefully explained analysis. It provides much more depth than the schools offered and is helpful in identifying both the positives and areas for more work.

bill zorn said...

how do we compare school systems without case cohort controls? is it the system, or the student characteristics which raise scores?



Dean said...

This is fantastic, thank you. -DS

Lee Becker said...

Bill,
You are correct, of course, that the comparisons of systems miss the differences among the systems you mention. That is one of the reasons I focused on the differences over time for Oconee County Schools.
Lee