So how did it happen? The fall (or death) of the PC dynasty. Rachel Notley and the impression she made on Albertans. Naheed Nenshi nudging Calgary voters to be open to alternatives. Prominent businessmen hectoring Edmontonians to stick with the status quo. Alberta’s changing demographics. Or how about mirrors, downturns, hope, fear, or math?

During the campaign, Jim Prentice dismissed the possibility of Albertans electing an NDP government, because, in his words, Alberta is not an NDP province. In the election aftermath, some suggested that, once you got past voter anger, the reason for the big NDP win was the province’s gradual shift to the left.

So does an NDP majority mean that Alberta is now an NDP province? Or is there some other explanation available in the voting data? Let’s start with a list of the NDP’s seats and compare the party’s share of the popular vote to the combined share of the PC and Wildrose parties.

The NDP had strong support in Edmonton and some surrounding communities as well as Lethbridge and Calgary Fort. But its remaining seats, in Calgary and other parts of central and northern Alberta, were won with a very slim margin in ridings where the majority of voters chose the parties on the right.

Support for the NDP clearly surged in this election, but where did the support come from? From a fired up base? From new voters? From voters switching from other parties? Or from habitual non-voters who decided to cast a ballot this time around?

To explore these questions, we built a mathematical model for the riding results data. We assumed that voters could be split into groups based on the party they’re most likely to support or align with. We also assumed that voters in each group don’t unconditionally vote for their aligned party; they could vote for other parties. Finally, we assumed that different ridings are made up of a mixture of these different voter groups.

We divided the voter population into 5 groups, 4 groups for voters aligned to each of the 4 major parties and 1 group for everyone else. Our objective function was the difference between the model’s predicted vote results and the actual vote results for each party in each riding. We defined 2 constraints: the probabilities assigned to parties in a group must sum to 1, and the group proportions in a riding must sum to 1. We then used the scipy optimization module to find the party preferences for each group and the group proportions for each riding that minimized the objective function.

The table below shows the party preferences (rows) for the 5 voting groups (columns) in the 2015 election. The numbers in each column represent the probability that the voters in the group will select the given political party. The percentage at the bottom of each column is the group’s proportion of the total voter population.

Voting groups in 2015:

The first group of voters preferred the Widrose party. Their probability of choosing the Wildrose was 43%, the PC party 14%, and the NDP 9%. They were 33% likely to not vote, and they made up 30% of the voting population.

The NDP group wasn’t the biggest, but it was the most likely to vote and the least likely to seek alternatives. The party also drew support from all the other groups, even the Wildrose. The NDP was even the choice of the non-voter group when it decided to vote.

So, a fired up base, support from other camps, and folks coming in from the sidelines plus other party supporters more likely to stay home and less likely to stay loyal — that’s the NDP story in 2015.

Voting groups in 2012:

The NDP group grew from 9% of the voter population in 2012 to 21% in 2015. This didn’t happen at the expense of the two conservative groups. Their total share remained about the same between the two elections (around 47%; voters migrated from the PC to the Wildrose group but stayed in the conservative fold).

The new NDP voters seemed to have come primarily from the non-voter group (which saw its share decline by 9%). This suggests the party courted and converted previous non-voters and increased its viability with other voter groups to expand its constituency. (Check out how this data-driven analysis lines up with what the party leaders had to say about their election strategy.)

Both these trends should concern conservatives. The NDP has gotten much better at political life, and Albertans like and trust the party’s leader. What should concern them more still is how homogeneous the PC and Wildrose groups really are (check the second and third choice parties for each group) and what this means for the unite the right movement.

In response to Jim Prentice, Rachel Notley said that “Albertans will decide what sort of province Alberta is.” Alberta may not be an NDP province yet (most Alberta voters are still in the Wildrose and PC camps), but you can sort of see it from here.

If you’re interested in the NHL and statistics (like us), you may have heard of something called the Corsi number. It’s one of many new methods to measure player performance being tracked by bloggers (like Behind the Net). Perhaps because it’s new, it’s not popular with traditionalists; witness Don Cherry’s rant against Corsi on Coach’s Corner last season. Despite misgivings over siding with guys who go with their gut, I agree with Cherry. The Corsi number is flawed. But it takes a better effort than Cherry’s to appreciate just why.

What is a Corsi number?

(Excerpted from The Cult of Hockey)

“Corsi is a plus/minus stat that measures shots directed at net. For example, if the Edmonton Oilers direct 30 shots at even strength at the Calgary Flames’ net in a game, while the Flames direct 45 shots at the Oilers’ net, the Oilers have a Corsi of -15 for the game. […] This team Corsi number is also broken down and applied to individual players. For example, if Sheldon Souray is out on the ice for 10 shots directed at the Flames’ net, while the Flames direct 15 shots at the Oilers’ net while Souray is on the ice, he’s said to have a Corsi plus/minus of -5.”

The Corsi number puts a premium on players who tend to spend most of their time in the offensive zone. If, when they are on the ice, there are more shots directed at the opposition net than their own, players will have a positive Corsi number. And the greater the shot difference (as measured by Corsi), the better the player.

Does shot difference make a difference?

But is it necessary for teams to outshoot opponents in order to outscore them? Not in every game, of course, but is it true on average? In the shots for and against stats for the 2009/10 season, the correlation between the shot difference and winning percentage is fairly high (0.63). On average, it appears winning teams tend to outshoot their opponents. This seems to support the idea that the Corsi number can be used as a proxy for performance.

But, if you take a closer look, the support falters. Teams that outshot their opponents won 580 games in the season. How many games did the outshot teams win? 588! More teams had a higher winning percentage when they were behind on shots (19) than when they were ahead (11). And a team’s winning percentage didn’t change by much whether it outshot or it was outshot by opponents — about 8% on average. If outshooting opponents is pivotal to winning games, we’d expect this difference to be wider. Instead, we see that a team’s ability to win games was fairly independent of its ability to outshoot opponents. Why?

The incentive to shoot

Teams shoot to score goals. And, depending on who’s ahead, each team’s incentive to score another goal (and hence take shots) is different. It’s always more important for the team that’s ahead to prevent a goal than to score a goal. Similarly, it’s always more important for the team that’s trailing to score a goal than prevent a goal. To put it in economic terms, the marginal value of a goal is higher for the team that’s behind than it is for the team that’s ahead, and the marginal cost of a goal allowed is higher for the team that’s ahead than it is for the team that’s behind.

The biggest problem with the Corsi number is that it assumes that incentives stay the same throughout a game. It penalizes players when they’re protecting a lead and have little to gain by venturing aggressively into the offensive zone. And it rewards players who are taking the space their opponents give them and taking shots from all angles without affecting the outcome of a game.

The value of a goal? It depends on the score.

Think of each NHL game as moving through a discrete set of states designated by the difference in goals from the home team’s perspective. The game starts tied (0); the home team scores (+1); the home team scores again (+2); the away team scores (+1); the home team scores again (+2); the away team scores again (+1); and the game ends 3-2 for the home team. The home team gets the win and two points, the away team gets the loss and no points.

That’s the story of one game. When you look at a whole season’s games this way, you can calculate the points that the home team earned on average when tied, ahead by a goal, ahead by two goals, trailing by a goal, and so on. You can, of course, do the same for the away team. If you actually did the calculations and plotted the results, you’d get a graph that looks pretty much like the one below.

Some goals are bigger than others

The graph tells a simple story: goals scored when the game is close are worth more than goals scored when the game’s out of reach. What’s interesting is how quickly goals that pad leads (or deficits) lose their value and how quickly a game gets out of reach. Outside a two-goal margin (+2 or -2), the value of goals scored or given up is relatively small. (Common player stats such as goals, assists, and plus/minus treat all goals equally.)

Goals that move a game to and from a tied state matter the most. For the seasons we analyzed, the tying goal was worth 0.51 points to the home team and 0.44 points to the away team, while the go-ahead goal was worth 0.37 points to the home team and 0.46 points to the away team. (This means a home team has more to lose by falling behind a goal than it has to gain by going ahead a goal.)

Of course, the expected value of a goal should take into account the time remaining on the clock as well as the difference in score. A tying goal in the last minute of a game is worth more than a tying goal in the first minute of the second period. But this is a fairly straightforward correction.

How can we use goal value to measure performance?

It’s simple: performance measures should focus on goals, not shots. And the plus/minus stat should be calculated using the expected value of a goal, because, as we’ve seen, some goals are bigger than others. (An alternative is to only count go-ahead and tying goals.) This way we adjust for context when measuring performance. And we make sure we get a measure of performance when the outcome of the game is still being contested, that is to say, when performance matters most.

As always, let’s start with the data: the PAT and diploma exam results released by Alberta Education. The chart below shows the trends for the two boards (along with the provincial averages) on the two sets of standardized exams. I’ve excluded the PAT results for 2013 as the CBE schools had very low participation in that year due to the June flood.

(Click on the image below to see a larger view.)

You can see in the top graph that, excluding ELA 9, grade 9 EPSB students posted PAT results that were typically 2 to 3 percentage points higher than CBE students (who were performing at about the provincial average). The bottom graph shows the performance gap reversed, with CBE students maintaining a significant and consistent edge (sometimes more than 4 percentage points) in the core diploma exam subjects and EPSB students now hovering around the provincial average.

To make things more concrete, let’s look at the cohort of grade 9 students that wrote the Math 9 PAT in 2011. EPSB students in this group achieved a 67.3 average and CBE students achieved a 64.7 average, a 2.6 percent difference in favour of the EPSB. When this cohort of students sat down to write the Math 30-1 diploma exam three years later in 2014, the CBE students posted a 67.9 average and the EPSB students managed a 64.2 average, a 3.7 percent difference now in favour of the CBE. Over 3 years of high school math in the two boards, the CBE students improved and the EPSB students got worse.

Now let’s look at the performance of individual schools in the two boards on the 2014 diploma exams. The table below shows the school’s overall diploma average (the average score for all diploma exams written in the six core academic subjects: Biology 30, Chemistry 30, English 30-1, Math 30-1, Physics 30, and Social Studies 30-1), its ranking among all high schools in the province, the change in rank from the previous year, the number of diploma exams written by students at the school, and the percentage change in the overall diploma average from 2013.

(Click on the image below to see a larger view. You can also check out our full list at Alberta High School Rankings or see how individual schools perform with our Alberta High School Dashboard.)

Old Scona is, as always, at the top of the list. Each year, hundreds of students compete for the roughly 120 grade 10 spots available at the academically focused Edmonton high school. The school ranks these applicants on their academic performance to date and other factors and offers admission to the top candidates. The school’s diploma results (an 85.1 average in 2014) are always impressive, but the Old Scona deck is clearly stacked. (In fact, assuming that the typical entrance average for Old Scona students is over 90, it’s puzzling that the school doesn’t do even better on the diploma exams than it does.)

Other public schools in Edmonton and Calgary play the hand they’re dealt; they’re called on to educate the broad population of students who arrive at their doors with diverse backgrounds, interests, motivations, and abilities. Here’s where the CBE schools shine, taking the remaining 9 spots in the top 10. Edmonton’s next best performing high school after Old Scona is Harry Ainlay which places just out of the top 10. Even if Harry Ainlay absorbed Old Scona and all its students, it would still place behind the top 3 CBE schools (Sir Winston Churchill, Henry Wise Wood , and Western Canada) on overall diploma exam average.

Does the diploma exam performance gap between the two boards really matter? Differences in marks matter wherever marks matter. Marks matter if you’re a student who’s planning to go to university. Society pounds home that post-secondary education is the right choice: higher education equals higher earnings, higher growth in earnings, greater assets and savings, even higher life satisfaction. There may be many roads to a successful, productive, and happy life, but the widest and surest roads pass through post-secondary institutions. But you need to get in, and to get in, you need good, bordering on great, marks, because these institutions use marks as a proxy for academic potential and, because demand for spots at these institutions exceeds supply, the entrance process is highly competitive.

In Alberta, final marks in grade 12 are an average of school awarded marks and diploma exam marks. The diploma exams are the last tests that students take in the Alberta K-12 system. They’re high stakes, standardized tests designed to be statistically fair, reliable, and valid assessments of course curriculum. Because all students taking 30-level courses write the same (or statistically equivalent) final exams, diplomas make it possible to compare students, schools, and school authorities over time.

You can argue the limitations of standardized tests in measuring student learning or argue against education’s fixation on grades and test scores. You can also put forward a good argument that standardized tests shouldn’t be used to measure educational quality.

But learning can’t simply be asserted; it has to be demonstrated. And, in all levels of our educational system, students demonstrate learning by writing tests and doing well on those tests. When two comparable populations of students (public high school students in Edmonton and Calgary) write the same statistically designed tests and one group consistently achieves significantly higher results, I think we’re justified in looking for reasons for these differences, say their level of preparation or their learning experiences leading up to the assessments.

In our public school system, we try for the best possible educational outcomes for all students. Students should have similar academic opportunities and support regardless of where they choose to go to school. This is the ideal. But something seems to be off in Edmonton public high schools, and it’s leaving EPSB students at a disadvantage in relation to their CBE peers.

I don’t have an answer to the question I posed in the title of this post, but I can offer some hypotheses.

1. The differences in diploma results between the two boards are small and they’re due to chance. This hypothesis can be rejected using the comparison of means test. It shows that the differences are statistically significant. What’s more, Calgary public students outperform Edmonton public students year in, year out, again suggesting systematic rather than chance factors at play.

2. CBE high schools have lower participation rates in the core academic subjects. This could mean that CBE schools are more effective at steering students who may perform poorly in the core diploma subjects into alternatives (e.g, from Math 30-1 to Math 30-2). If these students are being consistently factored out, CBE averages would tend to rise.

The participation rate data published by Alberta Education doesn’t support this hypothesis. There are differences in participation rates between the two boards across subjects (e.g., in 2014, CBE had a higher participation rate for Chemistry 30, English 30-1, and Math 30-1 while EPSB had a higher participation rate for Biology 30, Physics 30, and Social Studies 30-1), but the CBE rates aren’t consistently lower.

3. Calgary has many private and charter schools. Edmonton doesn’t. According to Jason Clemens of the Fraser Institute, “when parents are empowered to choose their schools for their kids and we force schools to compete with one another, we get better school performance.” The diploma exam results provide support to this conservative mantra, at least when performance is viewed narrowly in terms of standardized tests. Calgary public high schools may pay closer attention to diploma exam results because their private and charter school neighbours continue to draw parents’ attention to academic performance. CBE schools that are “forced to compete” have managed to achieve better academic performance than EPSB schools that face no similar competition in their region.

4. Calgary is a more affluent city and everyone knows that results on standardized tests are highly correlated with household income. I looked at this relationship in my critique of the Fraser Institute’s high school rankings. Using Fraser’s data, I found a positive correlation between diploma exam results and household income for Calgary and Edmonton high schools. I also found that nearly 70% of the variation in diploma exam marks was left unexplained by household income. Maybe affluence is enough to tilt the diploma results in Calgary’s favour, but then you have to wonder why the affluence effect doesn’t play out similarly in the PAT results.

5. Calgary is a white collar town. Edmonton is a blue collar town. Folks in Calgary are more likely to have a university degree than folks in Edmonton. This fact comes straight from Statistics Canada’s 2011 National Household Survey. The proportion of Calgary adults aged 25 to 64 with a university degree is around 35%. In Edmonton, the proportion is around 25%. Students whose parents hold a degree are much more likely to complete university. We can assume the academic expectations and academic support in these households are also likely higher. So the difference in marks between the two boards may reflect a difference in household environment rather than school environment. As to why this household advantage isn’t reflected in the PAT results, I’m not certain. Perhaps university-educated parents increase their academic expectations and focus when they sense it really matters — sort of like cramming for finals.

6. EPSB high schools appear to have a lower 3-year completion rate than CBE high schools: a greater proportion of EPSB diploma exams are written by students who are upgrading or completing high school outside the traditional 3-year route. These students tend to score below the provincial average on the diploma exams. What’s more, they’re upgrading to improve an earlier diploma mark, which we can safely assume was lower than what they needed to move on to the next stage in their lives. This population of students therefore affects a district’s diploma exam results twice, bringing down the overall average each time.

EPSB serves these students through Metro Continuing Education and Centre High (as well as other smaller alternatives) while CBE serves these students through Chinook Learning Services. The number of diploma exams written at Centre High (described on its website as a dynamic high school for fourth and fifth year students) has risen sharply since 2008 while the number of exams written at Chinook Learning Services has risen more modestly. CBE high schools and their students seem more focused on completing high school (successfully) in 3 years and this focus has resulted in better diploma exam results for the district.

Here are the links to the EPSB high school completion rates and the CBE high school completion rates. Note that EPSB doesn’t report its 3-year completion rate but CBE does. For comparisons sake, in 2012-13, the EPSB 5-year completion rate was 77.3 and the CBE 5-year completion rate was 80.7 (with a 3-year completion rate of 74.0).

7. The Old Scona Effect. EPSB high schools do face competition, not from private or charter schools (as with the CBE) but from one of their own.

Edmonton junior high students compete for the limited (about 120) grade 10 spots available at this small academic high school that consistently finishes at the top of any rankings derived from diploma exam results. The school receives far more applicants than it can accept. Applicants who make the cut based on the school’s scoring criteria — a weighted mix of academic achievement in junior high, a written test of cognitive ability, and letters of recommendation from junior high principals — enter a crucible of high academic expectations and competition for the next 3 years. Applicants who don’t make the cut enrol in other EPSB high schools — where it appears expectations are lowered.

Now these other EPSB high schools don’t measure themselves academically against Old Scona; the school is an outlier. They measure themselves against their peers, the schools that serve the general student population. But Old Scona does affect the EPSB ecosystem in an important way. When it enrols the city’s top academic achievers, it removes them (and their parents) as an influence on the culture of other Edmonton high schools, and it removes the positive effect of this high-achieving sub-population from their diploma exam results. When the other Edmonton high schools compare themselves to each other academically and set benchmarks, they’re therefore choosing as normal a lower range of academic achievement than they otherwise would if Old Scona didn’t exist or if it wasn’t allowed to pick the cream of the crop.

Why, with so much focus on developing number sense in elementary age students, are we seeing such mixed results with students graduating from our school system?

It may come down to how arithmetic is taught in today’s classrooms.

New math vs. old math

Old math emphasized memorization (think times tables) and constant practice (think worksheets) to develop number skills. New math holds that “[a] true sense of number goes well beyond the skills of simply counting, memorizing facts and the situational rote use of algorithms.”

New math sees rote learning as the worst type of learning, bordering on not learning at all. It’s dismissed as drill and kill, a sure way to choke off interest and motivation. With rote learning, students end up being able to do things they don’t understand and can’t easily explain. Any knowledge they gain is brittle, incoherent, and easily forgotten.

For new math advocates, memorizing standard algorithms and basic math facts is a one size fits all approach that doesn’t address the “diverse needs and learning styles of today’s students”. They instead favour more flexible approaches such as mental math (“alternative algorithms for calculating without pencil and paper”) and estimation (“techniques for determining approximate values using benchmarks and referents”) to develop a “deep understanding” of numbers.

3 good things about old math

Old-fashioned math has three virtues when it comes to preparing younger students for the study of higher level mathematics: 1) it emphasizes that problems in arithmetic have a right answer; 2) it defines a simple set of rules that, properly understood and followed, will always produce the right answer; and 3) it shows students the pleasure in not only getting the right answer but in being confident that the answer can’t be anything else.

1. Math problems have a right answer. 7 * 4 is always 28 no matter how you arrive at the result. You can estimate the answer to be 30, but 30 isn’t the right answer. Old-fashioned math draws a student’s attention to the fact that right and wrong in K-12 math isn’t controversial. You don’t debate a problem like 7 * 4. You solve it.

Understanding and fluency comes from tackling and solving problems, making and correcting errors. Whereas new math is an education in critical, open-ended thinking (questioning and often rejecting rules), old math is an education in correct thinking (understanding and following rules). Thinking correctly comes before thinking independently, creatively, or even critically.

2. Math is a set of rules. Taken from the Alberta curriculum’s beliefs about students and mathematics and learning: “Students learn by attaching meaning to what they do, and they need to construct their own meaning of mathematics. They must realize that it is acceptable to solve problems in a variety of ways and that a variety of solutions may be acceptable.” The emphasis is on developing “personal strategies” (which leads to deep understanding) rather than memorizing and applying fixed rules (which presumably leads to shallow and incomplete understanding).

While it’s true that mental math offers a variety of ways to solve arithmetic problems, it doesn’t really do away with memorization. In the case of multiplication, it replaces one general purpose algorithm (which always works) with many limited purpose algorithms (which work in specific cases). Mental math thus imposes an additional cognitive burden on the student attempting to perform arithmetic: she not only has to remember each mental math algorithm but she also has to determine which one to apply for any given problem.

New math has the admirable goal of helping students attach meaning to mathematical thinking. But it overlooks the fact that while we can follow rules without knowing what they mean, we can also discover meaning by following the rules and observing what happens. Keith Devlin talks about this in his excellent post How Do We Learn Math?:

3. Math is funner when you “get” it. It’s easy to fall behind in grade school math. Each new concept builds on an earlier concept, and concepts tend to become more abstract and sophisticated. If you haven’t mastered arithmetic with numbers, you’ll likely be baffled by arithmetic involving variables. Many students resort to blindly doing what they’re instructed to do. They can solve familiar problems, but they struggle when the same concepts are presented in new contexts. New math advocates see this as an easy trap for students to fall into, easily missed because high marks in solving familiar problems suggest that they’re doing well.

So why does basic math understanding elude capable and motivated students? The typical response is that it doesn’t, and that the explanation lies in some failing on the students’ part. They’re not motivated or they don’t have the talent for math. Another suggestion is that the failing lies with schools. Many teachers don’t understand what they’re teaching, and they confuse students with their lessons and explanations.

I don’t claim to have an answer here. But I’ve observed what students do once, having solved many familiar problems of a certain type, they solve a related problem that’s a little different and they just know they got it right. They’ve understood what they’ve been asked to do, they’ve seen the components of the problem, they’ve taken the right approach to solving it, and they’ve carried out the procedure correctly. And, once they’ve done all that… they smile.

Growing up liking math

Timothy Gowers, a Cambridge mathematician, writes that “any child who is given one-to-one tuition in mathematics from an early age by a good and enthusiastic teacher will grow up liking it.” Deep understanding, if that means thinking mathematically, takes time (take a look at the Wikipedia page on natural numbers and try and get your bearings). As a first step, it’s important to know and apply mathematical rules correctly. Here’s Gowers’ explanation of the benefits of this abstract approach.

The Fraser rankings, released annually and regularly reported by the media, have largely shaped public perception of school performance. The authors have stated they want to make it possible for parents and educators to easily compare and monitor the academic performance of Alberta high schools. They could have done this by creating a better interface for Alberta diploma exam data. Comparisons made with that data would be easy to understand and evaluate. Instead, they came up with a complicated (and arbitrary) scoring formula to rate and rank schools that essentially shifted the conversation from how schools are doing academically to how schools are doing in the Fraser rankings.

The rankings are arbitrary because the scoring formula is arbitrary. There is no theoretical reason for using the indicators the authors have selected (like gender gap or courses taken per student) or for combining them in the complicated manner they’ve chosen. (We tried to recreate the ratings from the 2012 Fraser report by following their methodology, but we had limited success.) A simpler alternative, such as comparing schools on their average diploma exam mark, would make much more sense.

This is why we collect data on a group of objects after all. We want to use it to make comparisons, track progress, and study relationships. Introducing a derived measure like a rating needlessly complicates things. Ratings take us away from the patterns and relationships in the original achievement data.

But let’s say that you agree that delayed advancement rate (defined by the authors as the extent to which schools keep students progressing in a timely manner toward the completion of their diploma program) is a relevant indicator of academic achievement that’s under the school’s control and that it can be accurately calculated using the authors’ convoluted algorithm.

How, then, do you go about weighing delayed advancement rate against average diploma exam mark? Given that you’re using average diploma exam mark as just another indicator, there’s no external standard of academic performance (ground truth) to test your decisions against or to help you tune the structure and parameters of your model.

So what do you do? Like the authors, you probably just make it up. A little intuition here, some personal judgement there, and, pretty soon, you’ve made up the entire formula — which leads us to the second problem.

The Fraser scoring formula, with its mix of academic indicators and ad hoc weightings, produces a different set of rankings than those produced from the raw exam results alone. We compared the rankings of Alberta high schools based on their average diploma exam mark to the school rankings in the Fraser Institute’s 2012 report. The chart below shows the results, and the differences are illuminating.

How to read the chart: The circles represent Alberta high schools. The x-axis measures Gr. 12 enrollment. The y-axis measures the difference between the the average diploma exam mark ranking and the Fraser ranking. Schools above the zero line did better in the Fraser rankings than on the average diploma exam mark ranking; schools below the zero line did worse. For example, Rundle College Academy ranked 45th in the Fraser rankings and 95th in the average diploma exam mark ranking. The school did better in the Fraser rankings, and its value on the y-axis is 95 – 45 = 50. The color of the circles represents the school authority, the size of the circles represent Gr. 12 enrollment.

Differences between the Fraser Institute rankings and the average diploma exam mark rankings:

• The Fraser rankings favour small schools over big schools. Small schools, as a group, rank higher in the Fraser rankings than they do in the average diploma exam rankings. Big schools, as a group, rank lower.
• The Fraser rankings punish big public schools. Big public schools are more likely to place lower in the Fraser rankings than in the average diploma exam rankings. There are more large green circles below the zero line than above.
• The Fraser rankings favour separate schools. Separate schools, and big separate schools in particular, fare better in the Fraser rankings than in the average diploma exam rankings. There are more large red circles above the zero line than below.
• Private schools, as a group, do better in the Fraser rankings. There aren’t that many private high schools in Alberta, but, as a group, they do better in the Fraser rankings than the average diploma exam rankings.
• Public schools, as a group, do worse in the Fraser rankings. Although small public schools generally benefit, when looked at overall, public schools come out worse than they would in the average diploma exam rankings.

When a conservative think tank (one that advocates for choice, competition, and markets) ranks schools in a way that’s systematically biased against public schools, you can be forgiven for seeing the results reflecting agenda more than truth. As such, the information the Fraser Institute provides parents and educators is not as useful as it could be and certainly not as useful as it’s made out to be.

Sticking to Diploma Exam Data

There are many interesting insights we can gain into the educational experience of Alberta high school students by looking closely at Alberta Education’s diploma exam results, a rich and detailed dataset. A case can be made, for example, that competition from private schools has helped Calgary public schools outperform public schools in other parts of the province, particularly Edmonton. A case can also be made that many large public schools are delivering quality education to a diverse population of students in less than ideal circumstances.

Alberta Education publishes the data but doesn’t provide an interface for comparing school performance. The Fraser Institute has made comparisons possible but only through a biased index that appears more agenda-driven than useful. So what can Eight Leaves do to help parents and educators easily compare and monitor the performance of Alberta high schools?

We took Alberta Education’s diploma exam data for the 2011-2012 year, and we built a visualization using Tableau. You can find it here: Comparing Alberta High Schools. We think it’s a much better use of a very interesting dataset. Let us know what you think.

Update: We also created a ranking of Alberta high schools based solely on Alberta diploma exam results and a dashboard to track how Alberta high schools have performed in different academic subjects over time.

Related Links

• The Real Problem with the Fraser High School Rankings – Part 1.
• Comparing Alberta High Schools: A visualization of Alberta diploma exam results.

The Fraser High School Rankings

Every year, the Fraser Institute produces a ranking of secondary schools in Alberta called the School Report Cards. According to the authors, Peter Cowley, Stephen Easton, and Michael Thomas, these report cards collect “a variety of relevant, objective indicators of school performance into one, easily accessible public document so that anyone can analyze and compare the performance of individual schools.” Parents can use the rankings to evaluate schools for their children. Administrators can track rankings over time to improve performance at individual schools and across the system.

The authors contend that they’ve developed an accurate measure of school performance that can be used to evaluate and improve the educational system. Not everyone agrees. The rankings’ many critics vigorously attack the philosophy (Can we measure school performance with standardized tests? Should we be ranking schools at all?), the politics (The rankings are produced by a right-wing think tank with an anti-public school agenda.), and the methodology (Aside from test scores, none of the other indicators are accurate proxies for achievement.) behind the work.

As a data agency, we do a fair bit of work in developing and evaluating KPIs for public and private sector clients, hence our interest in how people go about measuring performance in different settings. I thought it might be useful to take a data-driven look at the the report cards, focusing narrowly on how the authors calculate a school’s rating and the effect, intended or unintended, of their choices on the actual rankings they produce.

How are the Fraser Ratings calculated?

The information here is taken directly from the Institute’s 2012 Report which describes the authors’ methodology in detail. Overall school ratings are meant to reflect academic performance and are calculated using diploma exam results released by Alberta Education and other public sources of data. The authors base a school’s overall rating on its performance on eight indicators which are categorized into three groups.

Three indicators of effective teaching: (1) average diploma examination mark; (2) percentage of diploma examinations failed; (3) difference between the school mark and examination mark in diploma courses;

An indication of consistency in teaching and assessment (gender gap): (4) difference between male and female students in the average value of their exam marks in English 30-1; (5) difference between male and female students in the average value of their exam marks in Pure Mathematics 30;

Three indicators of practical, well-informed counselling: (6) diploma courses taken per student; (7) diploma completion rate (the rate at which first-time grade 12 students receive a diploma in the school year); (8) delayed advancement rate (the rate at which students fail to graduate or advance a grade in the school year);

To calculate an overall rating, the authors first standardize the results for each of the indicators. These standardized results are then weighted and combined to produce an overall standardized score. This overall score is then converted into a rating on a 10-point scale.

Indicator weightings: average exam mark—20%, percentage of exams failed—20%, school vs exam mark—10%, English 30 gender gap—5%, Pure Math 30 gender gap—5%, courses taken per student—20%, diploma completion rate—10%, and delayed advancement rate—10%.

In instances when Gender gap could not be calculated, school vs exam mark difference was weighted at 20%. When delayed advancement rate could not be calculated, diploma completion rate was weighted at 20%.

Some common criticisms of the Fraser Ratings

1. The ratings are not indicators of academic achievement. Test scores have a research basis to merit association with achievement. The other indicators do not. But the weighting formula assigns a mere 20% to the average diploma exam mark. With test scores contributing so little, the ratings clearly don’t reflect what they’re meant to: academic achievement.

What the data says: This one’s straightforward to counter. The correlation between the average diploma mark and the overall rating is a very healthy 0.90. You can see the strength of the relationship in the graph below (each point represents one of 276 Alberta high schools).

Why is the correlation high if the weighting is low? I’ll come back to this when I look at the covariance of the indicators. For now, if we accept that exam marks are a good measure of academic achievement, then the Fraser ratings, with their high correlation with average diploma marks, should be considered a good measure of academic achievement.

2. Everyone knows that schools which rank the highest are almost always those from higher income neighbourhoods. Upper income neighbourhood schools have far more advantages and resources than schools in lower income neighbourhoods and will obviously do better academically. What’s the point in comparing them?

What the data says: Can most of the variation in the academic performance of schools be attributed to the economic status of a school’s students? Does economic status determine academic performance?

To examine this issue, the authors combined census data from Statistics Canada with enrollment data from Alberta Education to estimate the average parental income of the student body attending each school. (This is a common and useful technique: marketers often use census data in this manner to study the demographics of their customers.)

Assuming the data is correct, I’ve graphed average exam mark against parental income in the graph below. Parental income is weakly correlated with average exam mark: correlation coefficients are 0.56 for Calgary and Edmonton schools, 0.24 for schools outside the cities, and 0.44 overall.

What can we learn? Although it’s true that higher income schools tend to have higher exam results than lower income schools, there is still a great deal of variation in academic performance that isn’t explained by economic status. Schools matter, and, given similar socioeconomic contexts, effective schools produce better academic results than their peers.

3. The factors that make up the ratings are clearly inter-related and reinforce each other. Larry Booi, former president of the Alberta Teacher’s Association, criticized an earlier version of the ratings that included only five indicators: “A school that scores high on the first factor (diploma exam marks) will almost certainly score high on the second factor (that is, low diploma exam failure rates)… Thus, while the rating is purportedly based on five factors, in reality it is based on five aspects of one factor—diploma examination courses.”

What the data says: The relationships between indicator variables can be seen in the correlation matrix below (AEM: average exam mark; CPS: courses taken per student; DAR: delayed advancement rate; DCR: diploma completion rate; ENG: English 30 gender gap; %FAIL: percentage of exams failed; MATH: Pure Math 30 gender gap; SCH: school vs. exam mark):

Average exam mark is correlated to all indicators except the two gender gap variables. This explains why we observe a strong correlation between the Fraser rating and exam marks despite the low weighting of exam marks in the scoring formula. We can examine the covariance structure more closely by performing and interpreting a principal components analysis (PCA) on the indicator data. (This is another very useful statistical technique for exploring and understanding relationships among attributes in datasets.) The graph of the first two principal components is shown below:

The first principal component (the x-axis) measures academic performance. Good indicators of academic achievement (average exam mark, diploma completion rate, and courses taken per student) have a positive projection onto the first component while bad indicators (delayed advancement rate, percentage of exams failed, and school vs. exam mark) have a negative projection. Academic achievement is the most important factor but not, as Booi suggested, the only factor we have to consider: the first principal component accounts for just 47% of the variance in the original data.

The second principal component (the y-axis) is interesting. Three indicators (diploma completion rate, courses taken per student, and school vs. exam mark) have a positive projection onto the second component, and they account for most of the loading. This means that the second component highlights schools with high diploma completion rates (a good thing) but also high discrepancies between school awarded marks and diploma marks (not a good thing). Let’s call it the grade inflation component.

The third component (which isn’t shown) has equally high loading from the Pure Math and English 30 gender gap indicators. All other indicators have a near-zero projection onto this component. This shows that gender gap is largely independent of the other academic achievement variables, a finding that helps explain a quirk of the ratings that I’ll get to a little later.

To sum up, PCA shows that the first component is academic achievement, the second component is grade inflation, and the third component is gender gap. Taken together, the first three components account for over 80% of the variance in the Alberta high school data set.

4. The rankings dramatically distort relatively small differences in exam marks, with the result that some schools appear to perform far worse than they actually do. Schools that are separated by over 200 positions in the rankings have less than a 10% difference in their average exam marks. Most schools are separated by a few percentage points. Why place such a big emphasis on such small differences?

What the data says: The distribution of average diploma exam marks has a mean of 63.8 and a standard deviation of 5.4. It ranges from a low of 37.0 (Calgary’s International School of Excellence – a private school) to a high of 85.6 (Edmonton’s Old Scona Academic – a public school). A histogram of the exam scores (with a normal distribution superimposed) is shown below.

Do small differences in average diploma scores between schools matter? It depends on the size of the difference, the number of students taking the diploma exams in each school, and the variance of the test scores in each school. These three factors can be used to perform a difference of means test, a statistical method for comparing means to see if one is significantly different from another. The test allows us to determine when a difference in means is unlikely to be explained by chance and thus points to a real, underlying distinction in the performance of the schools.

A difference of 10% in the middle of the distribution, it turns out, is highly significant, even when we’re dealing with the smaller schools (in the range of 30 diploma students). For larger schools (300 or more diploma students), a difference of 3% in average diploma scores is often significant.

5. Schools operate with different missions and in different contexts. They have vast differences in student populations. Some serve special needs students, some have a high proportion of ESL students, some are in small, rural communities, some are in dense, urban neighbourhoods, some select students based on high academic achievement, some serve students with no plans for higher education, some serve the general student population, some are private and exclusive. Yet the Fraser Institute ignores these differences and ranks all schools on the same scale.

What the data says: We work with national retailers operating in multiple markets. To evaluate stores, we often cluster them in terms of retail traffic, urban/rural location, and surrounding demographics. We then compare the performance of stores within each cluster. This allows management to set realistic benchmarks and create suitable programs for each individual store.

Schools, like stores, operate within contexts. Using k-means clustering on school characteristics such as enrollment, location, and parental income, we can identify distinct school segments (e.g., wealthy private schools, large urban schools, small rural schools, etc.) in the Fraser data. Some schools also have a very specific focus: Old Scona selects kids based on high academic achievement, Braemar is a school for pregnant and parenting teens, Jack James prepares students for a transition to the workforce, etc.

Using clustering techniques and an understanding of very specific school missions to segment schools, and then replacing a global ranking with a ranking in each school segment would be a more meaningful input for parents and administrators looking to evaluate schools.

(The % of special needs and ESL students — factors that are often cited by Fraser critics as important drivers of school achievement — are surprisingly uncorrelated to the average diploma exam mark in the Fraser data.)

6. The gender gap variables systematically bias the rankings to favour single sex, i.e., private, schools. This argument was put forward by Michael Simmonds in his 2012 PhD thesis, Interpreting the Fraser Institute Ranking of Secondary Schools in British Columbia. Simmonds noted that after the gender gap variables were introduced, BC public schools fared significantly worse in the rankings, with a 92% decline in the number of public schools scoring between 9.0-10.0, the top end of the scale. By contrast, the decline in top ranking private schools (more of which are boys or girls only) was only 32%. This resulted in a redistribution of schools at the top end, with public schools representing just 10% of the “best” with the new scoring formula, as opposed to 46% prior to the introduction of the gender gap variables.

What the data says: Gender gap data is used as an input to a school’s rating if it’s available. If it isn’t available, School vs. exam mark difference is weighted higher. As we’ve already seen, school vs. exam mark difference is correlated with the other academic achievement indicators in the scoring formula, but the two gender gap variables are not.

If you subtract a negatively correlated random variable from another random variable, you increase the variance of the difference more than if you subtract an uncorrelated random variable: high values go higher, low values go lower. High achieving schools without gender gap data will get a boost in their academic rating (compared to high achieving schools with gender gap data) due to the increased weight of the (negatively correlated) school vs. exam mark difference indicator to substitute for the missing (uncorrelated) gender gap variables.

(It’s as though, in a math course, two high-achieving students were asked to take one more test to determine their final score. The first student wrote a 20-question math test; the second student flipped a fair coin 20 times and counted the number of heads. Who ends up with the higher mark?)

This is one explanation for the improved performance of single-gender private schools in the BC rankings. It also points out the unexpected consequences that can flow from the use of complex scoring formulas.

Summing Up

• The Fraser ratings are highly correlated with academic achievement.
• Given similar socioeconomic contexts, some schools produce better academic results than others.
• The data used in the Fraser ratings measures three components: (1) academic achievement; (2) grade inflation; (3) gender gap.
• Relatively small differences in academic achievement between schools can be statistically significant.
• There are distinct clusters of schools; comparisons within clusters would be more meaningful than the global comparison used by the Fraser Institute.
• The gender gap variables systematically bias the ratings to favour single-gender schools and schools with no gender data.

Related Links

• The Real Problem with the Fraser Institute High School Rankings – Part 2.
• Comparing Alberta High Schools: A visualization of Alberta diploma exam results.

How do you break the cycle? An obvious strategy is to convert new single-ticket buyers into subscribers. Subscribers account for the bulk of a performing arts organization’s earned and contributed revenue. They’re likely to purchase additional tickets, renew, advocate, donate, and, if properly engaged, predictably increase their giving level over the years.

When is the right time to ask single ticket buyers to become subscribers?

One approach, recommended by Oliver Wyman consultants, is to take things one step at a time. Don’t telemarket first-time buyers and ask them to subscribe. Too pushy. Instead, tempt them back with a killer promotional offer (half-price tickets or buy one get one) tailored to their preferences and interests. Sell another ticket or two before asking for a commitment.

Reasonable advice if you’re trying to fill seats. But if you’re trying to grow your subscriber base, I can suggest three reasons why you should get right to the point with first-timers instead of taking things “one step at a time”.

1. First impressions are lasting impressions. From one of our most popular blog posts: The Loyalty Ladder: A Sideways Look. “[W]e often commit on the first visit. This first commitment is instinctive and emotional. It springs from our basic sense and appreciation of something done right. Customers don’t scrutinize the relationship every step of the way. They take leaps. What they’re doing in subsequent visits is continuing to explore and revisit an original experience they’ve already committed to, not finding reasons to commit at a later date.” When you ask first-time single-ticket buyers (who have had a wonderful experience) to subscribe, you’re not asking them to commit (the performance has taken care of that); you’re showing them what form the commitment they already feel can take.

2. They’re just as likely to subscribe now as later. Part of the logic of taking things one step at a time is that you’re more likely to get a yes later on than right away. In the attendance data that we’ve seen, this logic is flawed on two fronts. One, the same proportion of single ticket buyers who have attended one performance purchase a first subscription as do single ticket buyers who have attended from two to six performances. And, two, fewer customers come back a second, third, or fourth time, so, if you wait, you’ll be pitching subscriptions to ever smaller groups.

3. Offering the right option encourages the right behaviour. You want single ticket buyers to become subscribers. But when you offer them another performance instead of a subscription, you’re encouraging them to experience your shows and your venue as single ticket buyers. The habit they may very well develop is to attend once or twice in future seasons, and it’s a difficult habit to break. Repeat single ticket buyers continue to enjoy their visits. They feel good about supporting the arts. But they’re not quite in the fold. They’re a missed opportunity.

The right time to ask single ticket buyers to become subscribers?

Very soon after they attend their first performance. Give first-timers a killer price incentive on their first subscription to mitigate the risk. Help them see themselves as subscribers: where they’ll sit, their privileges, the upcoming shows. Describe the importance of subscribers to your organization. Use the enthusiasm and excitement they felt in the first performance to help them take the leap. And ask for the sale.

If they say no, by all means, follow up with a killer single ticket offer (and continue to ask them to subscribe at the start of each new season). That’s Plan B. Plan A is to always ask single ticket buyers to subscribe very soon after their first performance.

Extracting meaning from data and communicating it effectively is a specialist’s game. It requires a unique mix of skills, talent, and judgement. The IT department warehouses data and creates reports. Business analysts drill down and uncover trends. This is good work, but it leads to better questions: What should I look at and why? What does it mean? What’s driving it? Can I predict when it’ll happen again?

I call Eight Leaves a data agency to sum up what we do and to suggest our role in your supplier ecosystem. An ad agency works on creating and delivering advertising. A data agency works on uncovering and interpreting meaningful patterns in large data sets. An ad agency works long-term to develop a client’s image and shape the perception of its target customers. A data agency works long-term to improve a client’s understanding of customers and its decision-making around strategic concerns. An ad agency is creative. A data agency is objective.

Advertising has the advantage of being in the open. It’s meant to be seen. To attract prospective clients, an ad agency can simply point to the good work it’s done over the years. We don’t have this advantage. Due to the proprietary nature of data (and the competitive edge that analytics provides), our good work stays locked inside our clients’ walls. To attract new clients in new industries, we’re forced to describe what we do in abstract rather than concrete terms. This puts us in a bind: we can’t show you what we do until we actually start doing it.

There are some nuggets from studies we’ve done with public data sets which I’ll begin to share in future posts. For now, I’ll briefly describe the services we provide to our long-term clients and leave it to you to decide whether you, in fact, need a data agency.

What Does a Data Agency Do?

We organize and clean data. Not very glamorous but necessary work. The data sets we analyze tend to be messy. Records are duplicated. Values are missing or entered incorrectly. The data contains outliers which are sometimes anomalous and sometimes just unusual. The data set is at times a distorted sample of the population under study. These potential problems affect the quality of subsequent analysis. Because insights drawn from contaminated data are of little practical value (even misleading), we put in considerable effort to keep client data as clean as possible.

We help clients understand what their data is telling them. During meetings with our clients, we’ll hear opinions fly back and forth on why a store over performs, why customers leave, or why business units don’t collaborate. Someone at some point will suggest “Let’s see what the data has to say.” It’s a favorite phrase for long-term clients. They no longer see data as a sort of eye-rolling chaos or formless commodity. They instead rely on it to help them understand what’s really driving the events and behaviors that affect their world. Data grounds their thinking and decisions in reality.

We predict what happens next. Not always, but often enough, the past predicts the future. A customer that bought two suits every year for the past ten years is likely to buy two suits this year. A customer that doesn’t return in a fixed window of time from her most recent visit is likely lost. Patterns in customer data sets lie hidden in a background of randomness. Data specialists, like Eight Leaves, attempt to discover these patterns and test whether they hold up over time and under varying conditions. We’re supported in the effort by an arsenal of algorithms and computational tools and techniques. We bring to the task a mix of critical thinking, imagination, and care — so that our predictive models help our clients choose the best course of action, in circumstances big and small.

We communicate our findings clearly. One of our first presentations to a prospective client didn’t go as we expected. We felt our analysis was impeccable and that our findings were game-changing. But the reception to our many detailed slides was muted. Those present paid polite attention, asked polite questions, and politely showed us the door. We didn’t fail in the substance of our work; we failed to communicate the substance effectively. Happily, we learned our lesson — then we learned more lessons from information design, advertising, and cognitive science. We now take as much pain in crafting the message as we do in piecing it together, so that clients can see what we see and why it matters.

Eight Leaves Media has been developing sales and marketing programs for Origami Accounting for the past year. It’s been an exciting and successful project. For the holiday season, we created Origami gift boxes for the company to deliver to its roster of first-year clients. We think it’s a nice customer appreciation touch, and they’ve been very well received. Check out the Origami Blog to see the pictures.

• Groupon Works for Business — Groupon
• What to Expect When Taking the Group-Buying Plunge — The Globe and Mail
• Groupon Saved My Business — NBC
• Groupon’s Success Disaster — Redfin Blog
• Doing the Math on a Groupon Deal — New York Times
• To Groupon or Not to Groupon: The Profitability of Deep Discounts — Harvard Business School
• Is Groupon Good for Retailers — Harvard Business School
• How Effective Are Groupon Promotions for Businesses — Rice University
• Groupon in Retrospect — Posies Bakery and Cafe
• Burned by Daily-Deal Craze, Small Businesses Get Savvy — Wall Street Journal
• Wise for Some Restaurants, Coupons Are a Drain at Others — New York Times

Another very important consideration is how to measure the success of a daily deal. Measurement is close to our hearts, and we have a post in the works to help you sort things out.