Consensus Ranking

The computation of flight rankings proceeds by finding the overall score for each pilot from each judge. It then orders the pilots by their scores. The ordering by score produces a ranking for each pilot from each judge. The CR algorithm takes the individual judge's rankings as input and computes a ranking that satisfies the majority of the judges as closely as possible.

Here is the consensus rank for the first flight of the Sportsman category at the Bill Briski Acrofest, held on February 21, 2004 at Casa Grande, Arizona. The judges appear in columns across the top. The pilots appear in the rows of the table, ordered by consensus rank. The number in the column is the rank given the pilot by the judge. Judges one and four ranked Melissa Andrzejewski first. Judges two and three ranked John Van Hooten first.

It's easy to verify, going down the table, that a majority of judges ranks each pilot ahead of the pilots in the following rows.

Category rankings

The category result pages show two methods of combining consensus ranks from the flights into an overall winner for the category. The two methods are the "Borda winner" and the overall consensus winner.

Borda winner

The "Borda" method simply adds the ranks from each flight. In the Sportsman example, Melissa Andrzejewski ranked second in the first flight (according to CR) and first in the second flight. Her Borda total is three. The next higher competitor, Leonard Rulason ranked third in the first flight and second in the second flight. His Borda total is five.

Here is the Borda rank for the Sportsman category at the Bill Briski Acrofest, held on February 21, 2004 at Casa Grande, Arizona. Note that the Borda rank declares a tie for third place.

A second method for combining flight results into an overall category result is to combine the judges' rankings from all flights and compute the consensus. The preference tables in the category results show the result of combining the rankings from all flights. The example below is also from the Sportsman category at the Bill Briski Acrofest.

Read the preference table by comparing the number in (row, column) with the number diagonally opposite in (column, row). Row 3, column 6 for example, shows that judges ranked Bob Bogan ahead of John Van Hooten six times. Row 6, column 3 shows that judges ranked John Van Hooten ahead of Bob Bogan four times.

There were two flights with five judges, for a total of ten. You'll find that, in most cases, a majority of judges favors the pilot ranked ahead in the category. This pattern places higher numbers toward the upper-right and right areas of the table. It places lower numbers toward the lower-left and lower areas.

Conclusion

We hope these examples make it clear how the CR method works. Unlike the method of computing the mean score, a single judge cannot seriously skew the results. Ranking a pilot lower simply places the judge in the minority. Unlike the TBLP method, it is easy to verify the correctness of the result. The rank preference tables clearly present the data supporting the ranking.

We hope that you find the CR method intuitive, with readily verified results, and recognize that it is immune to manipulation by individual judges.

Reference