Results of the public multiformat listening test @ 64 kbps (July 2007)

These are the summary results of the public multiformat listening test @ 64 kbps.

User comments are available here. If your packing utility supports RAR archives, you can also download a signed, locked and solid RAR file containing all results for all samples.

Encryption key can be downloaded from here.

How to interpret the plots: Each plot is drawn with five codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis. The number of listeners used to compute the means (average ratings) and 95% confidence intervals are given on each plot. The mean rating given to each codec is indicated by the middle point of each vertical line segment and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is identical to the one used in Roberto Amorim's listening tests.

One codec can be said to be better than another with 95% confidence if the bottom of its segment is at or above the top of the competing codec's line segment.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Plot Comments
Plot Instrumental (acoustic guitar)

High anchor on top, followed by Nero on #2 and WMA Professional on #3. Vorbis is tied to the low anchor on #4.
Plot Instrumental (harpsichord)

High anchor on top, Nero comes second, WMA Professional is third, Vorbis is once again fourth and the low anchor loses on #5.
Plot Usual pop / trance

High anchor on top, followed by all three contenders tied on #2, low anchor loses.
Plot English female voice

High anchor is tied to Nero, Nero is tied to WMA Professional, WMA Professional is tied to Vorbis, but Vorbis is worse than Nero. Low anchor is last.
Plot Stereo separation, violins

High anchor is first, followed by Nero which is tied to WMA Professional, WMA Professional is tied to Vorbis, but Vorbis is worse than Nero. Low anchor is last.
Plot Symphonic orchestra

High anchor is on #1, Nero, WMA Professional and Vorbis are tied on #2, low anchor is third and loses.
Plot Bandwidth test

Once again, high anchor on #1, all three contenders on #2, low anchor on #3.
Plot Classical, very dynamic

High anchor is first, Nero follows tied to WMA Professional. WMA is tied to Vorbis with Vorbis being worse than Nero. Low anchor is last.
Plot High tones, sort of Indie music

High anchor is first, Nero and Vorbis are tied on #2, WMA Professional is third followed by the low anchor on #4.
Plot Rap, strong English male voice

All three contenders are tied to the high anchor on #1, low anchor loses on #2.
Plot Reggae

High anchor is first, Nero, WMA and Vorbis are second, low anchor is last.
Plot Electronic, pre-echo

Same as above.
Plot Pretty loud mixture of symphony orchestra with percussion and electric guitar

High anchor is first, WMA Pro and Nero are tied on #2, Vorbis is third and the low anchor is fourth.
Plot English male voice, easy listening / pop

High anchor is tied to Nero, Nero is tied to WMA Professional, WMA Professional is tied to Vorbis, but Vorbis is worse than Nero. Low anchor is last.
Plot Stereo separation, electric guitar

High anchor is first, all three contenders are second, low anchor loses on #3.
Plot German male voice

High anchor is first, WMA and Nero are tied on #2, Vorbis is third and the low anchor is last.
Plot 80's electronic, pre-echo

Again, high anchor is first, followed by all three contenders tied on #2 and the low anchor on #3.
Plot Jazz, French female voice

Ditto.

These are the bitrates used:

    Sample                              AoTuV        Nero        WMA Professional
    -----------------------------------------------------------------------------
    Poets_of_the_fall___Shallow         75           57          64
    Bachpsichord                        73           60          64
    MysteriousTimes                     72           73          64
    TomsDiner                           65           52          64
    Eleanor_Rigby                       61           64          64
    macabre                             61           63          64
    bibilolo                            85           56          64
    Paganini_Allegro_spirituoso         65           60          64
    Flyin___to_Fly                      75           65          64
    WhiteAmerica                        65           68          64
    Les_Voleurs                         71           65          64
    eig                                 89           71          64
    symphnoy_metal                      64           76          64
    BigYellow                           72           73          64
    Gypsy                               69           69          64
    spmg54_1                            68           43          64
    kraftwerk                           72           64          64
    Senor                               66           65          64
    -----------------------------------------------------------------------------
    Average: 25.8                       70.44        63.56       64

Overall rating: The results for each sample were grouped together without modifications.

Then I performed an ANOVA analysis. The results are graphed below.

Plot

The high anchor iTunes LC-AAC at 96 kbps is first, Nero and WMA Professional follow on #2 with Vorbis tied to WMA Professional, but statistically worse than Nero. The low anchor iTunes LC-AAC at 48 kbps loses.

Here is a zoomed version of the plot showing the competitors only and leaving out the anchors.

Plot

Finally, I would like to thank everyone who participated!