MP3 and Ogg Vorbis listening test

(Originally written on 18 Jan 2004, revised 3 Feb 2005)

Since I always have trouble finding my CDs, I am considering to rip my whole CD collection to Ogg Vorbis or MP3. But to which format and bitrate? Everybody seems to agree on the fact that Ogg Vorbis sounds better for a given bitrate, but there aren't yet that many hardware players.

Since I would replace my CDs, transparency is the main requirement. People on the net declare that for full transparency, one would need at least ogg -q6 or mp3 at 300 kbps bitrate. I mostly listen to classical music, which is a bit sparsely represented in the various online tests.

I compared the compressed sound files to the original and tried to describe in what respect their sound differs (for some reason, this is seldom mentioned in online listening test reports), at least as far as I can reliably tell the difference in a blind ABX test. I found it easier to find sensitive spots in short fragments, instead of to switch back and forth during a whole track.

Apparently I have tin ears, because the encoders become transparent at bitrates far below the -q6 or 300 kbps that is mentioned by golden-ear listeners elsewhere, or my music taste is not demanding. (I do use a good headphone and a good sound card and my ears can still hear up to 20 kHz.) If someone knows how I can upgrade my ears, please tell me. :-)

As far as the encoders are not transparent, Ogg Vorbis and MP3 turn out to sound very different. Note that I hardly ever listen to MP3s and the like, so this may be a well-known fact for you. The MP3s tend to give annoying ringing artifacts, often strongly localized in certain music fragments. Ogg Vorbis, on the other hand, tends to distort the stereo image and create an overall hissing background and some coloration. Ogg Vorbis's distortions are much less annoying, because they are comparable to coloration by loudspeakers.

The stereo-image distortion is strongest in the choral and harpsichord solo recordings. Likely, different microphones were recording the same sound source in those cases, which results in strong phase differences between the left and right channels, which are apparently hard to encode in mid/side mode. I suspect that the other recordings mostly have an intensity stereo image as opposed to a phase image. Strangely enough, I didn't observe stereo imaging problems in the MP3s.

Enough said, here are the results.

Encoders

Lame 3.93 --resample 44.1 --abr (xxx+offset)

I tweaked the --abr option such that the final average bitrate was with +/- 1.5 kbps the bitrate that I wanted. Lame wants to downsample at lower rates (below --abr 103), which my ABX test couldn't handle, so I forced a 44.1-kHz sample frequency. Maybe unfair to Lame at low bitrates, but that's life...

oggenc 1.0-7 -q xxx

I used oggenc simply with quality numbers.

Samples and ratings

I have deleted the original samples in order to save space, so you have to trust me on my findings.

antonilla

  Juan del Encina, Antonilla dees desposada
  Margaret Philpot (alt), Christopher Wilson(?) (lute)
  Hyperion CDA 66454
  Comment: the lute in the original sounds as an artifact but isn't...
  mp3 80k: horrible buzzing (22.05 kHz resampled)
  mp3 96k: transparent
  ogg q0 (52.7k): some ringing
  ogg q1 (68.5k): transparent

bachkyrie

  J.S. Bach, Messe in B minor: Kyrie Eleison
  Ton Koopman, The Amsterdam Baroque Orchestra and Choir
  Erato 4509-98478-2
  mp3 96k:  strong ringing
  mp3 112k: audible
  mp3 128k: audible artifact at 21-22 seconds (2nd "Kyrie")
  mp3 150k: audible
  mp3 160k: transparent
  ogg q0 (47.8k): strong distortion in stereo image
  ogg q1 (62.8k): stereo image; colored sound
  ogg q2 (79.2k): stereo image, colored sound, hissing (mainly in beginning)
  ogg q3 (101.8k): audible hissing
  ogg q4 (116.4k): transparent

bachwtk

  J.S. Bach, Das wohltemperierte klavier: prelude nr. 14
  Leon Berben, Harpsichord
  Brilliant Classics 99362
  Comment: A budget edition; I'm not really a fan of harpsichord
    music, but since harpsichord is reputedly hard to compress, I
    thought I'd give it a try.
  mp3 96k:  dull (missing high frequencies)
  mp3 112k: dull
  mp3 128k: somewhat dull
  mp3 150k: transparent
  ogg q0 (65.4k): stereo image, dull sound
  ogg q1 (79.2k): stereo image, a bit colored
  ogg q2 (96.5k): fuzzy, hissing middle register
  ogg q3 (131.6k): nearly transparent
  ogg q4 (162.0k): transparent

mignon

  Hugo Wolf, Goethe Lieder: Mignon
  Geraldine McGreevy (soprano), Graham Johnson (piano)
  Hyperion CDA67130
  mp3 96k:  small artifacts on consonants ("ein *S*anfter wind")
  mp3 112k: very small artifacts
  mp3 128k: transparent
  ogg q0 (38.1k): ringing and hissing in piano intro
  ogg q1 (51.9k): ringing in piano
  ogg q2 (74.7k): some hissing in piano
  ogg q3 (96.2k): nearly inaudible coloring
  ogg q4 (109.1k): transparent

salsa

  Roberto Roena, Mi Mambo
  Nascente NSCD 039 "Salsa Moderna"
  Lots of percussion and trumpet
  
  mp3 96k:  dull
  mp3 112k: somewhat dull
  mp3 128k: somewhat dull percussion
  mp3 150k: transparent
  ogg q-1 (53.2k): dull/distorted percussion (cymbals)
  ogg q0 (68.0k): transparent
  ogg q1 (82.0k):
  ogg q2 (95.5k):
  ogg q3 (118.8k):
  ogg q4 (137.2k):

Choir

I made a number of choir recordings, to be found on the Ostrochorus web site, encoded as mp3 --abr 128 and ogg -q3 IIRC. The recordings are quite noisy due to constraints on where I could put the microphones. At first I couldn't hear much difference with the originals (i.e., a 290 kbps ATRAC-encoded minidisc recording, see below), but later I discovered to my surprise that Ogg Vorbis still had audible distortions while mp3 was transparent to me, even with identical bitrates. (The ogg files on the above website are a bit smaller, about 105 kbps)

The ogg distortions are in this case a kind of "rough" sound. I didn't invent that term, but it fits quite well. It is a kind of noise, but not a white noise like tape hiss but rather a feeling that someone is making electrical discharges on the background.

Bottom line

Based on this test---encoders may have improved since January 2004---I'd say that ogg -q1 or -q2 is the best choice if you want to have acceptable sound (i.e. when travelling, or with a noisy computer in the background) in as little space as possible. If you don't care about the storage space and want sound that is undistinguishable from the original, the choice is less clear. For some types of music, there is hardly any difference in storage space. MP3 is of course better supported by hardware players, although the situation seems to be changing..

Atrac (minidisc)

Atrac is the compression algorithm used on minidiscs. There are a number of Atrac-versions depending on the brand and age of the minidisc player; it is not always very clear which version is used by a minidisc player. My Sharp MD-player (bought in 2000 or 2001) has only one recording mode, at 290 kbps. The bitrate is quite high, but keep in mind that the data compression has to be done by a tiny battery-powered processor. As a comparison, encoding a 2-minute MP3 with LAME keeps my old 450 MHz Pentium 3 processor busy for about 1 minute.

Reportedly, 290 kbps ATRAC is nearly transparent. My experiences are different, something I noticed when I was working with amateur choir recordings. I suspect that I gave the ATRAC encoder a hard time because I made the recordings at a rather low level with a lot of white noise on the background. The result is an artifact that sounds like a car with a owner that is desperately trying to start the engine. You can hear this artifiact both on the MP3- and Ogg Vorbis-recoded recordings on the Ostrochorus choir website (take the recordings taken in 2003 and 2004).

(When you make a live recording, you have to take a safety margin to prevent clipping loud passages and you can't adjust the recording level when you are singing in the choir. So it's -10 dB most of the time. Also, with just a single stereo microphone pair you can't put them too close or the people who happen to be close to the microphone will dominate the recording)

[Email]