As I described in the one (and only) series of formal ABX tests that
I ran in order to discover the visibility of an embedded data
structure in video we gave the test subject the ABX box and s/he
could switch as many times as s/he desired before voting and there
was multiple voting on each clip and there were multiple clips. Each
subject ended up voting about 150 times. They did not have to vote at
any point and they could watch the clip repeat 10x or more before
voting the first time. They could trigger the vertical interval
switch at any point and most, if not all, of the clips were frame
accurately matched in playback from the uncompressed server. All
video interconnects were serial digital at 270 Mb/s.
If that did not skew the results in favor of showing a perceptible
difference, I don't know what would.
I think those of us who suggest ABX testing feel that it is better
than not using it for certain things. One does have to make decisions
as to whether to deploy a specific technology or not and there are
often factors more than visibility/audibility that affect the choice.
I generally send some of my good friends files to listen to without
telling them which is which. One test I've done is the playback of
two audio tapes on three different recorders. The ranking of the
recorders has been at least 80% the same: Studer A80, Sony APR-5000,
Studer A810. Now, doing fresh alignments and choosing different tapes
to play might change that, but these tests are a LOT of work. It took
me about six hours to prepare the six audio clips, and, even with all
that, I don't think the test was perfect.
The vision tests I did professionally took two engineers, a
statistician, a programmer, and a wiring person weeks to
prepare...and there again, we did things that we'd redone if we could have.
I'm glad you're bringing a rigorous intellectual experimental
psychology perspective to this, but I also think that calling your
post "ABX for Dummies" was a bit condescending to those of us who
have looked into it with a bit more depth -- with the help of
professionals such as statisticians.
I, personally, attempt to discern quality differences without
prejudice, but recently I thought I could hear some quality
differences where a friend whose usually better than I at picking out
these things did not hear the quality differences. Perhaps I knew too
much about the tests or my ability to switch back and forth between
tracks was faster and more sensitive than his. I was not in a
position to create a blind test for myself that I could then easily
go back and figure out what I had done, so perhaps my prejudices were
getting in the way equally as much as my friend's slower switching method.
Richard L. Hess email: [log in to unmask]
Aurora, Ontario, Canada (905) 713 6733 1-877-TAPE-FIX
Detailed contact information: http://www.richardhess.com/tape/contact.htm
Quality tape transfers -- even from hard-to-play tapes.