I was a member of ASR (Audio Science Review) for +1.5 years. Originally I was attracted by the main selling point. The main selling point of ASR is that as long as Amir (owner of the site) measures a device as transparent and recommended it, it will sound the same as all other similar devices deemed transparent by him. This is a very enticing selling point. It means a $100 DAC (Digital Analog Converter) would sound no different than a $2000 DAC, as long as both are deemed transparent by him. I can enjoy the best quality sound and save lots of money by buying the lowest cost transparent devices.
Unfortunately, after buying multiple best value DAC (digital analog converter) and HPA (head phone amp) based on Amir’s recommendation and comparing them, I slowly lose faith in ASR measurements. The selling point no longer hold much value to me.
Here, I will provide 3 difference cases where I think ASR measurements are incomplete and can’t tell the whole story. Please search in ASR thread for more information.
Case #1:
Thread started date: July 7, 2021
Thread Title: Both DAC are transparent as measures by Amir, why one not as clean sounding as the other?
Summary: Typical measurements of Topping d30pro like SINAD, freq response, multitones, etc., were ok. But, it sounded different than Gustard X16. Eventually, using loopback captures null comparison software, it was discovered that d30pro had issues, as compared to Gustard x16 and Topping E30. Main things that were discovered:
*d30pro is not as stable as X16
*d30pro captures vs original file contain instances where null difference reached potential audible levels, whereas X16 is about 10db better, thus unlikely to have audible difference vs original file.
Conclusion: Amir’s typical suite of measurements are not able to discover audible issues in a DAC. Loop back captures null comparisons are much better in discover issues.
Case #2:
Thread started date: Dec 17, 2021
Thread Title: I feel bass difference between headphone amps. How to check phase mismatch?
Summary: SMSL HO200 HPA low bass sounded a bit less clean as compared to Topping L30. No one can explain why. Using loopback captures null comparison software, it appears that HO200 has the most different in phase around 40 Hz when comparing to original playback file.
Conclusion: Again, Amir’s typical suite of measurements are not able to discover audible issues. Loop back captures null comparisons are better in discover issues.
Case #3:
Thread started date: Feb 9, 2022
Thread Title:
DAC and amp combos did not give same clues when running online blind tests. Why? What would be the desired clue?
Summary: In another thread where people shared timing test results, it was mentioned that 1ms tests were easy for some people because there were tonality changes. That prompted me to check 5ms, 2ms, and 1ms using multiple different chains and got multiple different kind of clues.
Details:
Using different combination of DAC, HPA, and headphones, I heard different clues when running 5ms, 2ms, and 1ms tests.
Here is a screen shot of blind test results using different chains:
Side notes:
There are a few people that claimed that I need to perform “voltage matched” and “blind” online blind tests for it to be meaningful. They firmly believed that I need to perform online blind test “blind” (i.e. not knowing which chain is being used when running the online blind tests), else the results do not count. I agreed that “voltage matched” request is reasonable. Thus, I did voltage matched tests for 2 chains and I still got different clues:
Regarding running online blind test “blind”, they think that by looking at the chain, it would somehow create “bias” in my brain. The “bias” is influencing how I sensed the clues for each chain being used. This I disagree. In this particular case, the “bias” claim is not logical. I have two logical questions to them, using 5ms test as an example:
#1: When I can easily sense a special extra click clue and can easily pass a blind test with Topping chain, how likely that I can totally miss such easy clue in another chain (I can’t hear that second click in Gustard chain)? Here is the video of how I could easily pass the test using Topping chain:
So, which one is more logical:
a. I really can’t hear that special extra click clue in Gustard chain?
or
b. Gustard chain has the same extra click clue, but somehow by seeing the Gustard chain, my brain mysteriously create a “bias” that blocks me from hearing that clue forever, no matter what time I try?
#2: In the ASR thread, no expert can tell me which clue is expected or desired. If no one knows which kind of clue is correct or desired, there is no basis to even form a “bias” in the brain, correct?
So, which one is more logical:
a. I really can’t hear that extra click clue in Gustard chain?
or
b. Gustard chain has the same extra click clue, but somehow by seeing the Gustard chain, my brain creates a “bias” of unknown root cause, and mysteriously always block me from hearing Gustard’s extra click clues?
Conclusion: If ASR main selling point is indeed correct, shouldn’t all Amir’s recommended chains give me the same clue? The timing tests clearly showed that different chains created different clues. Thus, after this case, I have little faith left in ASR’s main selling point.