Reviewing reviewers

October 21, 2016
 by Paul McGowan

It’s likely we all have reviewers we trust as well as those we don’t. And, the reasons are obvious. You wouldn’t necessarily value your mechanic’s opinion on gardening.

Out on the town yesterday we turned to Yelp for help finding a restaurant. Yelp, an online app of personal reviews, is really a hit or miss affair—mostly miss. The problem with Yelp is simple. You don’t know anything about the person writing the review. If you’re looking for fine dining, trusting the review of someone whose diet consists of fast food isn’t meaningful.

One restaurant in Nashville, the Farm House, looked good, but the first review gave it but one star, and a thumbs down. “The problem with farm to fork is that all of the focus is placed on quality of the ingredients, and flavor takes a back seat.”

That negative review was the best recommendation we could have asked for, and we enjoyed a marvelous lunch of fresh foods. Flavorful too.

The point is simple. We choose reviewers because their tastes match our own. We read reviewers because they have better access to, and knowledge of, the gear we wish to consider.

Good reviewers, the really good ones, work hard at their craft and they have always been in short supply. But they’re important and many of us would be lost without their sage advice. If you find one you like and enjoy reading, send them a note of encouragement once in awhile.

It’s got to be a tough and lonely job.

Subscribe to Paul's Posts

47 comments on “Reviewing reviewers”

  1. I’d need to know a lot about a reviewers listening taste, habits and surrounding to trust his statements for more than a rough comparing orientation on the market.

    Often his selection of music and recordings or the used review gear already says enough.

    And are there any bad reviews?

  2. What is the most obvious aspect helping to distinguish between a claqueur and a reviewer (neutral (?) critic)? And the story of claqueurs in history of opera and theatre is long and most revealing. Sapere aude! Just being submissive to self-proclaimed authorities gets you the victim of a reviewer!

      1. Clacqueurs are payed for applauding and clapping enthusiastically after or during the performance of the artists. Sometime they are payed for barracking the artists. Quite normal in the period of Mozart’s creations.
        Same clacqeurs you find reading the articles of reviewers in automobile magazines. They rarely report about the permanent pollution by the combustion engines! Same phenomenon in stereo magazines: no articles about pollution of the original pure sound here! 🙂

  3. I’m waiting for a reviewer to say a piece of equipment is rubbish and a total waste of money should anyone be so naive as to buy it. Yet to see that. So that’s it as far as I’m concerned.
    I have subscribed to Gramophone since around 1980, with the odd gap, nothing else. They’ve been reviewing audio equipment since shortly after the magazine was launched in 1923. There is some consistency, as they have only had two audio editors in the last 50 years. Until the early 1980s most reviews were based entirely on measurements. Listening to music through the device was not considered necessary. As most of the technical challenges required to produce high quality domestic audio were achieved by then, measurements became rather redundant. I recall reading the review of the Quad 33/303. (I have just re-read it, from February 1968.) It was necessary to go and buy more accurate test equipment, at the considerable cost of £400, as the equipment normally used had higher harmonic distortion than the Quad units being tested.
    Reading reviews online from ignorant students or opinionated old blokes that mostly comprise subjective comments on compressed (vinyl/rock) music I’ve never heard of seems a waste of time the could be spent more usefully watching paint dry.
    I have chosen my audio system based on listening, but also on longstanding reputation based on proper research (Quad current dumpers, largely changed for 40 years & Harbeth, licensed from the BBC who spent millions over decades developing monitors), engineering quality (my turntable, it is after all a mechanical device) and faultless software design (Auralic Aries streamers). My P3 serves a purpose, bought on spec. CD transport and XLR cables all from the pro store catalogue.

  4. Going back decades ago, to when you were the first manufacturer making affordable dif balanced designs, I got the why, and replaced all my gear.

    From then on, I pulled my hair out when reviewers with primarily single ended systems would review a dif balanced product and ultimately end their reviews with they heard very little difference (if any). I would write angry emails to the various magazines as to the ignorance, nay, stupidity of repeating this egregious mistake which was/is totally unfair to the manufacturer

    So many times, it’s the blind leading the blind when any product is reviewed

  5. I would add that the only rag of less merit than an audio magazine is the Michelin Restaurant Guide. For holidays, I plan the tour and my wife plans the food. My wife has an extraordinary ability to walk into any restaurant at 8pm on a Saturday evening and get a table. Recently on a Saturday evening in Rome, after walking out of a Michelin restaurant, whilst in a taxi I googled “best restaurants in Rome”, called the first one and asked for a table. They laughed and hung up. My wife told the driver to go there anyway and 10 minutes later we were sitting at a lovely table. I should get her to choose my hifi.
    http://www.glass-restaurant.it/en/glass

      1. My wife is beautiful, talented, speaks about 6 languages and gets us into restaurants as and when needed, but she still cannot operate the Sonos app on her smartphone, let alone turn on the hifi.

        1. Sounds like she’s got you perfectly trained to do it for her. 🙂

          Or else she treasure’s your taste in music like you treasure her way with the maitre d’.

  6. Although the tastes, reputation and methodology of the reviewer has been discussed, it is important to give yourself credit for having the ability to read between the lines. Examples in this thread include relying on the description of farm fresh food and refusing to accept a telephone reservation refusal from a restaurant. When it comes to music the same thing applies. We can forget how many stars the reviewer assigns, rather focusing on their words for the clues we need.

  7. Music reviews for me are the most frustrating. For the life of me I cannot fathom what some reviewers are listening to when they wax enthusiastic about something. I distinctly remember more than one magazine going on and on about a St. Vincent album (with David Byrne) a coupe of years ago and I bought it based on the reviews and to be honest have never made it more than two songs. Fingernails on a blackboard are more appealing than what I was hearing. Others I am sure find that music the most endearing of their lives. So…..to each their own. I actually like the “1989” album by Taylor Swift, which I know for many others who have posted here is quite unlistenable (although Fremer wrote a review of it and liked it, LOL!!).

    Equipment reviews for me are a mixed bag but I am of the belief that once a certain level of quality is attained the components are very similar. A Yamaha top-of-the-line receiver is going to be pretty comparable with offerings from Onkyo or Marantz (same goes for the integrated offerings which are audiophile rated). At that point I start looking at the other features like the apps or number of connections. Sound quality I think is comparable. Again, just me.

    Finally, one more thing about yesterday’s post and the loudness of venues. I swear by this product from Comply Foam, if anyone has not tried them I recommend it. Cheap and well built and effective: http://www.complyfoam.com/products/comply-foam-plugs/

    Happy listening (and review reading).

    Larry

    1. How are you listening to “1989?”

      I’ve found that the CD is so compressed as to be unlistenable, but the vinyl is absolutely wonderful.

      That doesn’t help those who hate her songs, but I’ve certainly found the format effects sound quality, and sometimes in odd ways.

      1. I am not listening to it now but have many times in the past. I do not disagree that it is very compressed, but, it’s now a pop album and it seems that is the way of things for that market. Her prior country based offerings were far less compressed but certainly not perfect. I do NOT ever listen to it in a critical way, rather it is on the stereo as background while I do something else, or in the car but certainly not as a relaxing venture in a listening chair!!!

        A friend of mine is a sound engineer, having worked on many projects over the years from the vinyl days forward, and he tells me it has so changed in the industry. The name of the game now is just to be heard, and so the compression takes place in a way that the sound will permeate the din of a department store, for instance, as the music comes out of the commercial grade ceiling speakers (usually Bose from my observation). So for those purposes the volume has to be pumped up. The number of people who want to sit and concentrate on the songs as a sole function while listening are a small percentage of listeners as we know.

        I have not bought the vinyl and don’t plan to. I did buy the Adele album “25” in digital and vinyl and don’t think there is much difference, if any, between those mixes at all.

  8. I like the concept of Yelp but investigation after investigation has proven they are in the business of extortion. When a business has poor reviews, a Yelp representative will offer to remove those if the client purchases “enhancements” to their account. I know two small business owners who have experienced this.

  9. Like loop7, I’ve seen several negative reports locally on Yelp. First that merchants offer “special considerations” to customers for writing positive comments. And second, for protesting about negative reviews until they are removed.

    Now for component and music reviewers, I believe I must read a few reports to understand if they hold similar tastes and expectations to my own before I can accept their input. Otherwise, who knows?

  10. Paul, are you coming down from the acid trip that modern downtown Nashville has become? Actually there is an emerging vegan nouvelle cuisine scene if you seek it out, a vegetarian can fare well in the land of Tennessee barbecue.

    BTW -> this is hilarity …

    http://www.nytimes.com/video/us/elections/100000004721602/clinton-jabs-trump-at-al-smith-dinner.html?contentCollection=U.S.&contentPlacement=3&module=most-popular-videos&action=click&pgType=Multimedia&eventName=most-popular-videos-click

    If you find yourself in NYC with some extra down time, the Ivanka’s Salad at the Trump Tower Grill is only $18. 😀

  11. What happened to Harry Pearson’s advice, trust your own ears? You said you went from room to room at ROTFLMAO and every one sounded different. This one was shrill, that one was boomy, another bass shy, another honked like a goose. I’ll bet they’ll all get great reviews especially if they also take out ads in the popular hobbyist magazines. Then every one of them will have big signs with quotes from some reviewer filled with accolades about how great his equipment is. A lot of these reviewers get equipment at steep markdowns, or borrow them for extended periods to “evaluate” them. How long is “extended?” Until someone loans them one they like better. It breaks down on them? No problem. They quick ship you another in five minutes.

    Would you have speakers that look like this in your home? I wouldn’t. Even one of them might devour me in my sleep.

    http://www.stereophile.com/content/jadis-eurythmie-ii-loudspeaker#QmOiqf5QF5ucyGiG.97

  12. Granted reviewers have greater experience with greater number of components or food for that matter but they are human with likes and dislikes, preferences etc. That is why taking every thing as gospel truth is not advisable. Reviewers are like road signs. They can point in a direction but the choice whether to take that road is up to the individual. If a reviewer seems to try too hard to convince then a red flag should go up. When a review is shorter and to the point and one gets the impression that the reviewer is so impressed that he or she does not have to say more then it is time to sit up and take notice. That is my experience and it has served me well so far. Regards.

  13. Audio magazines exist to sell advertising and thereby make a profit for the publisher. At the end of the day, the reviews are the fodder to fill in the gaps between the advertisements. Yes or no?

    1. Some do, some don’t. TAS and Stereophile both started with no advertising, though of course they were working on paid subscriptions.

      I know many of the reviewers in our industry, and publishers too. The good ones are passionate about their art and their products – they pay no attention to the subscriptions or advertisers. They don’t interface with them, they don’t really think about anything than furthering the art. Those people are my heroes.

      1. How do you sort the good reviewers from the ignorati?

        Gramophone, my monthly read, was started by Sir Compton Mackenzie, a good author (Whisky Galore! etc.) and great eccentric, so he could write about music, publish correspondence with his musical friends and include reports from Gramophone Societies around the UK.
        Within 5 years (1928) it was publishing a series of articles about how to build your own Gramo-Electric Amplifier, and after 10 years (1933) it was already about one quarter or one third advertisements.
        Fortunately the musical core of the magazine, record reviews, has remained unchanged, and I occasionally buy a copy of Record Collector for the reviews.

        One of many issues I have with audio magazines is that they seem to be aimed primarily at already committed audio hobbyists. If someone without much or no knowledge wants to find out what to go and buy, they are generally completely useless. Compare with car magazines. I know nothing about cars, but can pick up a copy of “What Car” and it will almost tell me what to buy based on my own criteria. As audio magazines generally review components, it’s a bit like reading an auto magazine containing reviews of clutches, gearboxes, engines and shock absorbers. How much use is that?

      2. IMO the most “successful” reviewers in their publishing company (unfortunately the best ones which we admire often are not among them, unless they own the mag, which is unlikely ) are those who understand their job as keeping the business alive and the endless search of the customers steady.

        Which leads to hardly differential and mainly positive reviews, avoidance of setting obvious achievements above an average heightening and keeping big companies at the top independently of their true performance.

        Is there anything better for mags and industry than continuously searching, selling and brand-switching customers?

  14. The only reviews I trust are Stereophile’s sidebars, because I understand the highly consistent biases of Real Time Analyzers and how those squiggly lines translate to reproduction accuracy, when convoluted with details of cabinet, driver and crossover design.

    I don’t trust human reviewers because they use content that has little to no connection to reality, and they hardly ever train their ears to un-amplified music. The near-universal bias of listening to studio tracked, mixed or mastered recordings reinforces the vicious circle of producing unrealistic recordings for unrealistic systems.

    Reviews also include a substantial bias in the listening room, which is never set up according to principles that I know produce realistic reproduction, nor are they measured to reveal the anomalies. These factors are neither predictable nor translatable into my world of temporal, transient and spatial accuracy. Most “imaging” is artifactual – when reviewers rave about speaker imaging I will generally avoid that model.

    I do pay attention when reviewers use genuine audiophile productions like Chesky, 2L, MA and Water Lily – near coincident pairs with zero processing, by engineers who listen to acoustic music in the room regularly and don’t use studio gear nor studio acoustics. I also learned a lot from “shoot-outs” and listening to members’ systems and collections with the Connecticut Audio Society, which represented many lifetimes of searching for musical truth.

      1. Kavi is an inspiration for me. He has a higher level of involvement with the music, introducing players and capturing their first encounter. This is the raw music that I crave. I have recorded hundreds of premiers and free improvisation sets for the the same ephemeral quality which only exists in the wild. Studios are like farmed fish that have lost their place in the world.

        He also is not a dogmatist “one size fits all”, constantly trying different microphone configurations to find the one appropriate for the combination of players, program and venue.

        1. And that’s why i respect the likes of your good self, Kavi, Keith Johnson & Mark Levinson before us, because you truly understand the essence of music from the performance to capturing and documenting it live.

  15. Yelp is awful as a source for restaurants reviews… Chowhound is my go to site.

    Regarding audio reviewers.. I like Mickey Fremmer as I find him honest and his humor only adds to his reviews. Jonathan Valin does IMO very in depth reviews of (mostly) state of the art gear. Greg Weaver is coming on strong as well.

  16. I have been reading Stereophile since June 1993. The only way to trust a reviewer is to read them for a long time and listen to components they have reviewed. If I hear what they heard, I can then put some trust in their reviews. Some reviewers I enjoy reading, but I know we have different tastes. Sam Tellig was a good read, but after he went nuts over the Radio Shack portable CD player, I never considered any component based on his review. When I read about the RS player I drove right over to RS, and bought one. This was at like 1pm, by 4pm I took it back. I had CAL DX-1 at the time, their entry level player, and I heard nothing good from the RS unit. I shared this a while back and was told I had needed to tweak it. The one Sam had heard was stock. All the tweaking and power supply upgrades weren’t going to turn it into a decent player, in my opinion.
    And as far as bad reviews, if you know the writer well enough you can tell by there damning the component with faint praise. Or if they make it clear you should listen carefully before buying, that says a lot. When they buy the review sample or tell you if the budget allowed they would, that says it all.
    I know they try to choose products they were impressed by at shows. And if they do a real hatchet job, no more review samples from that company. I do know that Stereophile does not require advertising to review. Unlike some of the online sites.
    It still is only one tool to seek out components that will please.

  17. I will be eternally grateful to Robert Harley for his rave review of the BSG Technologies QOL Signal Completion Stage. His skillful use of the English language allowed him to articulate exactly what I also hear with that piece of kit in my home system. That thing supposedly restores some of the phase information that is lost during the recording process (or as Soundminded would call it “the sausage making process “). Maybe it does or maybe it doesn’t, but I believe it does to a certain extent because it really makes my “canned music ” sound very close to the live “absolute sound “. I try to do most of the things that acuvox recommends, and that includes going to many live unamplified orchestral and chamber music concerts. If I can, I always sit front and center, as close to the performers and recording mics as possible. I am fortunate enough to have several recordings of these concerts, and I am amazed and delighted by how close the sound is to what I remember. Like Soundminded, what is important to me most is the tamber of the instruments/voices. If you read the old reviews of the “qol”, most of them focus on how the imaging is improved or changed (what brings Paul closer to the music). But again, for me, it’s the tonality that matters most. I use the “qol” as instead of a preamp: I feed it the signal from my Direct Stream , and then pass it on directly to my amps or active crossovers depending on how my system is configured.
    I know how Soundminded feels about reviewers, but I am definitely in Paul’s camp on this one.

    1. I looked up the BSG Technologies QOL Signal Completion Stage as I’ve never heard of it. You got me reading reviews. Urgh. Some thought it added euphoric distortion. Others thought it added gain, which makes objective testing impossible.
      Remarkably, the Stereophile reviewer managed to replicate its effects by setting up an existing DSP device to match the BSG specification.
      So it seems to be a fixed DSP device working in the analog domain. Nothing wrong with that if you like the results, but retrieving lost data? That would be audio alchemy.

      1. Audio alchemy it may be. If you reread the Stereophile review (done by Soundminded’s most reviled reviewer, John Atkinson), you will see that Atkinson read the product’s patent papers, and he mentioned that the QOL incorporated the golden ratio into its processing. I am a firm believer in the golden ratio. Look at how much success George Cardas has had using it in his cables!
        I have done level matched comparisons compensating for the ~3dB gain the circuitry makes. All I know is that to my quasi acuvox trained ears (I also have almost, if not absolute perfect pitch), the circuitry adds a life-like character to the sound/music. If I have been sucked into “euphonic distortions”, so be it. I would also like to note that there is a group of critical listeners who have written a series of articles in TAS about subtle differences in the sound of various aspects of computer audio (file type, sampling rates, cables, …..); the last time I checked, they incorporated the BSG qol in their reference system.

        1. The Golden Ratio is a natural phenomenon and a basic mathematical rue applied in the arts and sciences for thousands of years. You don’t have to believe in it any more than believing that 1+1=2.
          My general view of a successful cable is one that conducts electricity and is well insulated from outside factors. I have never heard of George Cardas.
          The person who replicated the effect with reference to the patent using DSP appears to have proven what the device does. DSP shapes music often for the better. Creating something out of nothing, as is claimed, is reserved by the Lord Almighty.

          1. Yes, sad but true. Just before the last Axpona show, they sold the last of their units to Walter Leiderman (Underwood Wally) and he sold them for $1K. I am pretty sure that he sold all of them.

  18. Maybe, if this is the correct conclusion, reviewers are only good for manufacturers as they effectively lock consumers into a cycle of continuous replacement by endlessly feeding their paranoia with stories of revolutionary technologies and the dramatically improved transients from a $1,000 USB cable.

    The reason seems to be that the reviewers are far too close to, if not in the pockets, of the manufacturers. We all love music in its various forms. Paul counts audio reviewers amongst his heroes. I can now see why. Mine are people like Beethoven, Shostakovich and Jiri Kylian.

    My occasional perusal of a car magazine to buy a car leads me to believe they are of infinitely more value because the reviewers are entirely independent of the manufacturers, indeed the reviewers are usually anonymous, they review to a predetermined format and asssess standardised features, economy, comfort, space, etc. They provide the basis for an informed consumer decision.

    More significantly, auto magazines have readership in millions, whereas audio is probably thousands. So if an auto magazine says a certain car is terrible, the manufacturer might take note and try and improve their offering. Audio magazines and manufacturers just seem locked in a process of securing each other’s revenues, constant back-patting, or arse-licking as we call it over here. Is it Stereophile where the manufacturer gets to write a letter thanking the reviewer for saying such lovely things about their new product?

    I bought some audio yesterday. I changed my XLR leads. After a bit of rearranging due to excess heat, the leads were overlapping power cables. Things sounded a bit off. my existing Sommer/Neutrik leads have a copper shield and cost $15 each. OK for 100m mic cable. I went for the Mogami Quad cable, it just has much better insulation. Cost $50 each. The full specifications of each cable, including technical measurements, are on the manufacturers websites. No reviews required.

  19. What does it take to be an audio equipment reviewer? Ear training? Hearing tests? Any certifications of any kind by any organization at all? No, just the ability to write the same review a thousand times over and make each one sound different from the others. You could be stone deaf and you could still get a job to be a reviewer in an audio magazine. BTW, I not only don’t take anyone who demagnetizes plastic phonograph records seriously, I don’t take anyone who associates with, endorses, or has any respect for their opinions seriously either.

    1. And you know what? Nobody cares.

      Oh boy Soundmind doesn’t respect me. How can I go on. I mean he is such a great open minded guy. I so wanted to be best buddies.

Leave a Reply

© 2023 PS Audio, Inc.

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram