It's easy to pick out a bad lens with a few revealing tests. Focus is inconsistent, or results not sharp from one side to another. Perhaps contrast is weak or colors appear warmer or cooler than expected. Maybe the images are fine but areas beyond focus distract or otherwise detract from the overall shot.
If a lens passes these tests, and you pick up another design, then it gets interesting. How does one compare lenses that each pass the basic tests? For most of us, it becomes a balancing act between what else we already own and use, features like weather seals, the weight of one lens over another, compatible filters, or simple cost. If a lens costs four times as much but both give great results, was that cash well spent?
I own Pentax' 18-55 WR "kit lens" for foul-weather shooting. It's a decent lens, can focus close and go out in the rain. Yes it has more distortion than many at the wide end, and it's f/5.6 past maybe 40mm; these concerns will make a big difference to some, not so much to others.
Other options are available, which brings up the Big Question: if a lens costs several times as much as the 18-55, how many times 'better' should it be? Faster, smaller, more seals, incredible optics - some combination of these or other factors will be in play. No two people will weigh these factors quite the same, nor can they quantify 'twice as good' - so
like every lens ever made the better / faster / more expensive lens is not for everyone.
I have owned Pentax' 18-135 DC WR, two or three times in fact. Its specs are compelling relative to the 18-55: more telephoto range, a quiet focus motor, still f/5.6 but not until longer focal lengths. Good points - yet I sold it each time. I found issues with image quality above 100mm and did not wish to carry a bulkier, more expensive lens that I only found useful for 2/3 of its range. I could choose to correct most of its issues in the camera with a speed penalty, or fix them on my computer which also takes time. Or I could own the 18-55. Many others disagree strongly that this lens has any more problems than other lenses and is better than most; that's OK as their needs are different, just as their copies are different from mine.
So it comes to this: a Limited 20-40mm zoom has arrived on my doorstep, and tests against my other lenses will commence. It's a new open-box lens obtained for over $100 less than any price I'd seen, but still cost about six times what I paid for the used DA18-55 - all that for a narrower focal range! Why would anyone pay that much more for this lens?
Well, it has the weather seals but includes a DC motor. It's faster at f/2.8-4.0. Stopping at 20mm avoids the worst of zoom distortions. Primarily though, it's a Limited lens which means great build and superior image quality. Until now it also meant primes; this is the first Limited zoom so it's the proverbial odd duck. Many current Limited users cannot bring themselves to accept a zoom as deserving the Limited label while others have found it too expensive to bother with. Let's pretend that I replaced both the DA21/3.2 Limited (slightly faster at f/2.8) and the DA40/2.8 Limited (slower at f/4) for less money.
You could say I replaced the 35 Limited too but it's a 1:1 macro, so not quite an even trade - but the Limited primes have no weather seals.
Given its uniqe nature, should the LimiZoom be tested against primes, zooms, Limited lenses only? That's a problem for pro review websites, professional users or geeked-by-gear folks; I'll skip that question and just compare it with what I have. Its 'competition' in my bag comes from the 18-55WR and a Tamron 28-75 f/2.8, with a Pentax F 17-28mm fish-eye thrown in despite its fishiness. I shall also just let the 20-40 be itself on a solo jaunt or two, unencumbered by the abilities of other lenses.
How do these four zooms compare?
17-28: really wide fisheye view, fairly slow at f/3.5-4.5, compact & light, fairly inexpensive
18-55: weather seals, 1:3ish close focus, ~$100, slow f/3.5-5.6
20-40: Limited metal build, weather seals, 9 aperture blades, DC silent focus, f/2.8-4
28-75: fast f/2.8 = relatively large and bulky, 7 aperture blades
On to the 'tests', at which I'm pretty poor. I shall try to keep them on common levels re. shooting parameters but I always leave something out that makes a difference. Oh well...