Ryan E. Walters, Cinematographer

Cinematic Excellence at 24 Frames a Second

Portland Lens Test 2011

On December 10, 2011 Indent Studios organized a lens test that was very eye opening and educational for all that attended. I feel fortunate that I could partake in making it a successful event. :) With the release of "affordable" cameras like the Epic-X, Scarlet, C300, and F3 (among many others) everyone is wondering what "affordable" lens package they should invest in, or rent for their camera. Indent Studios organized this event to use 3 Epic-X's to shoot with 7 sets of lenses and find out the answer. After handling the lenses, and reviewing the footage we all came across some surprising results.

Check out the footage for yourself and download the R3Ds after the jump...

It took quite a coordinated effort to get all this gear and people in one place for one day. Everyone donated their time, expertise, and even traveled on their own dime to make this possible. So I want to say a BIG THANK YOU to all of the contributors who made this a successful event. It would not have been possible without these folks. Please show your appreciation for what these people have done by visiting their sites, and using their services.

Lens Packages Provided By:
Cooke Panchro /i
(18mm T2.8 / 25mm T2.8 / 32mm T2.8 / 50mm T2.8 / 75mm T2.8 / 100mm T2.8)
Christopher Barrett Studio
Christopher Barrett, Owner

Leica R Cine-Mod
(19mm F2.8 / 24mm F2.8 / 28mm F2.8 / 35mm F1.4 / 50mm F1.4 / 80mm F1.4 / 100mm F1.8)
Duclos Lenses
Matthew Duclos, Lens Technician

Luma Tech Super 35 Illumina
(18mm T1.3 / 25mm T1.3 / 35mm T1.3 / 50mm T1.3 / 85mm T1.3)
EMB Studios
Paul Nordin, Cinematographer

Luma Tech Super 35 Illumina
(135mm T1.8 - PROTOTYPE, rushed in just for this test. A BIG THANK YOU!!!!)
Gregory Mirand, CEO

Charles Pickle, Owner
(Provided tech support for the 135mm prototype.)

Red Pro Primes
(18mm T1.8 / 25mm T1.8 / 35mm T1.8 / 50mm T1.8 / 85mm T1.8 / 100mm T1.8)
Nelson Productions
John Nelson, Owner
Shawn Nelson, Owner

Schneider Cine-Xenar II
(25mm T2.2 / 35mm T2.1 / 50mm T2 / 75mm T2 / 95mm T2)
Illya Friedman, President

Unique Optics Kenji Suematsu Series
(18mm T1.9 / 25mm T1.9 / 35mm T1.9 / 50mm T1.9 / 85mm T1.9 / 100mm T1.9)
Unique Optics
Tim Zarraonandia, Sales & Marketing

Zeiss Compact Prime v1
(21mm T2.9 / 28mm T2.1 / 35mm T2.1 / 50mm T1.5 / 85mm T1.5)
Justin Alpern, Cinematographer + Post

(100mm T2.1)
Illya Friedman, President

What this is and what it is not.
The goal behind these tests is to see how each set of lenses holds up in a "real world" environment. We chose not to shoot charts, take MTF readings, etc. While this would have given us exact numbers by which to compare, it does not tell us how the lens renders the final image aesthetically. We were interested in what nuances the lens brings to the final image. We recognize that this is a more subjective testing methodology, however, we have controlled the testing environments as much as possible so that we can see objective differences in each lens. If you are looking for scientific numbers from a lens test, you are in the wrong place and you are about to be very disappointed. This is a test for cinematographers, by cinematographers. In the end, it is not the numbers that matter, but how the final image is conveyed through the lens, and how easy the lenses are to work with that matter most. Numbers can only point the way.

Originally, we had planned on including Zeiss Ultra Primes as the default standard comparison. Unfortunately, last minute they got pulled as they went out on a rental. While I am disappointed that we could not have the Lamborghini of lenses to use as a bench mark, the reality is that these lenses are in a whole different league. Let's be realistic here. The choice at this level of lens is not between Ultra Primes and any of these lens packages, it is in-between the sets themselves. If you are working at a budget level of Ultra Prime, then these lenses will fall off the list very quickly. That is not to say that they are bad, or shouldn't be used - far from it - they are just in a different category than Ultra Primes.

Ideally, additional time and testing would have been done to determine the sweat spot for each lens so that we could see how they look when at their best. However, we did not have the luxury of that kind of time or testing. Instead we chose to shoot wide open where the lens should be at its weakest and show off all of its flaws, and again at T5.6. If a lens cannot perform well at T5.6, then it is in need of serious help ...

All of the tests were shot at 3:1 compression. (Feature work is typical shot at 6:1 or 7:1.) At these very low compression rates everyone should be able to evaluate the R3D's to their hearts content without worrying that compression is playing a large factor in the image. Yes, 3:1 is overkill- but now you have the best image possible to make your evaluations. To view the R3D's you will need to download RedCine-X Pro.

Let the Testing Commence!
Below you will find the complete 1080p videos that have been compressed in the highest quality possible for web viewing. To provide consistent, unbiased results, everything was metered and and the metadata was set according to the light and color meters. After the videos I have posed a basic rundown of our experiences with each set of lenses. Links to download the 5k R3D snapshots from each test can be found with each review. These R3D's can be used to more critically evaluate each lens, and you can feel free to process it according to your own specific post processing methodology.

I have also included the thoughts and notes that were given to me by Randolph SellarsPhilip A. Anderson, and Jerry Turner. Each of them has spent considerable time working with lenses spanning the range from SLR's to Ultra Primes. They bring a wealth of knowledge & experience to this day of testing and the final writeup. Additionally, their perspectives help to balance out my own, as I know I have a natural bias towards Cooke Lenses.

Interesting Observation About Test 01:
My original intent was to use nets and scrims on all of the lights, but on the day, I decided to not net the Source 4 on the back wall. (Which is in keeping with a "real world" scenario as often times things change on the actual day of principle photography). My intent was to see how the lenses affected the roll off into the highlights. Inadvertently two things happened as a result of this choice. First of all on the longer lenses, the wall became a large source of light and showed off the flare qualities one might get if a large light source or window was in the frame. Additionally, it effects the perception of color of the image when seen as a whole. The key read at 3000k, while the source 4 read at 3400k. (Which makes it more blue in relation to everything else). The over-exposed nature of this bluer source, and its large size in the frame are perceptually influencing the overall perception of the color bias of the image. Even though the color balance of the image as a whole is not really changing. This portion of the frame feels more blue in every wide open shot across all sets of lenses. So I know it is not a function of the lens, rather it is a function of the light.

The good news is that the small chip chart center frame can be used to verify if there are any actual color shifts going on in the image. (Which I have done). Furthermore, to verify that it is the light, not the lens that is the issue, take a look at the skin tone tests from Test 02. Without the large bluer source in the frame, any color cast that the lens set may be exhibiting is much easier to see.

Thoughts About Test 02 from Philip A. Anderson:
As a cinematographer seeing how a lens looks with skin tones and faces is extremely important. Test 2 was designed primarily as a way for people to see the differences in the optics of the lenses while observing a human face.  I would have liked to have done a third test of an extreme ¾ close-up with the 50mm and longer focal lengths, of each lens set, but time constraints required us to forgo that test.

The flare test helped us to see how the lenses might react for flare and ghosts in a strong backlight situation.  A few of the lens sets showed strange shapes and unusual characteristics from the light before the actual source could be seen in the frame.  Ultimately this test helped us understand which lenses we could use for flare effects. If on the other hand, we were out to avoid flare effects, the test gave us an idea of how diligent we would need to be with hard mattes, eyebrows, and grip brother support.

As a final note I would like to discuss the importance of mechanics when determining your lens selection. As a director of photography we are responsible for both aesthetics and making our days. If we use a set of lenses only for their optics without considering their mechanical properties then we are only doing part of our job when considering which lens to use for that project.  For example, If a lens swap requires one additional minute over a mechanically superior lens then over the course of a feature film that may add up to hours of wasted time. 

Thoughts About Test 03 from Randolph Sellars:
The idea behind shooting a lens test outdoors is valid.  It would provide an opportunity to see how different lenses behave in a very common "real world" scenario - shooting day exteriors.  It would have been nice to see color fidelity (flesh tones, tree foliage, etc.), highlight/shadow rendition, edge to edge sharpness, etc. in a setting with lots of depth.  At first, I was concerned about constantly changing light conditions which would make it very difficult to keep all of the variables (color temp, contrast, etc.) consistent.  In order to work fast before conditions changed radically, we planned to shoot the same focal length lens of all brands at roughly the same time.  As it turns out, the weather gods gave us an unusually consistent day - cool color temperature and very overcast.  Actually, it was beyond overcast - it was quite foggy!  These conditions made the middle ground and the backgrounds rather soft and mushy.  Also, the model's face was in flat light as well with no eyelight.  I contemplated adding some light, but was again concerned with the possibility of shifting color temperature and contrast which would affect a fair test.  Since we didn't have an HMI, the point was rather moot.  

The overall results of test # 3 are fairly consistent due to the foggy weather and also consistently boring without much color or light contrast.  Personally, I found it impossible to make any significant evaluation between the various lens brands.  They all looked flat and soft due to the "thick" air and diffused light.  The interior tests (#1 #2) were much more helpful to me in pointing out subtle differences.  I'm glad that we have those to judge by.  I feel that the exterior test was a "bust" due to uncontrollable weather conditions.  I sure am glad we weren't shooting (wasting) film!

Chromatic Aberration:
When I viewed the results from test 01 the first time I noticed purple fringing occurring in the image from a lot of the lenses when wide open. My immediate first thought was, "AH! Chromatic Aberration!" (CA). However, after viewing and reviewing the footage for hours, I'm not entirely convinced that all of the purple fringing is a direct result of CA, and that some if it could be a result of flair. So I consulted with several people on this, most notably Matthew Duclos and the end result was that it is debatable if the purple fringing is 100% from CA or not. So rather then get pulled into a debate about CA, I am going to call it Purple Fringing (PF) for the purpose of this test. The PF is viewable, and thus "measurable" in the results, regardless of if you choose to call it CA or flair.
CA / PF Example

The metadata in the R3D's should be correct. However, I have been noticing that sometimes when I open the R3D's a slight green tint is getting added to the R3D. This tint WAS NOT THERE when shooting, and it was not there when processing out the R3D's with the Redrocket. (I checked and double checked all the metadata before rendering out.) I have NO CLUE as to why this tint is showing up, nor can I replicate the issue in any meaningful way. I have tried different versions of RedCine-X Pro, different computers, different OS's, etc. All this to say that you need to double check the metadata on the R3D's. Color temperature (Kelvin) should be set according to the specs below, and tint should be at zero. The only other adjustment in the R3D's was in the FLUT control for test 03. All lenses were set at T4, and meter readings were taken through out the day. The actual readings varied from 4/10 of a stop under to 7/10 of a stop over. So FLUT has been adjusted in test 03 to compensate accordingly. All of this should be reflected in the RMD files of the R3D's that are provided in the links below.

Okay, really this time, on to the testing ...

Testing Station 01:
Cinematographer: Ryan E. Walters / 1st AC: Jerry Turner

  • Shooting Specs: 5k Full Frame @ 3:1 / 24fps / 800 ISO / 3000k / RedColor2 RedGamma2
  • Goals: Evaluate & Compare: Color Fidelity, Skin Tones, Bokeh, Contrast, Sharpness, Focus Fall Off, Chromatic Aberration, & Breathing.
  • Notes: Light levels were controlled through the use of scrims and nets. Every light was adjusted except for the source 4 that rakes across the newspaper in the background. This was done to see how the lens affects the roll off into over-exposure. Focus was pulled by hand on the lens.  I instructed Jerry to rack between the foreground, mid-ground, and background to emphasize any breathing that may occur.
Testing Station 02:
Cinematographer: Philip A. Anderson / 1st AC: Patrick La Valley

  • Shooting Specs: 5k Full Frame @ 3:1 / 24fps / 800 ISO / 2900k / RedColor2 RedGamma2
  • Goals: Evaluate & Compare: Skin Tones, Focus Fall Off, & Lens Flare.
  • Notes: Light levels were controlled through the use of scrims and nets. 
Testing Station 03:
Cinematographer: Randolph Sellars / Assistant: Shawn Nelson

  • Shooting Specs: 5k Full Frame @ 3:1 / 24fps / 320 ISO / 5500k / RedColor2 RedGamma2
  • Goals: Evaluate & Compare: Exterior high detail scene.
  • Notes: 1 focal length from each set was shot under the same lighting conditions. For example, it is best to compare all of the 50mm with each other. Fortunately there was relatively consistent light levels and color temperate throughout the day.
(18mm T2.8 / 25mm T2.8 / 32mm T2.8 / 50mm T2.8 / 75mm T2.8 / 100mm T2.8)

What is great about this set-
- Overall this is a very well color matched set of lenses*. (Macbeth chart results read on the vector scope.)
- Tack sharp, without being overly sharp. I don't know how to best qualify it, but these lenses have a pleasing sharpness to them that allows the image to be sharp in all the right places, but ever so slightly softer in other areas. [Jerry concurs]
- Small and lightweight.
- Great focus fall off. [Jerry concurs]
- Ultra smooth mechanics (focus and aperture) making them very easy to work with. But they are stiff enough so that they do not slip when bumped.
- Nice bokeh wide open wide open.
- Look like the S4's. Clean and beautiful. [Philip]
- Only set of lenses that was accurate at all focus marks and infinity. [Randolph]
- Best lenses mechanically of the test. [Philip]

What is no so great about this set-
- *The 18mm is a little more saturated, and slightly warmer then the rest of the set.
- T2.8 maximum aperture. [Side note: I do not find this to be an issue, as I have been lighting to T2.8 or T4 at ISO 320 for years. So a T2.8 at ISO 800 is easy. I'm more interested in lighting creatively, than in just getting an exposure.] [Philip adds: T2.8 is a little slow for a modern prime lens.  Modern zoom lenses are exceptionally good, so the additional stop you trade for the convenience of the zoom is lost with these lenses.]
- All of the lenses were not evenly greased from the factory. (It felt like it at least). Although it could be that they need to be worked in more. [Jerry]
- Would prefer more focus marks on the close end of the lenses. [Jerry]
- Would like more resistance in the focus ring. [Jerry, Philip & Randolph]
- Very slight lens breathing. (Seems most noticeable on the 32mm)
- Very slight purple fringing when wide open, not present at T5.6

Misc. Notes:
- Stopped down to T5.6 the bokeh has a noticeable octagon shape to it.
- This is Jerry's favorite set.
- This is Ryan's favorite set. (Was that really a surprise ...)
- If the lenses were not so expensive, they would be Philip's favorite.
- Some additional thoughts from Randolph:
Mechanically, my favorite lens set was the Cookes with the exception of the "loose focus" which could probably be remedied with heavier grease.  As is, I believe it would be a struggle for an AC to make smooth pulls. However, all of the markings appear accurate which is also very important.  I agree that T2.8 is not a problem for me for most shoots.  I find that the super shallow DOF at T2 and below too distracting for most narrative applications.  For narrative lighting purposes, the Zeiss CP's and the Leica's are effectively a T2.9 and T2.8 set due to their slowest lens.  While it is nice to have a T1.4 lens when needed (perhaps available candle light), I wasn't very impressed with most of the lenses wide open with the exception of the Cookes at T2.8.  Although much of the soft contrast that I was seeing wide open was perhaps the result of soft flare coming from the brightly exposed newspaper.  Unfortunately, this made it more difficult for me to compare wide open performance.
Cooke Example:

(18mm T1.3 / 25mm T1.3 / 35mm T1.3 / 50mm T1.3 / 85mm T1.3)
(135mm T1.8 - PROTOTYPE, rushed in just for this test. A BIG THANK YOU!!!!)

What is great about this set-
- T1.3. Super speed aperture. The fastest set of lenses we tested.
- Matching apertures have a similar color cast to them.
- Have a unique look to them. They feel like super speeds, but a tad creamier and warmer.
- Solid mechanics. [Philip]

What is not so great about this set-
- Color cast changes as you change the aperture. This is visible on the monitor as the aperture is closed down from T1.3. At T1.3 they are all noticeably cooler than when stopped down. [Jerry & Philip concur]
- Wide open they really show off their look. They are soft, have purple fringing, and the bokeh becomes very smeary. [Side note: If you need a pristine image wide open, then these are not the lenses for you. But they have a unique look to them, which I would gladly choose as a creative decision for a project].
- Inconsistent color across the range. Long end of the range feels warmer. [Jerry & Philip concur].
- Bokeh changes when wide open across the set of lenses. The smeariness of the bokeh seems to go away around the 50mm range.
- Minor lens breathing.
- Heavy purple fringing when wide open, barely present at T5.6
Illumina Example

(19mm F2.8 / 24mm F2.8 / 28mm F2.8 / 35mm F1.4 / 50mm F1.4 / 80mm F1.4 / 100mm F1.8)

What is great about this set-
- Small, lightweight, and compact.
- Very sharp wide open. (And at T5.6.)
- Consistent color in the mid-range.
- Bokeh is nice, and smooth. [Philip concurs]
- They are like a high quality version of Canon L glass. [Jerry]
- Cine-Mod is done well. [Jerry]

What is not so great about this set-
- Unmatched apertures, slowest F2.8. [Side note: For practical purposes this makes the set a F2.8, but it does give you the option to open up more on a longer lens if you are chasing the light].
- 19mm feels more green, and mid-range feels more warm. The 100mm feels the most neutral.
- SLR glass with small focus throws. [Jerry & Philip concur].
- Not a cine housing. [Jerry]
- Iris gear moves too easily. [Philip]
- These lenses breath the most.
- Very slight purple fringing when wide open (mid-range has the most), not present at T5.6
- No ability to attach a motor to the iris [Philip]
- Weakest lens mechanically. (It's a still lens, what do you expect?) [Philip]
- F-Stops not T-Stops [Philip]

Misc. Notes:
- Phil's favorite set of lenses optically speaking.
Leica Example

(18mm T1.8 / 25mm T1.8 / 35mm T1.8 / 50mm T1.8 / 85mm T1.8 / 100mm T1.8)

What is great about this set-
- Well color matched throughout the set.
- Very sharp throughout the range. (Some might say too sharp...).
- Relatively fast lenses.
- Pristine, clean, modern image.

What is not so great about this set-
- Heavy & large. [Jerry & Philip concur].
- Witness marks do not match on both sides of the lens. (The difference is about 1/3 of a stop).
- Focus marks not consistently accurate. [Jerry & Philip]
- Decorative lens barrel interferes with quick swaping of lenses. [Jerry]
- Mechanics are a bit stiff. [Jerry]
- No hard stop at infinity.
- Slight edge vignetting on 18mm at 5k (See test 03)
- Very slight lens breathing.
- Very slight purple fringing when wide open, not present at T5.6 

Misc Notes:
- Bokeh is cat eye shaped wide open and octagon when stopped down.
- The RPP's are optically modern, cool, and pristine. This lends toward an image that while very accurate, also feels sterile and lacking in character. [Jerry & Philip concur].
Red Example

(25mm T2.2 / 35mm T2.1 / 50mm T2 / 75mm T2 / 95mm T2)

What is great about this set-
- Relatively fast lenses
- Although not perfectly matched apertures, they are close.
- Very clean image when stopped down. [Philip concurs]
- Pleasing bokeh & look. [Philip]
- Decently sharp image. Not as sharp as the RPP's, but not as soft as the Illuminas.

What is not so great about this set-
- Excessive slop in the mechanics of some of the lenses. [Jerry]
- Telescoping design requires the use use of a clip on matte box. [Jerry & Philip concur].
- Fitting a matte box, follow focus, or motors on a lens that telescopes as much as these is extremely frustrating. [Philip]
- Witness marks were confusing. It was difficult to tell which marks indicated which distances. [Jerry]
- Soft when wide open.
- Minor lens breathing, not as much as the Leica's but more then the rest.
- Purple fringing when wide open, very minor at T5.6

Misc. Notes:
- Bokeh is a round circle at wide open & at T5.6
Schneider Example

(18mm T1.9 / 25mm T1.9 / 35mm T1.9 / 50mm T1.9 / 85mm T1.9 / 100mm T1.9)

What is great about this set-
- Relatively fast lenses
- Very sharp
- Bokeh is round wide open (Long end has a bit of a cat eye look to it).
- Color matched well
- Lots of witness marks. [Jerry & Philip]
- Buttery smooth rotation. [Jerry]
- Pristine image.
- Good size. [Philip]
- The combination of price, optics, and mechanics makes them a very attractive buy. [Philip]

What is not so great about this set-
- Slight edge vignetting on 18mm at 5k [Randolph] (See test 03)
- Very slight lens breathing.
- Delrin focus gears that are not quite evenly spaced in some areas causing binding with the follow focus.
- Slight purple fringing when wide open (feels like more then the Red Pro Primes), not present at T5.6 

 Misc Notes:
- Stopped down the bokeh is octagon.
- The Unique's are optically modern, cool, and pristine. This lends toward an image that while very accurate, also feels sterile and lacking in character. [Jerry & Philip concur].
Unique Example

(v1: 21mm T2.9 / 28mm T2.1 / 35mm T2.1 / 50mm T1.5 / 85mm T1.5 / v2: 100mm T2.1)

What is great about these lenses-
- Small, compact, and lightweight.
- Well color matched
- Tack sharp [Jerry & Phil concur]
- Very accurate & pristine image
- The CP v1's are extremely smooth in rotation.
- TONS of witness marks. This is how a lens should be marked. [Jerry]

What is not so great about these lenses-
- Unmatched aperture effectively makes this set a T2.9
- Very slight lens breathing.
- Purple fringing when wide open, not present at T5.6

 Misc Notes:
- Perfectly round bokeh when stopped down to T5.6
- The CP v1 & v2's are optically modern, cool, and pristine. This lends toward an image that while very accurate, also feels sterile and lacking in character.
- The mechanics of these CP v1, and the 100mm v2 were great. This is in stark contrast to the CP v2's that I have used. That set was extremely stiff. From what I've heard from others that have used the CP v2's. this seems to be a common occurrence. The 100mm CP v2 we had for the test did not exhibit the stiffness of the other CP v2's I've used.
- I do have a strong affinity for Zeiss, as it is my lens of choice when I'm not using Cooke.
- While we were using the CP v1's the idea of switching the mounts on the CP v2's makes them an interesting choice when considering how it opens your rental possibilities to Canon DSLR clients.
Zeiss Example

I think a comment Philip made sums it up best: "What this test shows is that we have reached a point at which technology has allowed us to reliably make the Ford Escort of lenses." Are any of these lenses on the level of Ultra Primes? Nope, not even close. But they all perform decently at a fraction of the cost. Philip, Randolph, and I are all of the opinion that we would gladly choose each set of lenses for different projects depending on the needs and look of the project. Each set has its objective positives and negatives that will subjectively matter more or less depending on the situation and personal preference.

But there has to be a winner - doesn't there? Well, if you are looking for one winner that stood out from all the sets we tested, then it would be the Cooke's. The general consensus on the day of the test was that the Cooke Panchro /i's were the best all around lens package. They were the most uniform in consistency, had the best skin tone reproduction, covered 5k from 18mm and longer, the focus marks were accurate, and the build quality was very solid. That seemed to be the general consensus from what people were saying throughout the day of testing. (I freely acknowledge that I have a bias for Cooke glass, as it is my favorite. Which is why I'm reiterating that it was a general consensus - not just me saying so from a biased perspective).

I hope you have found these results helpful as you consider your next lens rental or purchase. Thanks again to everyone who made this test possible!

Until Next Time - Get Out There And Shoot!
Ryan E. Walters, Cinematographer

See Older Posts...