Martin Douglas Hendry

What is it?

A recent Apple patent application proposes a unique technique using optical stabilization technology (OIS) to increase the output resolution of a smartphone camera.

This is to be achieved by first capturing additional information about an image subject through multiple optical sampling instances, coinciding with the systematic movement of the OIS mechanism.

Secondly, the varying information from these sampling instances is then combined using an algorithm (which accounts for the movement of the OIS mechanism) into a “super-resolution” image.

This technology could be utilized in the next generation of iPhones to achieve increased image detail without having to significantly modify the camera module in the device. This could theoretically allow an iPhone camera to generate higher resolution images, with little impact on the iconic appearance/form of the device.

Is Apple Innovating In Order To Avoid Compromise?

This may be seen as evidence of Apple’s culture of design innovation – where they have developed an alternate (and completely novel route) to maximizing the perceived quality of the output of their imaging devices. It can also be seen as a calculated way of avoiding sacrificing the form factor of their iPhone devices in order to achieve comparative optical outputs to its competitors. (Samsung Galaxy S5 has a 16MP camera – compared to the iPhone 5S’ 8MP “iSight” module).

This decision may have a bearing firstly on the internal room within the device (engineering concerns), where a larger camera might mean less space for a battery/processer etc. – thus significantly limiting the device. Secondly such a decision may be influence by the external form of the iPhone and users relationship with the iPhone as an iconic design artefact. Here significant change/ compromise in the form may be regarded as wholly unacceptable to Apple and – indeed – users. Typically, improvement would be achieved either through including a larger sensor within the phone, or through more complex (and typically larger) lens modules.

The proposed “Super-Resolution” technology could potentially produce an image of comparative quality to much larger camera-centric devices (though no output specifications are provided) – without the requirement for significantly larger camera modules within the smartphone itself.

The Trade-Off and a Requirement for Informed Use

Theoretically however, using the OIS mechanic in this way would negate the use of image-stabilization for its traditional purpose – negating camera movement. Thus in use – “Super-Resolution” images would have to be made when the camera (and ideally the subject) are stationary.

It is also of note that iPhones do not currently include OIS technology, so its inclusion alone suggests continued significance of digital imaging capabilities within smartphones and of iPhones desire to keep up/beat camera-centric devices such as the Nokia Lumia 1020.

The patent outlines the unique use of this technology as:

“A system and method for creating a super-resolution image using an image capturing device. In one embodiment, an electronic image sensor captures a reference optical sample through an optical path. Thereafter, an optical image stabilization (OIS) processor to adjusts the optical path to the electronic image sensor by a known amount. A second optical sample is then captured along the adjusted optical path, such that the second optical sample is offset from the first optical sample by no more than a sub-pixel offset. The OIS processor may reiterate this process to capture a plurality of optical samples at a plurality of offsets. The optical samples may be combined to create a super-resolution image.“

Images provided in the patent application suggest that this will likely be a mode accessed through the camera UI by the user toggling a button (see later in this blog for illustration).

This suggests that the user will be made aware of the benefits of each function exclusively (e.g. for “super resolution”, ideal image subjects would not require image stabilization e.g. landscapes as opposed to “moving targets”) and would be guided toward correct/ ideal use.

Fig. 7 from the patent application suggests that in use the “super-resolution” image mode would warn users to ensure that images they capture are from stationary perspectives

Why is it important?

Apple’s Aggressive Defense of Intellectual Property and Refusal to Compromise

Apple as a company have largely defined the culture of smartphones as the design leader. This is reflected in their ongoing success – with 43.7million iPhone 5s and 5cs sold in the first quarter of 2014 (although Samsung have recently surpassed this total managing to ship 60 million units). Despite this, iPhones are widely regarded as the archetypal smartphone.

A patent war continues to rage on between Apple and Samsung – the former claiming that Samsung have infringed a number of intellectual properties that have become commonplace in smartphone UX across the board (most notably “swipe to unlock” or “square icons with rounded corners” amongst others).

Whilst it might seem a moot point to enforce (as this kind of function is common across all smartphones today), Apple argue that such design elements are (1) their intellectual and innovation properties, and (2) being freely used in the design language of other smartphones today. Thus, most smartphones, and our relationship with them through use are arguably all underwritten by Apple intellectual properties and innovation. It is likely that when granted this patent application (OIS for “Super-Resolution” Imaging Mode) will be defended with similar aggression.

As such, we could argue that this way of maximising the imaging potential of existing camera modules (albeit with the addition of a OIS mechanism) may be undertaken in order for Apple to avoid compromise and maintain a holistic consistency in the design of their future devices.

As such – here we may find that the development and innovation of imaging technology within the iPhone camera must adhere to the cultural values of what an iPhone is (a sleek, compact, powerful, metal device) and equally go to great lengths to avoid what it is not (other advanced imaging phones with large visible components / uneven weight distributions).

How might this affect the social camera?

iPhone Photography to Apple is More Important as a Culture than a Technology

iPhones (and smartphones in general) have become the most widely used cameras in the world – according to Flickr statistics. This is something that was touted by Apple in April 2013 with their advert for the iPhone 4S – where they lauded the iPhone 5 as the world’s most popular camera.

This remains to be the case today – as can be seen in the current statistics page on Flickr (as of May 2014 the iPhone 5 remains the most popular, followed by the 5S and 4S). As a consequence, the iPhone has been established for the past year as the pre-eminent camera of our times – and arguably the first that is not a dedicated imaging device.

Within the above patent – Apple point toward not only an innovation which maintains the form factor of their device (thus maintaining the iPhone as a design archetype) – but also through presenting a culture of use that will surround how “super-resolution” images should be created. This is discussed in further detail in the following section.

What is firstly important is the extent to which Apple appear determined to maintain the integrity of the iPhone as the archetype for snapshot photography moving forward – by maintaining its form of the device so whilst improving the detail captured within images and diverging from competitors (avoiding larger camera modules / sacrificing other functions due to space limitations).

The risk here is that a significant change to the camera module would have a “knock-on” effect on the perception of the iPhone as a camera within the device as a whole, culture, alongside its competitors etc. This kind of change can of course be positive – and even designed into the innovation itself. It appears that this may likely in how Apple appear to be designing access to the “super-resolution” function.

Apple Wish (and Are Largely Able) to Define Dominant Cultures of Use / Performance Through Design

In other devices increased resolution (or detail) in image outputs is a passive feature, images simply have increased detail because of improved sensors.

Through design, Apple appear to be engendering the heightened sensitivity of “super-resolution” images through a mode accessed in the camera UI (rather than a function that is automatically activated, when appropriate for use).

As stated previously, this is likely due to the issue that using the OIS components in this way negates their ability to be used for traditional image stabilization.

It also means that when people are looking to create images in this way they will be guided (by design) to the best way in which this is achieved – namely on stationary subjects, and when the camera is stationary itself (as seen in Fig.7 the camera may not allow images to be made when the camera is moving – negating the risk of “failed” super resolution images.)

Thus, where in other cameras the benefit of increased image sampling (and thus less satisfactory images)  is something that is not guided in the design of camera use but rather automatic – Apple may in their design engender a new performance of photography that suits higher resolution sampling.

This will lead to more beneficial image outputs (by ensuring that the user is utilizing the function effectively) as well as an appreciation of the function as distinct from the generic functional resolution of the camera. If used, it is likely that in the future guidance will also feature heavily in the marketing of the iPhone thus notifying users before purchase of distinct and appropriate usage.

The patent application includes an indication of how the function would be accessed by the user through the UI of the native camera app on the iPhone. This suggests that it will be marketed to users as a unique function – and will also involve informed usage

Super Resolution and the Social Camera

As a consequence – whilst the camera and “super-resolution” images may be of lesser intrinsic quality when compared to competitors with dedicated hardware – (if utilized as suggested). Apple potentially stand to benefit, not only by maintaining the iconic form-factor of their device, but also by engendering new mode – and appreciation of –  photography through the marketing and interaction design of user access to their “super-resolution” function.

This blog post is a work-in-progress. Additional content and supporting material for these views will be added over the course of time as more information comes to light. Thanks for reading.