Image Annotation Application

Overview:

Photographers, ad & film studios and their clients capture & process a large number of photographs on a daily basis. These photographs require professional retouching using image editors. The producer or the owner of the photo-shoot, the clients and the photo manipulation experts (retouchers) currently communicate using traditional emails and/or document sharing platforms. For a business or line of work that depends heavily on visual representation, text based communication was proving to be ineffective to relay client/producer requirements to the retouchers due to the assumptions that had to be made by the receiving party.

With the advancement in web technologies, a web based image annotation application was demanded that would be SaaS enabled and provide the studios & their clients with a platform to allow them to represent their requirements on images as visual indicators which would be easily picked by the retouchers. The retouchers would then view the visual indicators, make the required changes locally and update the revisions on the application. The client/producer then shall be able to view all the revised versions of an image and approve the required one. There would be no more massively long email or chat threads’ trying to convey what requires being touched or re-touched on images as the images would be available to be annotated in real time with versioning.

Mission Critical:

The following critical guidelines and high level requirements had been outlined which must be met by the proposed solution to meet the business needs of the studios, their clients and the retouchers to collaborate on the projects they are working on while seamlessly and get the communication flowing quickly:

• Allow studios to create their own separate instance where they can maintain several studio projects under a single account & share them with different customers & retouchers

• Annotate any uploaded image using variety of shapes, colors & text while identifying each correction through a unique marker

• Start a conversation & post against a marker to communicate with the participating project members

• View all the versions of the image right from the original to the final along with the comments

• Approve or reject manipulated versions of the images & be notified of the actions

• Super-fast rendering of high resolution images along with the super imposed annotations

• Support over 100 common & proprietary image formats used by the studios for image capture and editing

• Use latest web & cloud technologies available to make it as future proof as possible and minimize frequent updates

Challenges:

A web based prototype had already been attempted for the image annotation software by the client but it had serious performance issues and could not meet the critical requirements which were expected to be met for a successful launch. The prototype however aided in getting the UI done for the web application which was fully responsive & could be used in any device without any screen size limitations. Responsive design was yet to make an impact & adopted for full scale implementation as it was still in its nascent stage. There were no ready to use frameworks available & everything was required to be custom coded. The following are considered to be the major challenges for a successful implementation of a SaaS based online image annotation system:

• Studios, professional photographers and retouchers used a wide variety of image formats for manipulation, storage & sharing. With 100+ open & proprietary formats in use, it is very challenging to support all of them. Moreover, some of the RAW high resolution image format had image sizes in GBs and pose a problem while loading them on the browser for annotation.

• The studios are required to be able to add customers & retouchers independently to each of the projects. Apart from individual permit, they are also required to be provided explicit permissions of the actions they are allowed to perform. Also, it was pretty much possible that the same customer could be associated with multiple studios who all had instances on the image annotating system.

• The application was expected to be mobile device friendly & this imposed a restriction on integrating a flash based image editor. iOS devices did not support flash and all 3rd party integrations available for image manipulation & annotation were built on flash and aimed on computer use primarily. This required a visual editor and image manipulator toolkit to be custom built.

• Being a web application on the cloud and handling of image manipulation, it was necessary that the application made efficient usage of the allocated hardware resources to ensure smooth performance when handling multiple high resolution images across studio instances.

Solution:

The image annotation web application provided an online solution for information sharing and image proofing that allows the customers, producers and the collaborators (retouchers) to communicate visually so as to receive the required output. All the three types of users have their own account profiles and are able to create new accounts as desired. Only the producer of the studio is required to purchase the application and is billed monthly. Based on the membership plan subscribed; each producer shall be provided with a separate application instance accessible from a sub domain with allocated cloud storage and facility to create a preset number of customer/retoucher accounts.

The producer or customer ideally uploads the photographs to the application. The uploaded image is automatically converted to JPG format as per the resolution specified by the user. The image is then loaded onto an HTML5 canvas element to prep the image for manipulation & annotation. The customer provides indications of what is to be edited on the photographs through the various custom coded visual annotation tools that are available. Annotations are uniquely marked for easy identification and can be colorized as well. Comments against each marker can be submitted to explain what exactly is required to be done with the image. Once the comments are put in, the retoucher is informed who checks the comments & edits the photograph as required and uploads them for verification to the producer/client. The images are automatically versioned against the last upload to allow all the participating members view the progression of the updates performed. Summarized views of the images of a project & the project themselves are also available. The producer/client can either mark the uploaded pictures for more revisions by marking them with new indicators or approve them. The final versions of the images can then be delivered to the customers external to the application.

The application also has an automated feed applicable for each studio instance which depicts the latest comments and uploads performed against each image of the projects associated with the account. It provides a continuous stream of information to the end user who has access to the concerned projects to be informed of the latest updates happening with the projects.

Implementation:

The image annotation system was implemented by Allerin using the following technologies and delivered through the SaaS model:

• Language - Ruby 1.9.2
• Framework - Rails 3.2.11
• Database - MySQL 5.5
• Interface - HTML5 & CSS4
• iWeb Server - Phusion Passenger 3.0.19 with Apache
• RubyGems/Plugins - mysql2, sass-rails, coffee-rails, aasm, uglifier, jquery-ui-rails, active-model-email-validator, acts_as_commentable_with_threading, acts_as_list, apartment, rails, attribute_normalizer, awesome_nested_set, aws-s3, s-sdk, best_in_place, client_side_validations, cocaine, date_validator, delayed_job_active_record, devise, exception_notification, friendly_id, kaminari, liquid, plupload-rails3, rmagick, paperclip, roadie, jquery_evol_colorpicker, jquery_expander, jquery_gzoom, jquery_infinitescroll, jquery_jcanvas, jquery_livequery_min, jquery_mousewheel, jquery_placeholder, jquery_qtip, jquery_ui_core_min, jquery_ui_draggable_min, jquery_ui_widget_min, slimScroll_min
• JavaScript Framework - jQuery-1.7.1

Risks:

• With the restriction of flash use, emerging HTML5 web technologies was chosen to be implemented. HTML5 was still in its initial stage and hence browser support was not uniform. Moreover not all elements were supported by all the browsers and therefore much of custom coding was necessary to deliver the features. The custom features mimicked the anticipated HTML5 support which ensured that the users did not see any deviation in how the features operated through the interface.

• The application had to be marked as browser compatible with specific versions only, mostly the latest ones, for the application to work. It was a drawback in the sense that the end user had to actually upgrade or have a system the supported the use of the latest browser versions. The offset was however immense in the area of performance & usability which the end users greatly appreciated during launch. With all future browser versions to be compatible with HTML5 and its native features, the experience will only get better.

• Most photographs used by the studios are high resolution images required substantial time to process requests from the web browser and render it on the browser. As such, the entire image storage and request processing was required to be offloaded to a third party cloud storage. This helped in speeding up the image processing and display & application response time as a whole while maintaining high redundancy.

• A payment processor was implemented to perform both one-time & recurring transactions based on the credit card details available. During the implementation phase itself, payment processor revised their entire payment processing workflow and updated their APIs. Even though documentation was provided, this resulted in having to come in grips with the new payment structure to be implemented and feasibility studies had to be re-run to ensure a successful implementation. Subscription based payments procedures were updated within the payment processor and as such, revisions were required to be made in the application to accommodate Recurring Billing.

• Images were required to be automatically resized to specific resolutions during upload. The resolutions were more suited towards landscaped and wide screen images. Issues regarding incorrect resizing of portrait images were noticed. The resizing algorithm had to be re-worked so that images were compressed or escalated proportionally as per their aspect ratio and initial resolution.

Results:

The image annotation web application with its simple responsive UI and HTML5 implementation provided an excellent platform for the studios to bring together their customers and the retouchers so that text based communication can be replaced with visual indications which can be easily understood. The workflow of the application is kept close to the real world so that the producers are able to create projects and each project containing several takes. Each take then contains photographs associated with the concerned take that requires retouching. Customers are able to mark images visually & with text, what and where they are to be retouched; thereby forming unique automated markers for each change requested. Retouchers are then able to view the indicators & comments, make the revisions locally & upload the latest retouched image.

Versioning is done automatically and the project participants can view all the images being worked upon collectively in a wish list. The wish list automatically maintains the list of the images of take against which retouches have been requested & submitted. Changes can be approved or rejected as required. An activity feed on the home screen ensures all the updates made to a project the user is involved in is available to be viewed. Users are also able to comment on any project, take or image they can access thereby keeping communication flowing at all times.

The application provides a one in all web solution to meet the requirements of the studios to collaborate and communicate effectively with their customers & retouchers to meet their objectives. Implementation of the latest HTML5 & Rails technology ensures minimum maintenance and consistent high performance operation. The users were delighted at how fast the application performed when handling high resolution images. Moreover there were no restrictions on the image formats and as such they did not have to keep swapping the image formats and keep tweaking any configurations to maintain the image quality of the photo being uploaded. The images automatically converted to JPG to ensure high compression with minimal loss. The usage of HTML5 canvas element provided the much needed flexibility to operate the image annotation application on any mobile device with a modern browser. Since the launch, the application has seen numerous studios and professionals signing up with the available PAID plans to manage their projects to ensure smooth deliveries & cut through the chatter produced through massive email threads & inefficient messaging systems. With hundreds of users & GBs of images being processed on a daily basis, the application has maintained perfect balance between performance & stability which can be accounted to the stringent code review and testing processes included within the development sprints.