This page describes the state-of-the-art image recognition feature in Magnolia EE. Magnolia image recognition extends the search capabilities of the Find bar by automatically recognizing and tagging the content of image assets, allowing you to get more relevant results when searching.

What's in an image?

How can Magnolia's image recognition feature help your content editors? Consider the following example image that depicts the Mount Bromo volcano, a popular tourist attraction in East Java, Indonesia.

The image is used as a signature image for the Island hopping in Indonesia example tour in Magnolia's Travel Demo and is stored as an asset in the dam workspace.

Imagine the editor is at the beginning of the writing process. Only the tour's title Island hopping in Indonesia has been chosen and now an image with a volcano would be useful to accompany the tour's description.

  • Would searching for the term volcano in the assets produce a result like this image? 
  • What would assist the editor in deciding whether an image is suitable for the story? 

With image recognition and tagging

Magnolia's image recognition feature automatically tags image assets in the dam workspace, helping you obtain a more informed view of the content of each asset stored in the workspace:

The tags are then available for use in any other Magnolia content app or your custom content apps (the Assets app is shown below):

Integration with Find Bar

Magnolia's image recognition and tagging make it easy to search for specific images among your assets:

Without image recognition

Without image recognition and tagging, image asset(s) may not be available in a particular context:

The editor would have to rely on other ways which either may not be as effective or may not provide an answer at all. 

Direct visual inspection

The most direct way to find out would be inspecting images visually in the Preview window of the Assets app:

This approach could work for a small number of image assets stored in the dam workspace but would be tedious and time-consuming if you had hundreds or thousands of assets.

File name

A way to go might be relying on image file names. The image above is stored under the file name flickr_indonesia_michael_day_2564067897_8d17df6972_b. If the editor decided to search for volcano either in the Assets app or in the dam workspace through the JCR Browser app, no result would be returned. Searching for indonesia in these apps would return the image but the file name itself doesn't guarantee any direct relationship of the image content with the volcano idea the editor is working with.

Image metadata

The editor could resort to looking at image metadata (when available). For the flickr_indonesia_michael_day_2564067897_8d17df6972_b asset, this approach also turns out to be unusable mainly due to the following two reasons:

First, the image that's stored in the dam workspace is a reduced version of the original photo and contains only technical data, not image descriptions or labels.

Second, the original image's metadata in the Flickr image repository service specifies only the type and the setting of the photographic instrument when the image was taken ("Camera Model Name", "F Number", "ISO", "Focal Length" ...), but provides no additional clue as to the actual content. 

While it is true that in some image repositories you may send a REST request to the repository's API to get specific image data, such as image description tags or image labels, which can help you identify the content of an image (see Flicker's flickr.tags.getListPhoto method), access to a service like this may be restricted for example by requiring an API key. 

Technical overview of Magnolia image recognition

The image recognition feature is provided by the Image Recognition module which assists the Periscope module in the role of the back-end for the Find bar. By default the image recognition itself is not delegated to any recognition service.

Recognition services

Magnolia gives you the freedom to decide which image recognition service you want to use.

  • The local image recognition service uses the ImageNet image database, Deeplearning4j algorithms and the VGG-16 image recognition model.
  • If you need a more advanced image recognition solution to generate a more refined set of image tags, you can configure the feature to use the Amazon Rekognition. For details see the Image recognition via Amazon Rekognition page.
  • The Image Recognition module can also be configured for use with any other image recognition service through the ImageRecogniser interface. See for example:

Installation and configuration

For installation and configuration information, see the Image Recognition module page.

When is recognition executed?

The image recognition and tagging action is usually executed during the startup of the author instance, with the following provisos:

  • The images that have already been tagged are marked as such using a JCR poperty called lastTaggedDate and filtered out from any additional tagging process.
  • Image recognition and tagging is executed again:
    • For every new image asset uploaded to the dam workspace.
    • If the user edits an image asset and closes the edit dialog by clicking the Save Changes button.

The image recognition feature is available only on author instances.

Image credits

Michael Day,

#trackbackRdf ($trackbackUtils.getContentIdentifier($page) $page.title $trackbackUtils.getPingUrl($page))
  • No labels