Feeds

The RSS Aggregator module displays external feed content on a Magnolia page (feed aggregation) and generates feeds from Magnolia content (feed syndication). Displaying aggregated feeds provides continuous fresh content on your site, which encourages regular crawling by search engines. RSS feeds can also add a sense of timeliness and community involvement to your site. Feed syndication increases your content exposure. This in turn generates more traffic and backlinks that improve your site rank.

Planet feeds enhance standard feeds by collecting feed statics. Planet Magnolia  is an example site that uses the module to collect blog posts about Magnolia.

Feed aggregation

A feed aggregator displays external feed content on a Magnolia page. This provides continuous fresh content and encourages regular crawling by search engines.

Creating an aggregate feed

In this example we collect three individual RSS feeds into one aggregate feed:

• Yoast: http://feeds.yoast.com/yoast
• Moz Blog: http://feedpress.me/mozblog
• Search Engine Land: http://feeds.searchengineland.com/searchengineland?format=xml

These have a common topic – search engine optimization (SEO) – so they are good candidates for aggregation. A reader interested in one feed is likely to be interested in the others too.

To create a RSS aggregator, in the Feeds app:

1. Add a new feed.
   
   • Name: SEOBlogs. This is the internal name of the aggregator.
   • Title: Feed title.
   • Short description: Feed description.
   • Planet Feed: Check to mark this feed as a planet feed.
2. Add the feed URLs from above.
   
   • Title: Leave blank to get the title from the external feed title
   • URL: Feed URL.
     Image: Select an image from the DAM. 60x60 px works best. The image is displayed in the Planet Feeds component. Hackergotchi have become popular and are supported.
3. Add filters to refine content.
   
   1. Option: Select from a pre-defined list of options, Category, Title, Author or Description, commonly found in feeds.
      
   2. Regex: Type a regex pattern to meet the condition and option in the first two boxes.
   3. Condition: Select an AND, OR or NOT condition to include or exclude content.

Importing feed data

To import data for:

• All feeds click Import all feeds now.
• A single feed click Import feed now.

See also: Scheduling an automatic feed import.

Displaying a feed on the site

To display the aggregated feed on the site:

1. Go to the Pages app.
2. Edit a section page such as /demo-features/aggregation-paragraphs/rss-aggregation.
3. Add the RSS Combined Feed or RSS Feed List component in content area.
4. In the component dialog, select the SEOBlogs feed you created above.

Scheduling an automatic feed import

Importing feed data manually is not a long-term solution. You should configure an automatic import schedule. Choose the update frequency depending on how often feeds have new content. For example, if a blog gets one post a day then importing once a day at 6 a.m. is enough.

Setting automatic imports for all feeds

You can schedule automatic imports for all feeds in the Configuration subapp:

1. Periodic import: Select Enabled. You can switch off the automatic import at any time by disabling it.
2. Cron: Define a Quartz Cron Pattern to create a suitable import schedule. For example, the pattern 0 0 6 * * * imports data daily at 6 a.m. See Cron Maker for help.

The input in the app is stored in property nodes in the module configuration in /modules/rssaggregator/config/automatedImport and /cron.

Setting automatic imports for single feeds

Magnolia 5.3+ / RSS Aggregator 2.3+

You can configure automatic import settings for each feed in the Import Settings tab of the edit dialog:.

1. Settings to use: Select Use different settings for this feed to override the global settings. You can revert to global settings at any time.
2. Periodic import: Select Enabled. You can disable the settings at any time. Disabling stops automatic imports completely, the global settings will not take over unless Use global settings is selected.
3. Cron: Define a Quartz Cron Pattern to create a suitable import schedule.

Feed fetcher configuration

Feeds are imported by a fetcher class configured in /modules/rssaggregator/config/fetcher.

Properties:

• class:  FastRSSFeedFetcher retrieves RSS feed content over HTTP. Fetches all feed channels defined in an aggregate feed.

Feed data storage

Feed data is stored in the rss workspace. Configure a custom JCR browser to view the content in the repository. You can download the feedsJCRBrowser configuration XML files if you prefer not to configure from scratch.

Here's the SEOBlogs example data in the JCR.

Structure:

• <RSS aggregator name>
  • data: Data that was retrieved from the Internet. 
    
    • channel-<number>: A channel is created for each RSS feed.
      
      • entry-<number>: One entry in the feed such as one blog post.
        
        • author
        • channelTitle
        • content
        • .....
      • description: Feed properties retrieved from the internet.
      • link
      • .....
  • automatedImport: Internal feed properties.
  • cron:
  • .....

Feed components

There are two standard feed components:

• STKCombinedFeed: Combines all feeds in a channel and renders them sequentially. You can set the number, sort order and character limit of the entries.

• STKSingleFeed: Displays a defined number of entries for each feed in the channel. The external feed title is used if no internal title is set. You can set the number, sort order and character limit of the entries.

The components definitions are configured in STK > Template Definitions /components/teasers/stkCombinedFeed and /stkSingleFeed.

Here's the stkCombinedFeed definition:

The model class and template script are the important properties that determine content in the different components. All RSS model classes extend  AbstractFeedModel that provides the business logic to retrieve a defined feed and it's data from the rss workspace and supports:

• Entry sorting by title or publication data.

• Ascending and descending entry sorting.

• Maximum results property. Default is 20.

• Search capabilities

Any custom model class should extend AbstractFeedModel.

Planet feeds

Planet feeds have all the features of standard RSS feeds but also store additional data that is used to create feed statistics. For example, a planet feed will tell you the number of posts by an author. You can use this information in a Planet Statistics component.

Creating a planet feed

To create a planet feed check the Planet Feed box in the feeds dialog when creating or editing a feed. Feeds can be marked as a planet feed at any time. When you change a standard feed to a planet feed, re-import the feed data before generating the additional planet data (described below).

Planet commands

The module includes two custom commands that generate planet data. These are configured in /modules/rssaggregator/commands/planet/generatePlanetData and /collectPlanetStatistics.

• PlanetDataGenerator command is executed when planet data needs to be updated. Here's what the command does:

  • Generates data for a planet feed.
  • Processes all feeds in the channel and creates an archive of feed items if the feed is marked as a planet feed.
  • Stores individual feed items in the planet archive and assigns author, channelTitle, title, content and description properties. Posts without these properties are not stored in the archive
  • Builds checksums to eliminate duplicates. New items are only added if they don't already exist in the archive.
  • Ensures that the maximum nodes allowed under a single node is not exceeded.
  • Respects any filters set.

• CollectStatisticsCommand command is executed when planet statistics need to be updated. Here's what the command does:

  • Collects and generates statistics from feeds in a planet archive.
  • Stores the collected statistics in the JCR tree.

Scheduling a planet update

Use the Scheduler module to schedule planet updates. The update schedules are stored as a standard scheduler job in /modules/scheduler/config/jobs/generatePlanetData and /collectPlanetStatistics . The jobs execute the planet commands that do the actual work. 

To schedule a planet update:

1. Set the active property to true.
2. Define a Quartz Cron Pattern to create a suitable schedule. For example, the pattern 0 0 6 * * * imports data daily at 6 a.m. See Cron Maker for help.

There is no way to manually generate the planet data. You can use the pattern 0 0/5 * 1/1 * ? * to generate the data five minutes after changing the settings. 

Displaying planet components on a site

To display planet components on a site:

1. Make the components available in any area. You can make them available in a specific page template or globally for all pages in the template prototype. The planet components are not available by default.

2. In the Pages app edit a page where the components are available.

3. Add the components to the page.

Here are the planet components using our SEOBlogs example data.

• Planet Feed: Displays feed posts, including images. Editors can define the length of the text, and pagination is available.

• Planet Statistics: Shows a list of authors with statistics (number of posts).

Planet data storage

Planet data is used to generate content for planet components. The data is stored in the rss workspace. View in the data in your custom JCR browser.  

Here's the SEOBlogs example after marking it as a planet feed, re-importing the data, and running the planet commands.

Properties:

• <aggregator name>
  
  • planetData: The GeneratePlanetData command reads feed data and stores it here.
    
    • posts-<number>: All posts from all feeds. First all entries from the first feed, newest entry first. Then all posts from the second feed, and so on.
      
      • entry-<number>: One entry in the feed such as one blog post.
        
        • author
        • authorLink
        • channelTitle
        • checksum1/2: The module uses checksums to handle duplicate entries. Because feed data is deleted and recreated on every run, there is a high probability that a subsequent run will include entries that were contained in a previous run. Some entries may only have changed slightly, for example a different publication date. To avoid duplicates, the PlanetDataGenerator command uses checksums for each entry. Two checksum properties are generated. If an archive node with one of the checksums exists, no data is stored for the new item and an INFO level entry is written in the logs.
        • description
        • hidden: Set to true to hide the entry from the planet feed. Useful for hiding spam entries.
        • link
        • pubDate
        • rssLink
        • title

Planet statistics storage

Planet statistics are generated from planet data and stored in the rss workspace. View the data in your custom JCR browser.

Here's the data for the SEOBlogs example.

Properties:

• <aggregator name>
  
  • statistics: The  CollectPlanetStatisticscommand extracts statistics from the /planetData node. This node is deleted and recreated on every run of the command.
• • • authors: All authors from all feeds.
      
      • author-<number>: Each author is allocated a number. 
        
        • counted-posts: Each child node is a reference to a post in the feed.
          
          • <post UUID>
          • ...
        • author
        • blogLink
        • feedLink
        • postCount: Number of posts by this author in the aggregate feed.

Configuring how long to keep planet data

Planet data is generated and stored for the last 3 months by default. You can configure the time period for which the data is retained in /modules/rssaggregator/config/planetOptions/lastMonthsIncluded.

Properties:

• planetOptions
  • lastMonthsIncluded: Number of months. Default is 3.

Planet components

Planet component definitions are configured in /modules/rssaggregator/templates/components/.

Planet Feeds component

The Planet Feed component is an enhanced version of the STK Combined Feed component and can only be used with planet feeds.

The component:

• Displays the entries of all feeds in a channel with the latest appearing first.
• Uses the HTML of the source entry. This means that images and links are preserved and displayed.
• Allows pagination. 

The component definition is in /modules/rssaggregator/templates/components/planetFeeds.

Planet Statistics component

The Planet Statistics component uses the planet statistics data. The component:

• Displays a list of authors.
• Displays subscription links to the author's individual feeds (icon) and text links (author name) to their Website.
• Orders the list by post frequency with the top contributors appearing first.
• Allows editors to add a title and subtitle, define the number of authors to include, and select whether links and the post count should be included. 

The component definition is in /modules/rssaggregator/templates/components/feedStatistics.

Planet Authors component

The Planet Authors component allows users to subscribe directly to the external feeds on a Magnolia page. The links take the user to the external sites.

The component definition is in /modules/rssaggregator/templates/components/feedSubscriptions.

Feed syndication

Feed syndication increases your content exposure. This in turn generates more traffic and backlinks that improve your site rank.

Feed generators

Feed generators generate RSS feeds from Magnolia content and imported content stored in the rss workspace.

Four feed generators are registered in /modules/rssaggregator/config/feedGenerators.

• rss: RSSModuleFeedGenerator generates a SyndFeed based on standard aggregation feeds.  

• planet: PlanetFeedGenerator generates a SyndFeed based on planet aggregation feeds.

• category:  CategorySyndicator is registered by the Categorization module. It generates a SyndFeed based on articles tagged with categories.

• templateContent: PageSyndicator is registered by the Standard Templating Kit module. It generates a SyndFeed based on template categories and subcategories.

Servlet

FeedSyndicationServlet writes an XML feed to the response. Based on the request parameters, the feedGenerators configuration (above) is resolved and used to generate the XML feed. The content of the feed is written to the response with the appropriate character encoding.

The servlet is registered in /server/filters/servlets/FeedSyndicationServlet. 

Virtual URI mapping

The syndication components use virtual URI mappings to redirect the generated feeds. The mappings use regular expressions and are called by the feed generator classes to render appropriate content in the XML feed.

• Three mappings are configured in the RSS Aggregator module in /modules/rssaggregator/virtualURIMappings.

  Node name	Value
  modules
  rssaggregator
  virtualURIMapping
  rssFeeds
  class	info.magnolia.cms.beans.config.RegexpVirtualURIMapping
  fromURI	/rssFeeds/(.*)
  toURI	redirect:/rss/?feedPath=/$1
  planetFeeds
  class	info.magnolia.cms.beans.config.RegexpVirtualURIMapping
  fromURI	/planetFeeds/(.*)
  toURI	redirect:/rss/?feedPath=/$1&generatorName=planet
  categoryFeeds
  class	info.magnolia.cms.beans.config.RegexpVirtualURIMapping
  fromURI	/categoryFeeds/([a-zA-Z0-9,-]*)/(.*)
  toURI	redirect:/rss/?generatorName=category&categories=$1&siteRoot=/$2

• Standard Templating Kit module includes the contentFeeds mapping in /modules/standard-templating-kit/virtualURIMapping/contentFeeds.

  Node name	Value
  modules
  standard-templating-kit
  virtualURIMapping
  contentFeeds
  class	info.magnolia.cms.beans.config.RegexpVirtualURIMapping
  fromURI	/contentFeeds/([a-zA-Z0-9]*)/([a-zA-Z0-9]*)/(.*)
  toURI	redirect:/rss/?generatorName=templateContent&category=$1&subCategory=$2&siteRoot=/$3

Syndication components

Three modules provide syndication components: Standand Templating Kit, Categorization and RSS Aggregator. All components rely of the functionality of the RSS Aggregator module.

Content Type RSS Feed component

The STK module includes the stkExtrasContentTypeRSSFeed component. The component renders an RSS subscription icon on the page. Editors can define feeds that aggregate pages based on the Article, News or Events templates, and select a parent page.

Here's the component definition is in STK > Template Definitions /components/extras/stkContentTypeRSSFeed:

Properties:

• modelClass:  ContentTypeSyndicateModel creates a STK renderable definition and returns the appropriate content.

• templateScript: syndicate.ftl (GIT) renders the RSS icon in the component and itemLinks provided by the model class.

The contentFeeds URI mapping calls the templateContent feed generator class ( PageSyndicator ) that generates the XML feed for content based on templates with a category property equal to content, and a subCategory property equal to the selection in the dialog.

Here's the generated feed for all article pages in the demo-project/about/subsection-articles section.

Category RSS Feed component

The Categorization module includes the categoryRSSFeed component. The component renders an RSS subscription icon on the page. The generated feed includes all pages tagged with specified categories. Editors can define the categories and root page for the feed.

The component definition is in /modules/categorization/templates/components/categoryRSSFeed.

Properties:

• modelClass: CategorySyndicateModel model class provides the business logic to select the relevant entries for the feed.

• templateScript: syndicate.ftl (Git) renders the RSS icon in the component and itemLinks provided by the model class.

The categoryFeeds URI mapping calls the category feed generator class ( CategorySyndicator ) that generates the XML feed for content based on the categories and root page selected in the dialog. The URL for the generated feed of content in the demo-project/about section, tagged with the family category, is similar to http://localhost:8080/magnoliaAuthor/rss/?generatorName=category&categories=ab9437db-ab2c-4df5-bb41-87e55409e8e1&siteRoot=/demo-project/about/subsection-articles . Compare the feed URL to the categoryFeeds Virtual URI mapping. The long number sequence is the UUID of the family category set up in Categories app.

Planet Syndication component

The RSS Aggregator module includes the planet feedSyndication component that renders subscription Atom and RSS icons and links. Editors can add a title and select a planet feed.

The component definition is in /modules/rssaggregator/templates/components/feedSyndication.

Properties:

• templateScript: The feedSyndication.ftl (Git) script renders the icons and links, and the  PlanetFeedGenerator  generator class renders the feed.

Security

Granting public user permissions

The anonymous role does not have permissions to the rss workspace on the author or public instance. Public users cannot see the feed component content by default. You can check this by logging out of the public instance.

The RSS Aggregator module installs the rss-aggregator-base role that provides read permissions to the rss workspace.

To provide public access to the feed content, assign the rss-aggregator-base role to the anonymous systems user on the public instance.

Now when you log out on the page, the feed content displays.

App access

By default, all users have app launcher access to the Feeds app, but only superuser can work with feeds. The superuser role includes the following relevant permissions.

These permissions allow superuser to:

• Create feeds in the Feeds app.
• Access the Configuration subapp and schedule imports.
• Select feeds in the component dialogs.

Granting editors read permissions

You can provide editors read only permissions to feed content by assigning the rss-aggregator-base role to the editor or group. This will allow them to:

• View feed content in the components.
• Select feeds in components.
• View content in the Feeds app, but not the Configuration subapp.

Granting editors write permissions

To provide editors write permissions to the Feeds app, create a new role with the following permissions and assign it to the editor or group.

This will allow them to:

• Create feeds 
• Configure single feed imports.

Granting permission to schedule global feed imports

Users with write permissions to the rss workspace cannot view content in the Configuration subapp . The  RSSAggregatorApp app class requires users to have write permissions to the rssaggregator module in the config workspace.

To provide access to the Configuration subapp, create a new role with the following permissions and assign it to the user or group.