Mass Random Content Population from Wikipedia

Sometimes during Liferay web content development and testing, especially when working on search functionality, it’s desirable to have a large amount of content to work with. Obviously, creating hundreds or thousands of pieces of content manually would be a long and tedious process. Fortunately, we can use Wikipedia’s random page URL and a bit of clever code to do the work for us.

The idea is send an HTTP request to Wikipedia, then parse the HTML response to find the main content div. JSoup is a very useful API that allows you to do both of these tasks very easily. After you get the article content from the response, it’s just a matter of using the Liferay API to create a new article. This approach allows you to create several hundred articles in just a few minutes.

Here is some simple, straight-line code for you to look at as a reference for populating Liferay WCM with as many articles as you desire from Wikipedia – wrap this code in a portlet and call the doPopulate() method possibly when the user clicks a “Create Articles” button.

Code:

public void doPopulate(ActionRequest request) {   
        
        int count = 0;

       int NUMBER_OF_ARTICLES_TO_CREATE = 1000;

       
        for (int i=0; i < NUMBER_OF_ARTICLES_TO_CREATE; i++) {        
         
         try {
           Document doc = Jsoup.connect("http://en.wikipedia.org/wiki/Special:Random").get();
           String bodyContentHtml = doc.select("#bodyContent").get(0).html();
           String title = doc.select("#firstHeading").get(0).text();
           
           ThemeDisplay themeDisplay = (ThemeDisplay)request.getAttribute(WebKeys.THEME_DISPLAY);
               
           String content = "<?xml version="1.0"?>n"+
                   "n"+
                   "<root available-locales="en_US" default-locale="en_US">n"+
                   "  <static-content language-id="en_US"><![CDATA["+
                   bodyContentHtml+
                   "  ]]></static-content>"+
                   "</root>";
                 
           ServiceContext serviceContext = ServiceContextFactory.getInstance(request);
           
           Map<Locale, String> titleMap = new HashMap<Locale, String>();
           titleMap.put(Locale.US, title);
               
           JournalArticleLocalServiceUtil.addArticle(  themeDisplay.getUserId(), //userId
                                 themeDisplay.getScopeGroupId(), //groupId
                                 0, //classNameId
                                 0, //classPK
                                 StringPool.BLANK, //articleId
                                 true, //autoArticleId
                                 JournalArticleConstants.VERSION_DEFAULT, //version
                                 titleMap,
                                 null, //descriptionMap
                                 content,//content,
                                 "general", //type
                                 StringPool.BLANK, //structureId
                                 StringPool.BLANK, //templateId
                                 StringPool.BLANK, // layoutUuid
                                 
                                 1, //displayDateMonth
                                 1, //displayDateDay,
                                 1970, //displayDateYear,
                                 0, //displayDateHour,
                                 0, //displayDateMinute,
                                 
                                 0, //expirationDateMonth,
                                 0, //expirationDateDay,
                                 0, //expirationDateYear,
                                 0, //expirationDateHour,
                                 0, //expirationDateMinute,
                                 true, //neverExpire,
                                 0, //reviewDateMonth,
                                 0, //reviewDateDay,
                                 0, //reviewDateYear,
                                 0, //reviewDateHour,
                                 0, //reviewDateMinute,
                                 true, //neverReview,
                                 true, //indexable,
                                 false, //smallImage, 
                                 StringPool.BLANK, //smallImageURL,
                                 null, //smallImageFile, 
                                 null, //images,
                                 StringPool.BLANK, //articleURL,
                                 serviceContext);
           
           count++;
           System.out.println(count + " articles created");
         }
         catch (Exception e) {
           System.out.println("Error occurred in current iteration, continuing");
           System.out.println(e.getMessage());
         }
       
       }    
     
   }

Maven Dependency:

<dependency>

    <groupId>org.jsoup</groupId>

    <artifactId>jsoup</artifactId>

    <version>1.7.2</version>

</dependency>

Note: obviously you can make this portlet as full-featured as you need it to be by taking in the number of articles to populate from Wikipedia and more.