Home > javascript > Scraping reviews from Google Play using JavaScript

Scraping reviews from Google Play using JavaScript

April 10Hits:1
Advertisement

What I'm trying to do:

  1. Get user input for Google Play App page.

    e.g. https://play.google.com/store/apps/details?id=jp.scn.android

  2. Scrape 100 reviews from Google Play App and organize them into an array.

My JavaScript function:

function test() {     var urlAdd = document.getElementById('input').value;     var urlEnglish = urlAdd + '&hl=en';      var query = {     url: urlEnglish,     type: 'html',     selector: '[class=review-body]',     extract: 'text'     },     request;      request = 'http://example.noodlejs.com/?q=' +     encodeURIComponent(JSON.stringify(query)) +     '&callback=?';      jQuery.getJSON(request, function (data) {     document.getElementById('output').innerHTML = '<pre>' +     JSON.stringify(data, null, 4) + '</pre>';     }) }; 

Problems:

  1. This function feels bulky.
  2. It only returns 20 results. My guess is that this has something to do with how the Google Play DOM retrieves reviews.

Questions:

  1. How do I streamline/improve my scraper function?
  2. How do I get 100 results or more?

Comments:

My apologies if this is a simple solution. I just started learning JavaScript piecemeal in February.

Answers

Use scraping as last resort. I suggest you find available APIs for that. It's more robust, and easy to work with. One issue with scraping is that it fetches the static HTML generated by the url. You can't fetch what is loaded via JS. Although there are ways to do it, it's just adds too much complexity.

In terms of hacking the code, you could check out how Google Play loads the rest of the comments. It should be an AJAX call, check the network tab of your browser. You checkout that url, the response and modify it to your needs.

As for your script: You're using jQuery, use it all the way!

// Cache as well as put configurables here
var input = $('#input');
var output = $('#output');
var noodleUrl = 'http://example.noodlejs.com/?';

var query = {
    // You should check if the url ends with parameters. Otherwise, this fails.
    url: input.val() + '&hl=en',
    type: 'html',
    selector: '[class=review-body]',
    extract: 'text'
};

// $.param
var request = noodleUrl + $.param({
  q : JSON.stringify(query),
  callback : '?'
});

$.getJSON(request, function (data) {
    output.html('<pre>' + JSON.stringify(data, null, 4) + '</pre>');
}

Related Articles

  • Scraping reviews from Google Play using JavaScriptApril 10

    What I'm trying to do: Get user input for Google Play App page. e.g. https://play.google.com/store/apps/details?id=jp.scn.android Scrape 100 reviews from Google Play App and organize them into an array. My JavaScript function: function test() { var u

  • How do you feed reviews to Google places page for businesses? May 14

    Possible Duplicate: How to provide Google reviews information? Check out this google places page as an example: http://maps.google.com/maps/place?hl=en&sugexp=erf1&pq=virginia+honda+&cp=17&gs_id=50&xhr=t&bav=on.2,or.r_gc.r_pw.r_qf.

  • Fall Back Google CDN in JavaScriptMay 23

    In HTML5 Boilerplate it Grab Google CDN's jQuery with this JS code <!-- Grab Google CDN's jQuery, with a protocol relative URL; fall back to local if offline --> <script src="//ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"&g

  • How to achieve an "inner glow" effect on polygons drawn using the Google Maps v3 javascript API?

    How to achieve an "inner glow" effect on polygons drawn using the Google Maps v3 javascript API?January 10

    Is it possible to produce an "inner glow" effect on polygons drawn using the Google Map v3 javascript API? If not possible using the vanilla Google Maps API, is there a 3rd party tool that can do it? Or perhaps some other technique I could use t

  • Is there a way to leave an anonymous review on Google+?March 9

    You used to be able to leave anonymous reviews using Google+. Is there any way to do that anymore? Not sure if giving your full name and photo with your review is wise. --------------Solutions------------- There is no longer a way to leave anonymous

  • How to display drawings from ArcGIS on Google Maps using JavaScript or PHP?March 15

    my client has ArcGIS installed on his PC and he has designed roads, hospitals, houses, and parks of a specific area in ArcGIS he needs his system online. challenging task for me is how can I get all drawing information from his ArcGIS that I can show

  • Google Tag Manager Javascript Function to Return IP Address of Client December 24

    I need to somehow pull the client's IP address using pure JavaScript; no server side code, not even SSI. Any ideas? I'm not against using a free 3rd party script, if someone can suggest one. This is an emergency stop gap until we can deploy new code.

  • Google Releases its JavaScript Closure Tools

    Google Releases its JavaScript Closure ToolsNovember 12

    New JavaScript libraries and frameworks appear all the time, but it's not every day that Google release the client-side code that powers Search, GMail, Google Maps, Google Docs, and more. The company has open-sourced their Closure Tools and they are

  • Does the Google spider render JavaScript?November 19

    Does the Google spider knows how to render JavaScript, or only HTML? --------------Solutions------------- Apparently it does, some, according to Matt Cutts: "For a while, we were scanning within JavaScript, and we were looking for links. Google has g

  • The simplest way to track Google Analytics without JavaScript?June 13

    Is it possible to track Google Analytics on my site, when JavaScript is not available? --------------Solutions------------- It can be done. All Google Analytics seems to do is ultimately request an image file with analytic data appended to the URL. Y

  • Ideas to tackle unwanted bad press/review on Google's SERP?June 28

    After Googling our company name to our horror we've found someone on Yelp.co.uk has reviewed our company. On the SERP your eye is immediately drawn to the 2 star review some complete stranger has written, which to be honest is pure slander! The most

  • Will a reviewer be notified if a developer responds to their review on Google Play?

    Will a reviewer be notified if a developer responds to their review on Google Play? November 7

    As far as I have known, and also read in the description of many apps on Google Play, it is not possible for publishers to respond to user's comments, but recently I have come across this: Can someone give me more info on this? Also, if a publisher r

  • How to access Google Trends without javascript?November 20

    I noticed that Google Trends drastically changed it's UI and now fails to work without JavaScript enabled. Where to access the plain old good Web 1.0-ish Google Trends? Alternatively, how to configure NoScript to allow the necessary set of additional

  • Get google to index JavaScript pages as separate pagesFebruary 1

    I have a product page with say 10 products, that loads up all the content when the page loads. Then JavaScript hides content till a product is selected. Then JavaScript hides and unhides information. It is this way rather than separate html pages for

  • Will Google treat this JavaScript code as a bad practice?November 2

    I have a website that provides a custom UX experience implemented via JavaScript. When JavaScript is disabled in the browser the website falls back to CSS for the layout. To make this possible I've added a noJS class to the <body> and quickly remove

  • How to show Google Review in Google Search Result?

    How to show Google Review in Google Search Result?April 2

    Any one know how to do this? What I mean is Review box at the right hand side: --------------Solutions------------- This is known as the Google Knowledge Graph and Google will decide what information to show here from businesses listed in Google Loca

  • Changing Polygon shape on Zoom in and Zoom out in google maps using javascriptAugust 13

    In my application iam using google maps v3, on page load with default zoom level of 15 i load few polygons by passing points(latlng coordinates) from database on the google map now my question is when i zoom out the map the polygons shape has to be c

  • Web scraping from the Google Play storeSeptember 3

    I am using this R function to web scrape data from the google play store. Is there a way to increase its efficiency using R? This code takes about 4 seconds for 14 urls with my machine/internet connection. library(rvest) library(RCurl) library(foreac

  • how to add data in google charts using javascript ?January 20

    I am using Google Charts for my website and parse.com for server-side. I have one tag <div id="chart_div"> </div> which draws the chart. For Chart this is Javascript part : $( document ).ready(function() { var currentUser = Parse.Use

  • Open Google page in JavascriptFebruary 2

    I have the following code to open a google page and type "Hello" in the textbox. The code opens the page but the textbox is empty. Does anyone have an idea please ? Thanks. <html> <head> <script type="text/javascript">

Copyright (C) 2017 ceus-now.com, All Rights Reserved. webmaster#ceus-now.com 14 q. 0.510 s.