If you have to do web data scraping, what tool would you use?

Hello

I have used Soup, but I now prefer XMLHTMLParser, because I find it easier to relate the structure of the XML output to that of the original web page. It is also possible to use XPath to locate the target area more quickly. I particularly like Monty’s ‘Smalltalkish’ adaptation of XPath (see the last para of the class comment to XPath).

Hope this helps

Peter Kenny

I scrapped all the magic cards (bad bad practice) using Soup.

It could be easier but it worked for me. This is why I maintain it and added tests.

Stef

Hi,

Imagine that you have to do some data scraping work, what tool would you use?

I know about ZnClient, Soup, NeoCSV, NeoJSON, is there something else that i'm not aware of it?

 

thanks.

 

 

--

Bernardo E.C.

 

Sent from a cheap desktop computer in South America.

Bernardo

Being now retired, I do programing just for intellectual stimulation. Your problem looked as though it would provide more interest than cryptic crosswords or Sudoku, and it touches on areas of Pharo use that I have some experience with. So…

The attached file, BernardoDemo.st, shows how to use XMLHTMLParser with xPath and NeoJSON to tackle your problem – or at least a large subset of it. I cobbled it together in a Playground, and the easiest way to use it is to copy it into  a Playground and ‘do it and go’ for each block of code. There are liberal comments, but if anything is not clear come back to me.

A few caveats:

1.      XPath is a whole other programming language, embedded in Pharo, which takes some learning. I am by no means expert in it, and it may be that I have used it clumsily. One advantage of embedding it in Pharo is that you can intersperse Pharo and XPath, which I do whenever I can’t solve something entirely with XPath. Probably most of the places where I use #collect: followed by more XPath could be done entirely in XPath if I knew how.

2.      This is the first time I have tried to use NeoJSON, so do not take my code as an example of how to use it. It all works, as far as I can see. I cannot claim more than that.

3.      The easiest way to generate an object (or map) in NeoJSON is to start with a Pharo dictionary, which I have done everywhere. However, this means you have no control over the order in which the attributes appear in the JSON file. This is of no importance to a computer, since by definition the attributes are unordered, but it makes it a little odd to a human reader of the JSON.

4.      In your spec, the desired output has a lot of unquoted strings for attribute names, for example nbd_no. The code produces these strings with double quotes, which as far as I can see is necessary for legal JSON.

5.      Note that all numerical values appear in the output as strings. No doubt they could be converted to numbers, but I was too lazy to find out how.

6.      I have done this using Moose 5.1 (Pharo 4.0, build #40613), with versions of XMLHTMLParser and XPath which I downloaded quite a while ago. There are no particularly abstruse uses, so I hope you will be OK if you use more recent versions.

Hope this is helpful.

Best wishes

Peter Kenny

From: Pharo-users [mailto:[hidden email]] On Behalf Of Bernardo Ezequiel Contreras
Sent: 27 June 2016 15:17
To: Any question about pharo is welcome <[hidden email]>
Subject: Re: [Pharo-users] If you have to do web data scraping, what tool would you use?