[squeak-dev] Extracting data from web pages using Squeak

> Hi All,
>
> I'm interested in extracting data from web pages using Squeak.  Can I use
> Scamper or parts of it?  Any ideas or hints on where to start will be greatly
> appreciated.  Thanks.
>
> Lou
> -----------------------------------------------------------
> Louis LaBrunda
> Keystone Software Corp.
> SkypeMe callto://PhotonDemon
> mailto:[hidden email] http://www.Keystone-Software.com
>
>
>

>

>
> HtmlTokenizer helps here.  Here's a bit of code I added to String class to
> give you an idea of how to use it.
>
> tagsOfType: aString
>         "return all tags found in self of type aString"
>
>         | endTag |
>         endTag := '</' , aString , '>'.
>         ^ ((HtmlTokenizer  on: self) upToEnd
>                 select: [ :ea | ea name = aString])
>                 reject: [ :ea | ea source = endTag]
>
>
>
> Here's another example that is slightly richer (and probably could be
> improved but what the heck).
>
> textOfType: aString
>         "return a collection of triples of all tags found in self of type
> aString with start tag, intermediate text if any, and end tag if any"
>
>         | stream element endTag triple answer |
>         endTag := '</' , aString , '>'.
>         answer := OrderedCollection new.
>         stream := ReadStream on: ((HtmlTokenizer  on: self) upToEnd).
>         [stream atEnd] whileFalse: [
>                 (element := stream next) name = aString ifTrue: [  "start
> tag found"
>                         triple := Array new: 3.
>                         triple at: 1 put: element.
>                         stream peek class = HtmlText ifTrue: [
>                                 triple at: 2 put: stream next.
>                                 stream peek source = endTag ifTrue: [
>                                         triple at: 3 put: stream next
>                                         ]
>                                 ].
>                         answer add: triple
>                         ]
>                 ].
>         ^ answer
>
>
>
> Louis LaBrunda <[hidden email]>
> Sent by: [hidden email]
>
> 06/16/08 11:57 AM
>
> Please respond to
> [hidden email]; Please respond to
> The general-purpose Squeak developers list
>  <[hidden email]>
> To
> [hidden email]
> cc
> Subject
> [squeak-dev] Re: Extracting data from web pages using Squeak
>
>
>
>
> Hi Cédrick,
>
> Thanks for the hint.
>
>>I would use:
>>HTTPClient httpGet: 'http://url.com' to get the html stream.
>>Then you can parse it...
>
> Are there parsers available to get say table data into some kind of
> collection?
>
> Lou
> -----------------------------------------------------------
> Louis LaBrunda
> Keystone Software Corp.
> SkypeMe callto://PhotonDemon
> mailto:[hidden email] http://www.Keystone-Software.com
>
>
>
>
>
>
>

> Hi!
>
> f_'http://www.google.com' asUrl retrieveContents.
> cont_f content.
> html_HtmlParser parse: cont.
> html explore
>
> gives you the dom-tree as squeak-objects you can then work on
>
> greetings, Bernd
>
>

> Giuseppe Luigi Punzi schrieb:
>> I tried with lordzealon.com and was a bit slowly.
> wow that is an understatement, it actually takes forever on that
> particular website because it transmits a lot from unrelated sites
> like googlesyndication.com. If it is really that website you want to
> parse: use a normal browser to save the .html, then use squeak to
> parse it (you can use Squeak to only grab the html from the site but
> its complicated, see the posts of the other newsgroup members)
> A progress bar is probably not what you want because what takes long
> here is the transmission of the website, not the parsing. If it were
> the parsing you could tell at any Moment what percentage is completed,
> in the case of the transmission thats difficult to tell.
> But if you need ProgressBars for something else: there is class
> ProgressMorph. You will have to tell a ProgressMorph which value
> (percentage %) it is supposed to show.
> Greetings, Bernd
>
>

> Hi Cédrick,
>
> Thanks for the hint.
>
>>I would use:
>>HTTPClient httpGet: 'http://url.com' to get the html stream.
>>Then you can parse it...
>
> Are there parsers available to get say table data into some kind of collection?
>
> Lou
> -----------------------------------------------------------
> Louis LaBrunda
> Keystone Software Corp.
> SkypeMe callto://PhotonDemon
> mailto:[hidden email] http://www.Keystone-Software.com
>
>
>