Making requests to an HTML server

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Making requests to an HTML server

DUFOUR, Stephan (ext.)
Hi,
 
I have several Urls, each giving me a piece of information.
I want to write some code that gathers these pieces of information and make a display of the whole situation.
 
The principle is: I send a request to one URL then parse the result and make a new request to another URL using the parsed result.
 
I need some code that somehow mimic a human behaviour: it should
        1) go to a URL and get the content (easy part)
        2) go to a URL and retrieve some session ID then make a more elaborated request (POST, GET, or even trigger some Javascript)
 
Stephan
 
 
 

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: Making requests to an HTML server

David Mitchell-10
Start here:

Good intro to Squeak Networking is the Networking Squeak chapter from
the Squeak NuBlue
book:
http://stephane.ducasse.free.fr/FreeBooks/CollectiveNBlueBook/parsia%202.pdf

And then I would check out Albatross, which is a Squeak framework for
testing websites (including exercising Javascript). Here is a PDF
intro:
scg.unibe.ch/archive/projects/Brue06a.pdf

Google Squeak Albatross for more info.

--David Mitchell
blog: http://www.withaguide.com

On Tue, Dec 22, 2009 at 8:04 AM, DUFOUR, Stephan (ext.)
<[hidden email]> wrote:

> Hi,
>
> I have several Urls, each giving me a piece of information.
> I want to write some code that gathers these pieces of information and make
> a display of the whole situation.
>
> The principle is: I send a request to one URL then parse the result and make
> a new request to another URL using the parsed result.
>
> I need some code that somehow mimic a human behaviour: it should
>         1) go to a URL and get the content (easy part)
>         2) go to a URL and retrieve some session ID then make a more
> elaborated request (POST, GET, or even trigger some Javascript)
>
> Stephan
>
>
>
> _______________________________________________
> Beginners mailing list
> [hidden email]
> http://lists.squeakfoundation.org/mailman/listinfo/beginners
>
>
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: Making requests to an HTML server

cedreek
Hi,
you can also have a look at squeak soup [1] which is a tolerant HTML/XML parser (Squeak port of Beautiful Soup). The code repository is on squeaksource [2].
hth

[1] http://n4.nabble.com/squeak-dev-ANN-Soup-0-1-tt80191.html#a80191
[2] http://www.squeaksource.com/Soup.html  

2009/12/22 David Mitchell <[hidden email]>
>
> Start here:
>
> Good intro to Squeak Networking is the Networking Squeak chapter from
> the Squeak NuBlue
> book:
> http://stephane.ducasse.free.fr/FreeBooks/CollectiveNBlueBook/parsia%202.pdf
>
> And then I would check out Albatross, which is a Squeak framework for
> testing websites (including exercising Javascript). Here is a PDF
> intro:
> scg.unibe.ch/archive/projects/Brue06a.pdf
>
> Google Squeak Albatross for more info.
>
> --David Mitchell
> blog: http://www.withaguide.com
>
> On Tue, Dec 22, 2009 at 8:04 AM, DUFOUR, Stephan (ext.)
> <[hidden email]> wrote:
> > Hi,
> >
> > I have several Urls, each giving me a piece of information.
> > I want to write some code that gathers these pieces of information and make
> > a display of the whole situation.
> >
> > The principle is: I send a request to one URL then parse the result and make
> > a new request to another URL using the parsed result.
> >
> > I need some code that somehow mimic a human behaviour: it should
> >         1) go to a URL and get the content (easy part)
> >         2) go to a URL and retrieve some session ID then make a more
> > elaborated request (POST, GET, or even trigger some Javascript)
> >
> > Stephan
> >
> >
> >
> > _______________________________________________
> > Beginners mailing list
> > [hidden email]
> > http://lists.squeakfoundation.org/mailman/listinfo/beginners
> >
> >
> _______________________________________________
> Beginners mailing list
> [hidden email]
> http://lists.squeakfoundation.org/mailman/listinfo/beginners



--
Cédrick

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

RE: Making requests to an HTML server

DUFOUR, Stephan (ext.)

Hi,

 

I had little time to look at the stuff but I have a global idea of what does what.

 

Albatross is running via SUnit on localhost.

SUnit creates a component, this component is registered as a Seaside application which makes http calls to _localhost_.

 

What I want is simpler and different, I want to call external URLs.

I wonder if I can replace SUnit by a simple object and run ABrower with external URLs.

Some functions seems to require Seaside and may be not usable if I try them on external URLs.

 

The two other tools:

-          Selenium / Squelenium / WebTester: The firefox module allows you to easily write some tests but it cannot export code to Smalltalk. It also seems to be limited at finding predefined text, maybe the API allows you to search for variable content or you have to use something else. i.e. for example you have a customer account, you enter login/password, enter the customer account, it can check that you have NAME and SURNAME in the resultpage for exemple, but it won’t give you what the name and the surname are (maybe I simply didn’t dig enough).

-          Beautiful Soup can parse the result of a page and be a good complement for Selenium.

 

At the moment, I’d like to focus on Albatross because if I can use all the functions, it’ll be easy to code.

 

Stephan

 


De : Cédrick Béler [mailto:[hidden email]]
Envoyé : lundi 4 janvier 2010 14:52
À : DUFOUR, Stephan (ext.); [hidden email]
Objet : Re: [Newbies] Making requests to an HTML server

 

Hi,
you can also have a look at squeak soup [1] which is a tolerant HTML/XML parser (Squeak port of Beautiful Soup). The code repository is on squeaksource [2].
hth

[1] 
http://n4.nabble.com/squeak-dev-ANN-Soup-0-1-tt80191.html#a80191

[2]
http://www.squeaksource.com/Soup.html  

2009/12/22 David Mitchell <
[hidden email]>
>
> Start here:
>
> Good intro to Squeak Networking is the Networking Squeak chapter from
> the Squeak NuBlue
> book:
>
http://stephane.ducasse.free.fr/FreeBooks/CollectiveNBlueBook/parsia%202.pdf
>
> And then I would check out Albatross, which is a Squeak framework for
> testing websites (including exercising Javascript). Here is a PDF
> intro:
>
scg.unibe.ch/archive/projects/Brue06a.pdf
>
> Google Squeak Albatross for more info.
>
> --David Mitchell
> blog:
http://www.withaguide.com
>
> On Tue, Dec 22, 2009 at 8:04 AM, DUFOUR, Stephan (ext.)
> <
[hidden email]> wrote:
> > Hi,
> >
> > I have several Urls, each giving me a piece of information.
> > I want to write some code that gathers these pieces of information and make
> > a display of the whole situation.
> >
> > The principle is: I send a request to one URL then parse the result and make
> > a new request to another URL using the parsed result.
> >
> > I need some code that somehow mimic a human behaviour: it should
> >         1) go to a URL and get the content (easy part)
> >         2) go to a URL and retrieve some session ID then make a more
> > elaborated request (POST, GET, or even trigger some Javascript)
> >
> > Stephan
> >
> >
> >
> > _______________________________________________
> > Beginners mailing list
> >
[hidden email]
> >
http://lists.squeakfoundation.org/mailman/listinfo/beginners
> >
> >
> _______________________________________________
> Beginners mailing list
>
[hidden email]
>
http://lists.squeakfoundation.org/mailman/listinfo/beginners



--
Cédrick


_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: Making requests to an HTML server

cedreek
In reply to this post by DUFOUR, Stephan (ext.)
not sure I understand correctly but to me you don't need albatross.

With Soup:


        1) go to a URL and get the content (easy part)

soupTag := Soup fromUrl: 'yourUrl.html'.

otherTag := (soup findTagByClass: 'whateverClassNameWhereTheSessionIdBelong').

otherUrlString := (messageSoupTag childAt: 3) attributeAt: 'href'.  "if the url is in the 3rd child of otherTag"

secondSoupTag := Soup fromUrl: otherUrl.

"or convert as a true URL object that you can change"
 
url := otherUrlString asUrl.

etc...
hth,

Cédrick

 
        2) go to a URL and retrieve some session ID then make a more elaborated request (POST, GET, or even trigger some Javascript)

 

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: Making requests to an HTML server

Göran Krampe
In reply to this post by DUFOUR, Stephan (ext.)
Hi!

DUFOUR, Stephan (ext.) wrote:

> Hi,
>  
> I have several Urls, each giving me a piece of information.
> I want to write some code that gathers these pieces of information and
> make a display of the whole situation.
>  
> The principle is: I send a request to one URL then parse the result and
> make a new request to another URL using the parsed result.
>  
> I need some code that somehow mimic a human behaviour: it should
>         1) go to a URL and get the content (easy part)
>         2) go to a URL and retrieve some session ID then make a more
> elaborated request (POST, GET, or even trigger some Javascript)

I have written such a "robot" for a customer, and I need to update it in
the coming days, I will see if I can publish the reusable part of it.

regards, Göran

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: Making requests to an HTML server

Ken G. Brown
In reply to this post by DUFOUR, Stephan (ext.)
At 4:00 AM -0800 1/7/10, [hidden email] apparently wrote:

>Date: Thu, 07 Jan 2010 11:13:48 +0100
>From: G?ran Krampe <[hidden email]>
>Subject: Re: [Newbies] Making requests to an HTML server
>To: "A friendly place to get answers to even the most basic questions
> about Squeak." <[hidden email]>
>Message-ID: <[hidden email]>
>Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
>Hi!
>
>DUFOUR, Stephan (ext.) wrote:
>> Hi,
>>
>> I have several Urls, each giving me a piece of information.
>> I want to write some code that gathers these pieces of information and
>> make a display of the whole situation.
>>
>> The principle is: I send a request to one URL then parse the result and
>> make a new request to another URL using the parsed result.
>>
>> I need some code that somehow mimic a human behaviour: it should
>>         1) go to a URL and get the content (easy part)
>>         2) go to a URL and retrieve some session ID then make a more
>> elaborated request (POST, GET, or even trigger some Javascript)
>
>I have written such a "robot" for a customer, and I need to update it in
>the coming days, I will see if I can publish the reusable part of it.
>
>regards, G–ran

I am very interested in this.
Go G–ran!

In my case I wish to start with some numbers parsed from the first url, use each of them to create a search at a second pre-known url, then operate on the results of the search to perhaps figure out if there is a pdf download link available.

Thx,
Ken G. Brown


_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners