Posted by
Offray on
Apr 08, 2015; 3:20pm
URL: https://forum.world.st/ZnClient-getting-more-that-19-tweet-for-data-scrapping-tp4818162p4818390.html
Hi Paul and Sven,
I will try Twitter API if is necessary, but I'm not trying to get
support here on how to violate Twitter ToS. I'm pretty aware of them,
but surely there are exceptions. The mail I shared with you (the one
Paul point to) about why I would like to not use API but instead use
scrapping doesn't go in details, it just said that it was because people
who have not a twitter account or signed the ToS should be able to get
some twitter info. That special kind of info is the one regarding public
political/politicians discourse and my idea to scrap *public* and
*specific* data from twitter happens in the context of a project for
citizen oversight of political issues empowered by ICT. The project is
discussed on detail here:
https://www.newschallenge.org/challenge/elections/entries/datapolis-data-narratives-visualizations-for-citizen-oversight-of-politicians-discourses-and-public-contracts-in-social-media-and-the-webSo, for the moment, I will get properly account permissions for getting
twitter data, but my conviction for the long term is that public
political discourse (among others), even the one that circulates on
private networks like Twitter or Facebook, should be under
Constitutional Terms (like free speech and wide political participation)
and not under the restricted ones of Twitter or Facebook.
This is a sensible issue and surely needs more talk, may be on another
time, but I will follow your advice and extract data from Twitter API
and come here with questions about it.
Thanks for all your help and support,
Offray
El 08/04/15 a las 08:29, Paul DeBruicker escribió:
> Offray - What Sven said is correct. You're not getting an answer about how
> to violate their Terms of Service because this isn't that kind of place.
> You've asked 3 times. Once is usually enough. Use the API. For the Soup
> questions get an inspector on an instance of a SoupTag and start sending it
> messages it understands and see what you get. Trial and error. Or read the
> python Soup docs as the commands probably have an equivalent in the
> Smalltalk library. Most of this programming stuff is reading, doing a
> little experiment, thinking, then trying again.
>
> Sven - I only showed him that SoupTag has a #text message. I'm sure you're
> busy and had forgotten that the first time he/she asked they stated that
> they don't want to use the api:
>
http://forum.world.st/Data-scrapping-in-pharo-Extracting-tweets-contents-td4817746.html> and provided the download code in an ws.stfx.eu snippet.
>
> Hope this helps
>
> Paul
>
>
> Sven Van Caekenberghe-2 wrote
>> What Paul showed is basically just a hack.
>>
>> What you probably what is full API access to Twitter, that gives you the
>> real thing, but it is more work and you have to understand all the
>> technical details (unless somebody already did it for you, I don't know -
>> I know that Zinc-SSO can connect to Twitter).
>>
>>
https://dev.twitter.com/overview/api>>
>>> On 07 Apr 2015, at 20:23, Offray Vladimir Luna Cárdenas <
>
>> offray@
>
>> > wrote:
>>>
>>> Hi,
>>>
>>> Recently Paul DeBruicker taught me how to refine my code for getting
>>> tweets properly. Consider this:
>>>
>>> =[1]====================================
>>> | source anUrl tweet |
>>>
>>> anUrl := '
https://twitter.com/offrayLC'.
>>> source := Soup fromString: (ZnEasy get: anUrl ) contents asString.
>>> tweets := (source findAllTagsByClass: 'ProfileTweet-text') collect:[:ea |
>>> ea text].
>>> ========================================
>>>
>>> Is working fine, but I would like to get more that 19 tweets, that is
>>> what you get by default. There is any way to tell ZnEasy and friends to
>>> get more tweets, something similar to what you do when you scroll down
>>> into a twitter page?
>>>
>>> And by the way, I would like to make more sense of the Soup I got in the
>>> last line. ea text gives me the tweet contents, but how can I interpret
>>> the metadata in the soup? (is a retweet, date of publishing and so on). I
>>> could make this for most part of the twitter profile page, but the tweet
>>> is kind of elusive, for example how to know that "text" is the proper
>>> message for getting the tweet content? Any pointer to how to make sense
>>> of it by myself is greatly appreciated.
>>>
>>> Cheers,
>>>
>>> Offray
>>>
>
>
>
>
>
> --
> View this message in context:
http://forum.world.st/ZnClient-getting-more-that-19-tweet-for-data-scrapping-tp4818162p4818361.html> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
>
>