Status of Twitter (api/analysis/visualization) in Pharo?

Hi,

I want to do some interaction with the Twitter API (retrieve tweets based on hashtags mostly) and visualize results of some analyses using Roassal.

I saw there are some projects on Smalltalkhub, but none has much/any/up-to-date documentation. What is the current status of these projects? Does anyone know about any up-to-date documentation or examples?

I am thankful for all hints.

Best regards, Martin.

Hi,

We're working on a project that involves retrieve tweets and visualize
results using Roassal also. Our approach includes data scraping and is
based on the idea that public political discourse in covered under
constitutional law which is above Twitters API Terms and Conditions. Is
related with data activism and slow data and doesn't use Twitter's API.
Scraping is a sensible issue but I think that we're covered by fair use
and a lawyer working on digital rights thinks the same. Scraping happens
at client level using chrome/chromium, without any use of Pharo for
that. Pharo is used for extracting structured information of the tweets
and visualizing it. Most of the detailed raw documentation is in Spanish
as a product of our local workshops, but you can get and idea of what
the project is doing at [1] to see is this approach works for you.

[1]
http://mutabit.com/offray/static/blog/output/posts/visualizing-politicianspolitical-discourses-on-twitter.html

Next week we well be doing a week long data visualization workshop [2],
so I think that we will have better visualization, scraping algorithms
and documentation (in Spanish). I will keep you posted on the general of
how our workshop advances and sharing preliminary results.

[2] http://mutabit.com/dataweek/

Cheers,

Offray

On 15/06/15 09:12, MartinW wrote:

Thank you very much, Offray, that looks very interesting.

When was the last time, you used the scraping plugin? It does run and scrolls down the twitter pages, but there is no result at all to be found in the console. Can it be that Twitter changed the format lately?

I will try to find a way to harvest tweets first, and then I will try your visualization engine.

Martin.

I succeeded in installing parts of TwitterRoassalPlayGround in a Pharo 3 image.
http://smalltalkhub.com/#!/~arturozambrano/TwitterRoassalPlayGround

- The RESTRetriever seems to work fine, but the Rest results are very limited due to Twitter's own limitations.
- The StreamingRetriever does simply not produce any results at all, but also no errors.

MartinW wrote

I succeeded in installing parts of TwitterRoassalPlayGround in a Pharo 3 image.
http://smalltalkhub.com/#!/~arturozambrano/TwitterRoassalPlayGround

- The RESTRetriever seems to work fine, but the Rest results are very limited due to Twitter's own limitations.
- The StreamingRetriever does simply not produce any results at all, but also no errors.

I got the StreamingRetriever to work..

Hi Martin,


On 16/06/15 10:14, MartinW wrote:
> Thank you very much, Offray, that looks very interesting.

Thanks. I think that is important to keep politicians and public
accountable for what they say and do and social networks could help in
doing that in a more dynamic way (of course is not enough).

> When was the last time, you used the scraping plugin? It does run and
> scrolls down the twitter pages, but there is no result at all to be found in
> the console. Can it be that Twitter changed the format lately?

The last time was April, but if the scroll works, you should be able to
inspect the DOM from the browser and export it following the
instructions at the blog post. The browser knows about the DOM of the
page that is being showed currently and if you are capable of seeing it
in the browser, you should be able to inspect and save. We will test the
script again and improve the code next week in a week long workshop on
data visualization using Pharo/Roassal, so I will keep you posted about
what's happening with this approach.

> I will try to find a way to harvest tweets first, and then I will try your
> visualization engine.

Ok. Let me know what happens.

Cheers,

Offray

Thanks Arturo. Seems a really interesting advance. The seaside UI uses the Twitter API? Let us know when the project is uploaded.

Cheers,

Offray

Hi Martin,

 this are good news. There is a seaside base UI for controlling tweets download. I will let you know as soon as it is uploaded. 

Visualizations are working, the code is crappy but works... I have some documentation I will send it to you (all of them are very basic examples we built for the last world cup), unfortunately it is Spanish only, but you will get the idea from the code snippets. 

regards

art

On Tue, Jun 16, 2015 at 12:44 PM, MartinW <[hidden email]> wrote:

MartinW wrote
> I succeeded in installing parts of TwitterRoassalPlayGround in a Pharo 3
> image.
> http://smalltalkhub.com/#!/~arturozambrano/TwitterRoassalPlayGround
>
> - The RESTRetriever seems to work fine, but the Rest results are very
> limited due to Twitter's own limitations.
> - The StreamingRetriever does simply not produce any results at all, but
> also no errors.

I got the StreamingRetriever to work..




--
View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-tp4832460p4832684.html

Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

Offray wrote

Ok. Let me know what happens.

Hi Offray,
I got the scraping, importing and visualization working. Not sure, what went wrong the first time.

I found one little bug, when the avatar was a jpeg. It can be solved by changing the following lines in TwitterProfile>>scrapAvatarFrom:

from:

        ((avatarUrl asLowercase endsWith: '.jpeg') or: (avatarUrl asLowercase endsWith: '.jpg'))
                ifTrue: [
                        avatarImage := ZnEasy get: avatarUrl.
                        "(PNGReadWriter on: avatarImage ) nextPutImage: (JPEGReadWriter on: avatarImage )" ].

to:

        ((avatarUrl asLowercase endsWith: '.jpeg') or: (avatarUrl asLowercase endsWith: '.jpg'))
                ifTrue: [avatarImage := ZnEasy getJpeg: avatarUrl.].


Very nice result. Thank you a lot for sharing it!
Best regards,
Martin.

Hi Martin,

Thanks, we had already detected the problem with Jpeg images and even
reported here to the list, but until now the solution eluded us. Your
solution is already on the repo.

Cheers,

Offray

On 21/06/15 10:53, MartinW wrote:

Offray wrote

Thanks, we had already detected the problem with Jpeg images and even
reported here to the list, but until now the solution eluded us. Your
solution is already on the repo.

Cool :)