Login  Register

STON distributed control version and long lines (it was Re: Citizen example for manipulating a bibtex file)

Posted by Offray on Apr 24, 2015; 9:51pm
URL: https://forum.world.st/Citizen-example-for-manipulating-a-bibtex-file-tp4784240p4821729.html

Hi Sven,

I would like to retake this conversation. I wasn't able to answer before
because I was trying to get minimal functionality for grafoscopio and
now that is there, I would like to tackle the issue of long lines in
STON, so it can become Distributed Version Control Systems friendly.

To understand better your proposal, lets try it with a real "document"
stored in STON, the one at [1]. This is the grafoscopio alpha manual in
Spanish, with long lines. I would like to test it with your second
propossal, and change the newline inside strings as CRLF. A successful
test case after applying the change would be that I can diff two commits
of the manual and compare them inside our DVCS (fossil-scm), without it
getting the file confused with a binary file.

[1]
mutabit.com/deltas/repos.fossil/grafoscopio/doc/tip/Docs/Es/Manual/manual-grafoscopio.ston

I imagine you need to do some change to STON to implement this, but I'm
willing to make any test from this side and report them back to you.

Making STON DVCS friendly would open a lot of collaboration on
interactive documentation and other places where STON can be used as an
storage format (like we are doing).

So, how can we proceed?

Thanks,

Offray

El 26/11/14 a las 10:40, Sven Van Caekenberghe escribió:

> Hi Offray,
>
> I think I better understand what you are doing now, and how you are using STON.
>
> What I think might help you (in your use case), is to add two options:
>
> 1 - to STONWriter #keepNewlines that converts any newline inside Strings to a newline as specified by #newline: instead of encoding it as unprintable
> 2 - to STONReader #convertNewlines that converts any newline inside Strings to a newline as specified by #newline: instead of keeping it as is
>
> Now, 2 is already there, except that any newline read is kept as it is, while I think it would be better to convert CR, LF and CRLF to just one (to be specified).
>
> Similarly, 1 would convert CR, LF and CRLF to a single one (to be specified).
>
> Without these conversions, mixups between different line end conventions could easily happen. I think it is too much work to do this in your own models.
>
> If you think this would help, I'll put this on my todo.
>
> Sven
>
>> On 18 Nov 2014, at 02:50, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>>
>> Hi Sven,
>>
>> Sorry for my late response. The constructive comments on the list and yours in particular are very valuable to my. I was finishing some details, so only until now I have the time to implement your suggestions. The new code for custom keys on bibtex files from pharo is published at [1] (by the grace of Doru's easy publishing of playgrounds on your stfx server)
>>
>> [1] http://ws.stfx.eu/3CEKQQQ3NL2E
>>
>> By the way the article I'm writing is about a tool for open, citizen, garage research and science developed in Pharo. It is published at [2] and was stored nicely using STON[3] and is superb. To test the tool, the article was wrote on it.
>>
>> [2] http://mutabit.com/deltas/repos.fossil/grafoscopio/doc/tip/Docs/Es/Articulos/Libertadores/bootstrapping-objeto-investigacion.pdf
>>
>> [3] http://mutabit.com/deltas/repos.fossil/grafoscopio/doc/tip/Docs/Es/Articulos/Libertadores/bootstrapping-objeto-investigacion.ston
>>
>> [4] http://mutabit.com/deltas/repos.fossil/grafoscopio/doc/tip/Docs/Es/Articulos/Libertadores/bootstrapping-objeto-investigacion.markdown
>>
>> The only thing I would add to STON would be an option to support line breaks so long character sequences can be broken to make the format DVCS friendly (git, fossil, etc) and support collaboration and changes tracking. At this moment, because of the long lines in STON, the files are treated as binaries by fossil :-/.
>>
>> Thanks for STON and your lessons,
>>
>> Offray
>>
>> El 22/10/14 a las #4, Sven Van Caekenberghe escribió:
>>>
>>>> On 22 Oct 2014, at 18:42, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Thanks again. I have a small script, using Citezen which does the trick. I can explore and modify the BibTeX File from the playground with this:
>>>>
>>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>> | bibFile bibliography bibStream bibOutputer |
>>>> bibFile := ((FileLocator documents / 'U/Libertadores/Grafoscopio') children
>>>> detect: [:each | each basename endsWith: 'bib' ]).
>>>> bibliography := CZBibParser parse: bibFile contents.
>>>> bibStream := '' writeStream.
>>>> 1 to: (bibliography size) do: [:index |
>>>>    (((bibliography entries at: index) fields at: 2) key = 'shorttitle')
>>>>        ifTrue: [
>>>> (bibliography entries at: index)
>>>>    key: ((bibliography entries at: index) fields at: 2) value].
>>>>        bibOutputer := CZBibtexOutputer new.
>>>>        bibStream nextPutAll:
>>>>  (bibOutputer entryToBibtexString:
>>>> (bibliography entries at: index)); cr.].
>>>> bibliography.
>>>> bibFile writeStreamDo: [:stream |
>>>>   stream nextPutAll: bibStream contents withUnixLineEndings ].
>>>> bibStream contents.
>>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>
>>> Some constructive comments about your code (smaller is always better, especially for interactive snippets):
>>>
>>> - you can change the #detect: to [ :each | each extension = #bib ]
>>> - you can iterate directly over the entries with #do: as in bibliography entries do: [ :each | .. ] which saves you the #at: index
>>> - there are handy unary shortcuts for accessing elements, like #first, #second and so on (up to #ninth) which also save you parenthesis
>>> - you can also construct strings using the idiom String streamContents: [ :bibStream | .. ]
>>>
>>> Sorry, these jumped to me when I saw your code, I hope you don't mind ;-)
>>>
>>>> I will put some functionality inspired by this on my prototype this weekend.
>>>>
>>>> Cheers,
>>>>
>>>> Offray
>>>>
>>>> On 10/21/2014 01:20 AM, stepharo wrote:
>>>>> Check in the tools there is a bib writer.
>>>>>
>>>>> Stef
>>>>>
>>>>> On 21/10/14 03:33, Offray Vladimir Luna Cárdenas wrote:
>>>>>> Thanks Stef and Damien,
>>>>>>
>>>>>> I have this small script as a proof of concept:
>>>>>>
>>>>>> ===================================
>>>>>> | bibFile bibliography |
>>>>>> bibFile := ((FileLocator documents / 'U/Libertadores/Grafoscopio')
>>>>>> children
>>>>>>     detect: [:each | each basename endsWith: 'bib' ]) contents.
>>>>>> bibliography := CZBibParser parse: bibFile.
>>>>>> 1 to: (bibliography size) do: [:index |
>>>>>> (((bibliography entries at: index) fields at: 2) key = 'shorttitle')
>>>>>>     ifTrue: [
>>>>>>         (bibliography entries at: index)
>>>>>>             key: ((bibliography entries at: index) fields at: 2) value
>>>>>> ]].
>>>>>> bibliography.
>>>>>> ===================================
>>>>>>
>>>>>> Now I want to write back the corrected file to the .bib ... just
>>>>>> having problems finding which message does the job. I'll keep searching.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Offray
>>>>>>
>>>>>> On 10/16/2014 06:40 AM, Damien Cassou wrote:
>>>>>>> from Damien Pollet:
>>>>>>>
>>>>>>> You will need to load the .bib file from zotero (read the file however
>>>>>>> you like, then pass the stream to the CZ parser). You'll get a
>>>>>>> CZBibSet (I don't recall the name exactly) which represents the
>>>>>>> contents of the file. A Set is composed of entries, each of which has
>>>>>>> a key and a set of fields. Finally, fields accept a few different
>>>>>>> kinds of values.
>>>>>>>
>>>>>>> Your processing is just iterating a set then setting the key of each
>>>>>>> entry (or possibly removing and re-adding the entry, I don't recall if
>>>>>>> it's implemented like a dictionary or more like a list).
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 13, 2014 at 2:57 AM, Offray Vladimir Luna Cárdenas
>>>>>>> <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>
>>>>>>>      Hi,
>>>>>>>
>>>>>>>      I'm using a Zotero collection for keeping track of several
>>>>>>> references I have
>>>>>>>      found for my article about the experience of the
>>>>>>> outline/tree-like metaphor
>>>>>>>      for writing inside Pharo (as soon as I have a presentable
>>>>>>> working draft I
>>>>>>>      hope to share it with you).
>>>>>>>
>>>>>>>      Now I want to make a post-processing of the bibtex file exported
>>>>>>> from
>>>>>>>      Zotero. The idea is to use "shorttitle" field instead to replace
>>>>>>> the Zotero
>>>>>>>      auto-generated one and have custom keys. So for example instead of:
>>>>>>>
>>>>>>>      =======
>>>>>>>      @misc{_holistic_????,
>>>>>>>               title = {Holistic software assessment (Uni Zurich -
>>>>>>> 2011) on Vimeo},
>>>>>>>               shorttitle = {Girba-holistic-2011},
>>>>>>>               url = {http://vimeo.com/42073344?__from=outro-local
>>>>>>>      <http://vimeo.com/42073344?from=outro-local>},
>>>>>>>               urldate = {2014-08-19},
>>>>>>>               note = {00000}
>>>>>>>      }
>>>>>>>
>>>>>>>      =======
>>>>>>>
>>>>>>>
>>>>>>>      I would like to have:
>>>>>>>
>>>>>>>
>>>>>>>      =======
>>>>>>>
>>>>>>>      @misc{Girba-holistic-2011,
>>>>>>>               title = {Holistic software assessment (Uni Zurich -
>>>>>>> 2011) on Vimeo},
>>>>>>>               shorttitle = {Girba-holistic-2011},
>>>>>>>               url = {http://vimeo.com/42073344?__from=outro-local
>>>>>>>      <http://vimeo.com/42073344?from=outro-local>},
>>>>>>>               urldate = {2014-08-19},
>>>>>>>               note = {00000}
>>>>>>>      }
>>>>>>>
>>>>>>>      =======
>>>>>>>
>>>>>>>
>>>>>>>      I have already installed Citizen and open it on the browser to
>>>>>>> see the code,
>>>>>>>      but I can find any place to start with examples.
>>>>>>>
>>>>>>>      Any advice on how to solve this issue will be appreciated.
>>>>>>>
>>>>>>>      Cheers,
>>>>>>>
>>>>>>>      Offray
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Damien Cassou
>>>>>>> http://damiencassou.seasidehosting.st
>>>>>>>
>>>>>>> "Success is the ability to go from one failure to another without losing
>>>>>>> enthusiasm."
>>>>>>> Winston Churchill
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
>