[ANN] Phoedown - Markdown to HTML

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] Phoedown - Markdown to HTML

Pierce Ng-3
Hi all,

I've published Phoedown, an FFI to hoedown, the standards compliant, fast, secure Markdown
processing library written in C.

- https://github.com/PierceNg/Phoedown
- https://github.com/hoedown/hoedown

A simple example:

    | md |
    md := (FileSystem memory / 'somefile.md')
    writeStreamDo: [ :ws |
    ws nextPutAll:
    '
     ```smalltalk
    Transcript show: ''Happy New Year!''; cr
     ```
    ' ];
    contents.
    HdHtmlRenderer new
    setMarkdownExtension: HdMarkdownExtensions FencedCode;
    setMarkdownExtension: HdMarkdownExtensions NoIntraEmphasis;
    render: md

This is the output:

    <pre><code class="language-smalltalk">Transcript show: &#39;Happy New Year!&#39;; cr
    </code></pre>
 
Pierce


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Tim Mackinnon
I’m loving all the little utility libraries coming out (and appreciate the write ups on how things are done).

I’m curious why you chose to use ffi for this one in particular as it would seem to be quite straightforward to do it all in native Smalltalk (not to down play your integration in any way of course). Was performance a big thing? Or was it simply relying on others to keep it up to date?

Tim





Sent from my iPhone

> On 1 Jan 2020, at 10:40, Pierce Ng <[hidden email]> wrote:
>
> Hi all,
>
> I've published Phoedown, an FFI to hoedown, the standards compliant, fast, secure Markdown
> processing library written in C.
>
> - https://github.com/PierceNg/Phoedown
> - https://github.com/hoedown/hoedown
>
> A simple example:
>
>    | md |
>    md := (FileSystem memory / 'somefile.md')
>        writeStreamDo: [ :ws |
>            ws nextPutAll:
>    '
>     ```smalltalk
>    Transcript show: ''Happy New Year!''; cr
>     ```
>    ' ];
>        contents.
>    HdHtmlRenderer new
>        setMarkdownExtension: HdMarkdownExtensions FencedCode;
>        setMarkdownExtension: HdMarkdownExtensions NoIntraEmphasis;
>        render: md
>
> This is the output:
>
>    <pre><code class="language-smalltalk">Transcript show: &#39;Happy New Year!&#39;; cr
>    </code></pre>
>
> Pierce
>
>


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Pierce Ng-3
On Wed, Jan 01, 2020 at 02:35:03PM +0100, Tim Mackinnon wrote:
> I’m curious why you chose to use ffi for this one in particular as it
> would seem to be quite straightforward to do it all in native
> Smalltalk (not to down play your integration in any way of course).
> Was performance a big thing? Or was it simply relying on others to
> keep it up to date?

I wanted something that works out of the box. I did not want to
implement MD parsing myself.

Also I saw hoedown's wiki page on bindings and thought I'd put Smalltalk
on the map there.

  https://github.com/hoedown/hoedown/wiki/Bindings

Pierce


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Sean P. DeNigris
Administrator
Pierce Ng-3 wrote
> Also I saw hoedown's wiki page on bindings and thought I'd put Smalltalk
> on the map there.

Great!



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Tim Mackinnon
In reply to this post by Pierce Ng-3
That makes sense - I’m getting the impression that ffi is getting very easy these days and maybe we should use it more to focus on “other” things.

It does complicate deployment a bit - but if you’re server/cloud based then that’s less of an issue anyway .

This said, many of our nastiest bugs seem to centre around ffi a c based libraries - but maybe using them more will tease this out.

Tim

Sent from my iPhone

> On 2 Jan 2020, at 01:27, Pierce Ng <[hidden email]> wrote:
>
>> On Wed, Jan 01, 2020 at 02:35:03PM +0100, Tim Mackinnon wrote:
>> I’m curious why you chose to use ffi for this one in particular as it
>> would seem to be quite straightforward to do it all in native
>> Smalltalk (not to down play your integration in any way of course).
>> Was performance a big thing? Or was it simply relying on others to
>> keep it up to date?
>
> I wanted something that works out of the box. I did not want to
> implement MD parsing myself.
>
> Also I saw hoedown's wiki page on bindings and thought I'd put Smalltalk
> on the map there.
>
>  https://github.com/hoedown/hoedown/wiki/Bindings
>
> Pierce
>
>


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Sean P. DeNigris
Administrator
Tim Mackinnon wrote
> I’m getting the impression that ffi is getting very easy these days and
> maybe we should use it more to focus on “other” things... This said, many
> of our nastiest bugs...

While I dream of a world where everything is in-image as pure Smalltalk,
given the reality of limited manpower, I think of outside library use as a
way to "cheat" and get a lot more from that limited engineering resource. I
look forward to the mythical day where "everything's done" and we can come
back around and replace some of these with Smalltalk, but then of course
we'll own the maintenance. That said, as you pointed out, bugs seem to be
more severe and difficult to diagnose, so we'll see how it goes as FFI use
becomes more and more common...



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Esteban A. Maringolo
On Thu, Jan 2, 2020 at 3:47 PM Sean P. DeNigris <[hidden email]> wrote:
>
> Tim Mackinnon wrote
> > I’m getting the impression that ffi is getting very easy these days and
> > maybe we should use it more to focus on “other” things... This said, many
> > of our nastiest bugs...
>
> That said, as you pointed out, bugs seem to be
> more severe and difficult to diagnose, so we'll see how it goes as FFI use
> becomes more and more common...

It seems to have played very well for the Python folks.
It should work even better for Pharo. :)



Esteban A. Maringolo

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Pierce Ng-3
In reply to this post by Sean P. DeNigris
On Thu, Jan 02, 2020 at 12:56:55PM -0600, Sean P. DeNigris wrote:
> bugs seem to be
> more severe and difficult to diagnose, so we'll see how it goes as FFI use
> becomes more and more common...

There's a saying about (aspects of) building software which source I
cannot recall:

  If it hurts, do it more frequently.

I believe FFI should be pushed as much as possible to work out bugs and
develop new capabilities. Looking forward to playing with threaded FFI.

Pierce

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Pharo Smalltalk Users mailing list
+1

> On Jan 2, 2020, at 9:24 PM, Pierce Ng <[hidden email]> wrote:
>
> On Thu, Jan 02, 2020 at 12:56:55PM -0600, Sean P. DeNigris wrote:
>> bugs seem to be
>> more severe and difficult to diagnose, so we'll see how it goes as FFI use
>> becomes more and more common...
>
> There's a saying about (aspects of) building software which source I
> cannot recall:
>
>  If it hurts, do it more frequently.
>
> I believe FFI should be pushed as much as possible to work out bugs and
> develop new capabilities. Looking forward to playing with threaded FFI.
>
> Pierce
>


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

EstebanLM
In reply to this post by Pierce Ng-3
Hi,

Good work :)

{ #category : #finalization }
HdBufferOpaque class >> finalizeResourceData: aHandle [
        HdFFILibrary uniqueInstance
                ffiCall: #(void hoedown_buffer_free (ExternalAddress aHandle))
]

wow… does this works at all?
I admit it is not how it is intended to work :P


> On 3 Jan 2020, at 06:24, Pierce Ng <[hidden email]> wrote:
>
> On Thu, Jan 02, 2020 at 12:56:55PM -0600, Sean P. DeNigris wrote:
>> bugs seem to be
>> more severe and difficult to diagnose, so we'll see how it goes as FFI use
>> becomes more and more common...
>
> There's a saying about (aspects of) building software which source I
> cannot recall:
>
>  If it hurts, do it more frequently.
>
> I believe FFI should be pushed as much as possible to work out bugs and
> develop new capabilities. Looking forward to playing with threaded FFI.

+1
FFI introduces a whole new universe of possibilities we cannot reasonable cover in-image nowadays.

Now, while *there are* problems still to solve, I think people have the wrong impression the FFI layer is causing most of the bugs around (TTF for example) while this is not true at all :)
For example, recent bug around memory corruption was claimed several times it was because of FFI (and even I thought it was around it at the beginning) while in fact it was a problem on how compaction was working.
Other example, TTF itself: problems were around when we had a plugin and we still have some with the FFI implementation. But the problem is not around the plugin or the FFI implementation but around the bad usage (not thread safe) we do of it in-image.

The reason why people think some problems are FFI related is because we are pushing Pharo in directions not-fully-tested before. The usage of FFI just unveils a lot of problems on how we handle things there were present before, but they were not as visible as they are now.

Esteban

>
> Pierce
>


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Kasper Osterbye
Just while we are at the notion of markdowns, I have at https://github.com/kasperosterbye/PillarRichTextRender a first build of a pharo based github markdown to pillar. From pillar you can go to html.

But just to prove the point of the strength of importing working software from outside using FFI, the markdown parser I am working on currently has two serious limitations:

a) it do not do table (because my project really was about rendering pillar inside the image, and could not figure out how to render tables inlined in text).
b) image address has to be hard-coded - cannot be relative to some url.

— Kasper
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Ramon Leon-5
In reply to this post by Sean P. DeNigris
On 2020-01-02 10:56 a.m., Sean P. DeNigris wrote:
> While I dream of a world where everything is in-image as pure Smalltalk,
> given the reality of limited manpower, I think of outside library use as a
> way to "cheat" and get a lot more from that limited engineering resource.

Agree, I've used the original Markdown.pl implementation for years same as I would any other shell script, via OSProcess

markdown: someContent
  ^UnixProcess pipeString: someContent throughCommand: (FileDirectory default fullPathFor: 'Markdown.pl')

Never saw a need to rewrite what already works in its original form.

--
Ramón León
VP of Technology
Alliance Reservations Network

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Ramon Leon-5
In reply to this post by Sean P. DeNigris
On 2020-01-02 10:56 a.m., Sean P. DeNigris wrote:
> While I dream of a world where everything is in-image as pure Smalltalk,
> given the reality of limited manpower, I think of outside library use as a
> way to "cheat" and get a lot more from that limited engineering resource.

Agree, I've used the original Markdown.pl implementation for years same as I would any other shell script, via OSProcess

markdown: someContent
  ^UnixProcess pipeString: someContent throughCommand: (FileDirectory default fullPathFor: 'Markdown.pl')

Never saw a need to rewrite what already works in its original form.

--
Ramón León


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

giorgiof
In reply to this post by Ramon Leon-5
Hello, 
first of all, have a wonderful 2020..
sorry for jumping in, I'm not a contributor to Pharo,  and actually never a user of it ( when I work in Smalltalk, I use VW mostly, and still need to find  the time to try Pharo seriously), but I do work in Smalltalk (and Pharo is a dialect of) from early 80'... and , as for many of you, Smalltalk is not the only language I use (I wish it was the most used, but not the only one, because one tool is not enough).
Well, the idea that all should be done in Smalltalk seems to me something that us as a community have had as a goal from day one, but, are us sure it's not something that reduce the power of Smalltalk instead of allowing broader adoption?
  • First, there are lots of libraries and tools already well written, maintained and with full documentation. Why to reinvent the wheels if we can integrate them seamlessly?
  • Second: Smalltalk is not the fastest language out there, there are works that are not to be developed in Smalltalk. If we develop in Smalltalk something very CPU intensive we just made bad advertising for the language. 

Smalltalk is at is  best for modelling difficult problems. Look at Python, it's very popular between  Data Scientist, but it just expose in a nice way an interface to Macchine Learning libraries, For example TensorFlow has a lot of optimised c++ code inside. Python made easy to interface with C like languages, and it has good modelling capabilities, so it was chosen.  But Smalltalk is better on the latter, this is a campground we should dominate, instead we are absent. And Pharo, being open source, could have been a good player in this field.
If we spend our time to reinvent the wheel  we can't get to far ..
Look at Node.js, you can find libraries for connecting everything, people doesn't rewrote everything in JS/ So if you work in Node, you are fast at building stuff not because of the power of the language, but because to the libraries you can pick from the shelf and use.

So I think a wonderful and easy integration framework is time better spent that redoing something already well done on other languages. Object are for reuse, but we try to rebuild...
And on top of the framework, lot of smalltalk classes for an easy usage of the outside work already done. 
Obviously, all I said is not valid if Smalltalk is considered only an experimental language or a playground, but this was, and probably still is, the place for Squeak.  If I remember well, Pharo was born for industrial grade application...


Sorry for this rant, but I liked Smalltalk from day one, the day I read the famous Byte's  article, and  still try to understand why it's not the number one language, but I think we, as a community, did a lot on the wrong direction.
BTW: I still like Smalltalk a lot and really appreciate the work done by all of the Smalltalk communities.

again, have a wonderful year 

giorgio 




On Fri, Jan 3, 2020 at 4:55 PM Ramon Leon <[hidden email]> wrote:
On 2020-01-02 10:56 a.m., Sean P. DeNigris wrote:
> While I dream of a world where everything is in-image as pure Smalltalk,
> given the reality of limited manpower, I think of outside library use as a
> way to "cheat" and get a lot more from that limited engineering resource.

Agree, I've used the original Markdown.pl implementation for years same as I would any other shell script, via OSProcess

markdown: someContent
  ^UnixProcess pipeString: someContent throughCommand: (FileDirectory default fullPathFor: 'Markdown.pl')

Never saw a need to rewrite what already works in its original form.   

--
Ramón León
VP of Technology
Alliance Reservations Network

Reply | Threaded
Open this post in threaded view
|

R: [ANN] Phoedown - Markdown to HTML

Lorenzo

Ciao Giorgio,

 

come stai? Mi fa piacere risentirti dopo tanto tempo.

Forse (visto il successo della tua mail), sarebbe opportuno cominciare collaborazioni in Italia!

 

Sentiamoci.

 

Lorenzo

 

Da: Pharo-users [mailto:[hidden email]] Per conto di giorgio ferraris
Inviato: sabato 4 gennaio 2020 20:38
A: Any question about pharo is welcome <[hidden email]>
Oggetto: Re: [Pharo-users] [ANN] Phoedown - Markdown to HTML

 

Hello, 

first of all, have a wonderful 2020..

sorry for jumping in, I'm not a contributor to Pharo,  and actually never a user of it ( when I work in Smalltalk, I use VW mostly, and still need to find  the time to try Pharo seriously), but I do work in Smalltalk (and Pharo is a dialect of) from early 80'... and , as for many of you, Smalltalk is not the only language I use (I wish it was the most used, but not the only one, because one tool is not enough).

Well, the idea that all should be done in Smalltalk seems to me something that us as a community have had as a goal from day one, but, are us sure it's not something that reduce the power of Smalltalk instead of allowing broader adoption?

  • First, there are lots of libraries and tools already well written, maintained and with full documentation. Why to reinvent the wheels if we can integrate them seamlessly?
  • Second: Smalltalk is not the fastest language out there, there are works that are not to be developed in Smalltalk. If we develop in Smalltalk something very CPU intensive we just made bad advertising for the language. 

 

Smalltalk is at is  best for modelling difficult problems. Look at Python, it's very popular between  Data Scientist, but it just expose in a nice way an interface to Macchine Learning libraries, For example TensorFlow has a lot of optimised c++ code inside. Python made easy to interface with C like languages, and it has good modelling capabilities, so it was chosen.  But Smalltalk is better on the latter, this is a campground we should dominate, instead we are absent. And Pharo, being open source, could have been a good player in this field.

If we spend our time to reinvent the wheel  we can't get to far ..

Look at Node.js, you can find libraries for connecting everything, people doesn't rewrote everything in JS/ So if you work in Node, you are fast at building stuff not because of the power of the language, but because to the libraries you can pick from the shelf and use.

 

So I think a wonderful and easy integration framework is time better spent that redoing something already well done on other languages. Object are for reuse, but we try to rebuild...

And on top of the framework, lot of smalltalk classes for an easy usage of the outside work already done. 

Obviously, all I said is not valid if Smalltalk is considered only an experimental language or a playground, but this was, and probably still is, the place for Squeak.  If I remember well, Pharo was born for industrial grade application...

 

 

Sorry for this rant, but I liked Smalltalk from day one, the day I read the famous Byte's  article, and  still try to understand why it's not the number one language, but I think we, as a community, did a lot on the wrong direction.

BTW: I still like Smalltalk a lot and really appreciate the work done by all of the Smalltalk communities.

 

again, have a wonderful year 

 

giorgio 

 

 

 

 

On Fri, Jan 3, 2020 at 4:55 PM Ramon Leon <[hidden email]> wrote:

On 2020-01-02 10:56 a.m., Sean P. DeNigris wrote:
> While I dream of a world where everything is in-image as pure Smalltalk,
> given the reality of limited manpower, I think of outside library use as a
> way to "cheat" and get a lot more from that limited engineering resource.

Agree, I've used the original Markdown.pl implementation for years same as I would any other shell script, via OSProcess

markdown: someContent
  ^UnixProcess pipeString: someContent throughCommand: (FileDirectory default fullPathFor: 'Markdown.pl')

Never saw a need to rewrite what already works in its original form.   

--
Ramón León
VP of Technology
Alliance Reservations Network

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Phoedown - Markdown to HTML

Offray Vladimir Luna Cárdenas-2
In reply to this post by Pierce Ng-3
Hi,

Really interesting work.

I wonder if something similar could be done to bridge simdjson[1] with
Pharo/NeoJSON (and maybe a fast binary objects serializer) to read huge
JSON streams and files in seconds.

[1] https://github.com/lemire/simdjson

Thanks for this work :-),

Offray

On 1/01/20 4:40 a. m., Pierce Ng wrote:

> Hi all,
>
> I've published Phoedown, an FFI to hoedown, the standards compliant, fast, secure Markdown
> processing library written in C.
>
> - https://github.com/PierceNg/Phoedown
> - https://github.com/hoedown/hoedown
>
> A simple example:
>
>     | md |
>     md := (FileSystem memory / 'somefile.md')
>     writeStreamDo: [ :ws |
>     ws nextPutAll:
>     '
>      ```smalltalk
>     Transcript show: ''Happy New Year!''; cr
>      ```
>     ' ];
>     contents.
>     HdHtmlRenderer new
>     setMarkdownExtension: HdMarkdownExtensions FencedCode;
>     setMarkdownExtension: HdMarkdownExtensions NoIntraEmphasis;
>     render: md
>
> This is the output:
>
>     <pre><code class="language-smalltalk">Transcript show: &#39;Happy New Year!&#39;; cr
>     </code></pre>
>  
> Pierce
>
>