A playground for Markdown?

Hi,

Can you explain to me the abstractions you mention about pandoc
because I'm curious?

I think that the best places to start are the Filters documentation[1] and the Pandoc types package [2]. They essentially show how Pandoc provides structures to build a document programmaticaly including reading, exporting and manipulate it. Filters are explained in Haskell, but the documentation shows how you can use other languages to access the doc programmability.

[1] http://pandoc.org/filters.html#summary
[2] https://hackage.haskell.org/package/pandoc-types

Quoting from the filters Summary:

"""
Pandoc consists of a set of readers and writers. When converting a document from one format to another, text is parsed by a reader into pandoc’s intermediate representation of the document—an “abstract syntax tree” or AST—which is then converted by the writer into the target format. The pandoc AST format is defined in the module Text.Pandoc.Definition in pandoc-types.

A “filter” is a program that modifies the AST, between the reader and the writer:

INPUT --reader--> AST --filter--> AST --writer--> OUTPUT

Filters are “pipes” that read from standard input and write to standard output.
"""

BTW you did not say:
    Ok guys I want to write a Markdown parser (to contribute to
pillar, which would be cool.)
    You said I want to have a playground to edit markdown. This is
quite different to me.

Because the second thing (Markdown support inside the Playground) is the one I want to contribute to Pharo(!), not to Pillar. Contributing should not be so hard, but each time I bring this theme is an uphill battle in just getting the initial exploration questions answered (because they're about Markdown and not about Pillar). I think that Markdown support in Pharo is healthy for Pharo and a meaningful learning experience for me. At least I should be able to test that by myself in this community with a little of support like "hey we have CommonMark inside PettitParser" or "here is how you increase the Playground Syntax support, by specializing this class". 

About the not invented syndrome remarks, why then don't we drop Pharo?
Now if each time we want to install pharo we should install Python,
why not installing lua and Ruby and a bit of Javascript in case.

Because is not an all or nothing proposal and it has never been. It's about Pharo *playing well with* others, not about *being replaced by* others. I don't see any worth in taking this stuff to the extremes. When someone advise me Pharo, I don't think that he/she is advising me to drop my operative system and to run Pharo from bare metal because you go the Pharo way or the No Way(!).

You are under estimating the effort that we put in Pillar. We
extracted pillar from pier to be able to have an autonomous document
system with which we can write good pdf and good HTML for
documentation point of view. Because people did not like to write in
LaTeX.
Now if we would stop something each time somebody mention something
else we would be doing python, Javascript or ...
all the time. 

I don't underestimate anyone's work. Nor in Pillar, nor in Pharo, nor in Pandoc. In fact trying to make them work better together and giving back to this community by making software with Pharo and teaching people Pharo (for free) are clear examples of my appreciation for the work done in several fronts, inside and outside Pharo. The choice of LaTeX and HTML in your case, shows practical appreciation for those formats and I don't think that we should reimplement them in Pharo (despite of several advantages of live objects as documents instead of dead-files). Instead of putting valuations I have never done on me, I would advice to try seeing worth in the reasons behind a lot of people outside this community is using (Pandoc's) Markdown for documentation and the added value that adding Markdown support brings by having more people exposed to Pharo ecosystem, ideas, community, values and technology.

So yes I'm dense and I focus on my stuff and my stuff is Pharo.
And yes I do not care about pandoc because I want to write my
exporters and tools in Pharo and I will fight for that.

But we don't need to fight for that. I value your decisions and work and I think that is healthy to ask for reciprocity and build from that, instead of wasting time because we don't think the same. Diversity is healthy in a community. The only thing I'm asking for (besides such reciprocity) is pointers to start *my* exploration.

This is why I want to use Scale instead of *&%*^)&%^ bash.
And this is why I spent most of time trying to improve Pharo because
it is not at the level I want but I can contribute
and make it better.

Stef

I don't use bash or Json config files for anything related with documentation in Grafoscopio. Just plain Pharo inside an interactive "%metadata" playground node in the Grafoscopio notebook. Spoon, Coral, Scale, would be welcomed when integration with them is ready. I spent good time also trying to make my contributions and improvements to Pharo. We don't need to fight because our ways of contribution are different.

Cheers,

Offray

Offray Vladimir Luna Cárdenas-2 wrote
> no problem with adding *extra* support for Markdown
> (not killing Pillar) :-).

Go, go, go!! It seems clear (from this thread and others) that the important
thing about Pillar is the /model/, not the syntax. Since it now also seems
clear that no one is asking the Pillar devs to do the work, how could
facilities to import/export other formats do anything but support whatever
we are trying to achieve with Pillar? The reality is that there is a wealth
of information already in other common formats. It seems obvious that we
would want to have access!



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

FYI Pillar has the following architecture

Input --Reader -> AST -> (Transformer -> AST)*  -Outputers (visitor) -> Save

So nothing new under the sun. Notice that we can do several
transformation passes
like expansions.


So you will code your own text Model, AST, and visitor?
Did you see that GT people are using Pillar model for the their editor?



I think that Markdown support in Pharo is
> healthy for Pharo and a meaningful learning experience for me. At least I
> should be able to test that by myself in this community with a little of
> support like "hey we have CommonMark inside PettitParser" or "here is how
> you increase the Playground Syntax support, by specializing this class".

Go ahead and learn.


You are exagerating and more important under estimating the COST of
using multiple systems.
Let me take four simple cases

- We spend hours debugging Yaml to embedd bash into a travis (no
debugger of course) and we failed
while we try to understand why the validator went wrong and did not
like space (we tried many many combinations of `"' and more.
And gave up. Guille is much better than me and Andrew on bash and we
all three failed.

- Pillar was using external mustache unix based library and I forced
them at that time to use Norbert implementation
and we can debug it. We can avoid to have bugs because converting a
stream into a file then invoking make then reloading the file.

- Looking during two days for a bug in bash script that tells that
eveyrthing is ok while one package that should have been loaded
was dropped. Now we use metacello instead relying on bash now.
Do you want more... I have tons of them like that ...
Each time you have a different stack you had another level of
complexity and it is not linear but exponential.
This is the same with servers and processes.

Pillar archetypes were zipped on a Jenkins server that may be blocked
from time to time. Smalltalkhub may be slow and the bash
will continue without failing and your installation breaks. So we are
flushing everything.

No more mustache on disc, no more spurious metadata, no more separate
server, no more make (see below I was not even able to add one entry
to the help), no more
cp -r (that is not compatible on different OSes!!!). Just plain pharo
as much as possible.






# Some shell magic to auto-document the main targets. To have a target appear in
# the output, add a short, one-line comment with a double ## on the same line as
# the target.
#
# See http://marmelab.com/blog/2016/02/29/auto-documented-makefile.html

.phony: help

help: ## Describe the main targets (this list)
@echo "Main targets you can build:"
@awk -F ':|## *' \
'/^[^\t].+:.*##/ {\
printf "  \033[36m%s\033[0m#%s\n", $$1, $$NF \
}' $(MAKEFILE_LIST) \
| column -s# -t
@[ -n "$(ALTERNATEPRINTFORMATS)" ] \
&& echo "Print format alternatives: pdf $(ALTERNATEPRINTFORMATS)"
@echo "Combined format+volume targets: pdfbook, htmlchapters…"
@echo "To make a single specific file/format, ask for it explicitly:"
@echo "  make $(OUTPUTDIRECTORY)/$(firstword $(CHAPTERS)).pdf"

# Check that given variables are set and all have non-empty values,
# die with an error otherwise.
#
# Params:
#   1. Variable name(s) to test.
#   2. (optional) Error message to print.
#
# See http://stackoverflow.com/questions/10858261/abort-makefile-if-variable-not-set
check_defined = \
$(strip $(foreach 1,$1, \
$(call __check_defined,$1,$(strip $(value 2)))))
__check_defined = \
$(if $(value $1),, \
$(error Undefined setting $1$(if $2, ($2))))





The argument that Coca cola is good because many people drink it
does not hold. You see Pythoners have their own systems with their own
syntax. So what?


Pandoc is the same as Pillar. I do not care about the Pillar syntax.
I care that this is implemented in Pharo.
Now I will not write a markdown parser myself because I have what I need.
if other people propose a not super complex parser for markdown then
we can integrate it and let people use their own.




> So yes I'm dense and I focus on my stuff and my stuff is Pharo.
> And yes I do not care about pandoc because I want to write my
> exporters and tools in Pharo and I will fight for that.
>
>
> But we don't need to fight for that.

Yes because we should do the web in Javascript and the math in R and ....
This is always like that.


I value your decisions and work and I
> think that is healthy to ask for reciprocity and build from that, instead of
> wasting time because we don't think the same. Diversity is healthy in a
> community. The only thing I'm asking for (besides such reciprocity) is
> pointers to start *my* exploration.


Then explore.
You can also learn from Pillar :)
If you want to have a look the pipeline in version 5 is obscrure and
in the new pipeline guille dramatically cleaned it.
and we will continue.

Ok try and learn.



>
> Cheers,
>
> Offray

you forget impedence mismatch and reducing accidental complexity.
Now I really hope that we will be able to have Pharo inside a dll so
that we can embed everywhere and a much better interaction with
OS and libraries out there. But if you get access to a C library you
have to pay the price of the API exposition and glue.

Stef

On Sun, Dec 31, 2017 at 4:41 AM, Kjell Godo <[hidden email]> wrote:

I like also the Borg approach and having Pharo ideas and tech glued/enhancing not only to other techs, but also to other cultural practices, like writing . That's why I'm supporting (Pandoc's) Markdown and data storytelling in Pharo. They connect us to several communities beyond programmers: activists, journalists, (not computer science) scientists and researchers.

Cheers,

Offray

i like Pharo

     and i want Pharo to do well and i want to use Pharo

     and i like very much all the work being done on Pharo yes

          I’m very impressed and hopeful 

     and i would not wish for that to slow down

but can Pharo do everything by itself? i don’t think so

i used to feel like i could not leave Smalltalk

     and andy’s comments and bullet proof correctness 

     because the outside world was such a barrier to entry

     and the outside world was always missing something

     that makes Smalltalk essential and so i felt like i 

     could not use those other things

but if Smalltalk could be added to those other things

     then they could be ok    and i could see using them

     like Racket clojure CommonLisp Shen Coq Haskell SwiProlog 

     StrawberryProlog Pixie Forth Factor etc   Ruby except it’s got a hell of a syntax

     erlang

     if Smalltalk could be added to them then they could be ok

     but without Smalltalk they look just like hell      to me  what a travesty

     unusable 

i don’t like anything that smells like C all the C like like java etc

     although turbo C was very nice    so they had to kill it   get rid of it

     because it’s like they are so inferior and yet everybody’s 

     first language is just like the quick sand you sink right into

     never to be heard from again because God forbid you

     should have to go through that again once is enough

     so you sink right in never to be heard from again in any other venue

     and it’s like these were just the langs that worked first

     they got the network effect first and became the monopoly

     and that’s why people use them because God forbid

     for bid that you should learn something new we all say

if Smalltalk could be like the front end of some other things

     Racket CommonLisp Shen etc then That is what i want

and Smalltalk could be a back end too for things that it

     does well like i want to use gemstone etc

let everything do what it does best and let them all talk

     to each other is what i want

that’s the BorgLisp idea

Offray Vladimir Luna Cárdenas
<[hidden email]> wrote:

Perhaps just reading the original Pier and Magritte thesis work by Lukas
Renggli would be helpful then. Anything new or better in Pandoc?

Stephan

Hi,

 

I have argued time and again and in length about Markdown support in Pharo, so I will not do it again. I'll just repeat that, in order to make Pharo less isolated, Git support for managing software source code has the strategic importance, in the same way that Markdown support for managing documentation source code has strategic importance. This doesn't preclude support for native/alternative DVCS in the software front (Monticello, Fossil, etc) or markup languages in the documentation one (Pillar, Dokuwiki, t2tags, etc).

It's kinda funny because me and Esteban have been very early on huge supporters of git and github when the rest of the community was mildly interested to passionately against it.  I have been very vocal about it, to the point of annoying people and being accused of just supporting a overhyped product. 4 years later working with git is the official way of doing things and we even have our very own github client in the image.

Well it's kinda funny to experience the same resistance in the documentation front that we have in the Git/GitHub, even using the same old arguments. I don't like Git/GitHub, but I understand the strategic importance of it for the community. In the documentation front, each time Markdown is mentioned (with any external tool that works with it) we restart from zero to have the same old arguments (just look at the current thread).

[...]

My personal opinion on this is that using syntax for documentation in 2017 feels to me like stone age. We did not even do this in 1980. I will dare to predict and you can bookmark this post that Markdow will slowly disappear to be replaced with the proper way of doing documentation and that is through a dedicated documentation editor like Libreoffice.  Even Google docs seem to return in favour of Markdown.

In 1980 we don't have Wikipedia and more that 5 million of articles written using a light markup language (LML). We didn't have GitHub and several devs communities writing collaboratively documentation using same kind of languages. We didn't have Ulysses[2], Ghost[3] and other apps directed towards non-devs, but bloggers and writers, using/advocating plain text and LMLs to keep you focused on writing (not typography or margins) and being able to write in small form factor screens. So I see the return in favor of LML as a progress over complicated and bloated XML files or cumbersome verbose markup languages for documentation (and their supporting GUIs).

[1] https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
[2] https://ulyssesapp.com/
[3] https://ghost.org/

The wide range of documentation formats take only a tiny fraction of the market. Adobe may have lost the war with Flash but when it comes to PDF they have won and won it easily. 

EPub for example is the least relevant format nowadays, it used be widely popular format, nowhere near to pdf but popular indeed at the time of PDAs but then iPhone came out and killed it together with Flash. Nowdays is used mostly on Kindle which is also another device that has lost most of its popularity.

ePub is still important for librarians working on public access to several works, in places where people (fortunately) doesn't have and iPhone, like in the wide Global South, from Mexico to Patagonia and beyond. In fact, recently we have a talk in the Grafoscopio mailing list about exporting to ePub the Data Driven Journalist Handbook (with the Pandoc source it was just matter of changing one parameter and voilà).

I consider myself an expert on documentation because my day job is being a lawyer and basically what we do is we write complex technical documentation for courts. Especially my kind of lawyer that do not do criminal law. I am writing legal documentation in a week what takes our community to write in years. 

I have been also close to documentation since 2002 (and I like to write long before that). From 2004 to 2008 we have the biggest wiki collection of Spanish recipes and how-to's about FLOSS with 4000 community articles. Of course this was a project in the "Global South", so nobody noticed it then, and now it slept lost in the cyberspace and somebody's offline backup. I was the main translator of TeXmacs[4] documentation to Spanish. An interactive mathematical documentation system with multiple backens long before Jupyter. So yes, I'm pretty sure that this community contains a lot of expertise in the documentation front.

[4] http://texmacs.org/

Microsoft Office has been monopolising our field to an extreme extend and I must be one of those rare exceptions that I use Libreoffice and quite frankly I dont blame them because it has its issues. 

No I am not a believer in Markdown which usage became relevant only because Github supported it out of the box. With Github Markdown is nothing.

Well, Markdown has several use cases beyond programmers and beyond GitHub, as shown above (most programmers fail to see this).

I am not a believer in LaText again, another specialised software that is used by a very specific community.  

I think its syntax is too verbose, but you don't need it all the time. You could write in (Pandoc's) Markdown (or Pillar) and tweak the parts you need using only LaTeX in the places you want, leveraging decades of experience in layout, templates, typography or kerning, without paying upfront expertise you don't need. So combining LML with LaTeX and its engines is pretty powerful (that's the approach used in most LML systems to produce PDFs).

I am not a believe in Pandoc, why bother with a million formats when only 2 dominate the market ?

Because people use them, and fortunately reality is not only about market dominance dictate and its simplistic view. To the wikipedians, the format that dominates their reality is the one in their wiki, as happens with any wiki engine user, with their particular syntax. A lot of small communities are better serve by their particular LML despite of the "global market" efforts to reduce reality to the two oligopolistic "options".

[...]

Why ?

Because either they are unnecessary hard to use or limited to what they can do. 

You can have a combination of extensible LML that combine with HTML, LaTeX or YAML, so they're easy to learn and not so limited.

If it was up to me, Libreoffice would have been our official way of documentation and the only alternative would have been an in image editor probably leveraging the existing Libreoffice API.

I am a GUI guy, anything that helps me type less and visualise more its far better option to the alternatives. 

Fortunately this was a community decision and not up to a single person. Good luck with a XML diff for a document wrote among several authors(!). TeXmacs for me had a lot of potential: WYSIWYM edition, math typography, diverse interactive backends, S expression all the way down, Lisp (Guile) scripting. But, because of the time it started, the toolkit was kind of isolated with a lot of ad-hoc, in-house solutions in a small community (at the end some Qt front-end was implemented). With Pharo we have the advantage of a continuum between the app, the IDE, the document, and that's something nobody has so evolved until now (not even the people at Mathematica or Jupyter Lab) and that makes a huge difference in the future of the application and its adaptability to the future/context.

But I have never stopped anyone from doing anything nor I tried to discourage. 

I can guarantee to you the same people that have not embraced Pillar is extremely unlikely they will embrace Markdown and Pandoc. There are two things that coders hate a) Writing documentation b) Creating GUIs. This is not going to change any time soon. Well not this millennium at least. 

In any case whatever your goal I wish you very good luck. You are going to need it.  

Coders are not my target and I'm already having luck, because our small diverse local community is already using LML for documentation.  They're librarians, students, philosophers, historians, teachers, journalist, and even we have coders using it. So your millennium of skepticism and resistance have already expired, at least locally :-), which is why I bet on serendipity instead of luck (the encounter of luck, opportunity and preparation).

Cheers,

Offray

On 31/12/17 07:59, Stephan Eggermont wrote:
I will read it. Knowing that both, Pandoc and Pillar share kind of a
similar approach would allow me to connect their AST (I mentioned such
idea before, but now the path is more clear). The issue I care in this
case is not about hot new stuff, but about bridges between cultural
practices, ie, people who is using LMLs for documentation and discussing
them extensively and people in the Pharo community. I need such bridges
myself and I know enough people that could use them.

Cheers,

Offray

Hi,


On 31/12/17 05:13, Stephane Ducasse wrote:
Cool. As I said in this thread in response to Stephan Eggermont, my
concern is not about new under the sun, but in bridging cultural practices.

I don't need to. I will use CommonMark and PettitParser. I imagine
GT-Tools can use the models there in the Playground.

> Go ahead and learn.

Wow! Finally!

That was precisely my point. To show the absurd of exaggeration. I don't
think that proposing a versatile Playground that supports several syntax
highlighting (and some supporting behavior) is the same as proposing to
install everything, kill Pillar efforts or anything like that. In the
same way that using Pharo doesn't mean stop using anything else. But
proposing anything distinct from Pillar as a documentation *alternative*
in Pharo is faced time and again with overreactions.

I use just plain Pharo as much as possible in Grafoscopio. I don't have
any external bash, config, yaml file to produce a document. Jus plain
Pharo dictionaries and dynamic arrays stored *inside* the notebook. I
use Pandoc to import/export to several formats (a place where it is
superb and without any closer competitor), we started to use some Lua
scripts (which are already supported in Pandoc) to query/script its
document model and I have thought in using Pharo Mustache to program the
Pandoc Templates. So in my "borg approach" I glue Pharo to existing
ecosystems (including techs and communities) to improve them, while
making bridges between what they already know and what Pharo has to offer.

>
>
>
>
> # Some shell magic to auto-document the main targets. To have a target appear in
> # the output, add a short, one-line comment with a double ## on the same line as
> # the target.
> #
> # See http://marmelab.com/blog/2016/02/29/auto-documented-makefile.html

I document targets as a "%metadata" node inside the Grafoscopio
notebook. No shell magic beyond Pandoc one line flags.


I'm not using such ad-populum argument. I'm talking about *bridges* to
what people is already using (in the backend) so we can show and use
Pharo with them (in the front end) for data storytelling.

> Pandoc is the same as Pillar.

Maybe in the technical side, but with a more diverse and agile community
devoted exclusively to think about light markup languages. Bridging with
other communities shouldn't be seen as negative.

> I do not care about the Pillar syntax.
> I care that this is implemented in Pharo.
> Now I will not write a markdown parser myself because I have what I need.
> if other people propose a not super complex parser for markdown then
> we can integrate it and let people use their own.
>

Well I just want Markdown syntax highlighting in the Playground (besides
the current Smalltalk one). It amaze me how much resistance about what I
want to explore and can be able to contribute, it's faced every time.

Well is not. Having bridges with R (like the ones we have now) doesn't
preclude or diminish the work in Polymath. Or having Garage to bridge
with external data bases doesn't preclude any work in native OODBs or
bridges with JSON don't preclude any work STON. Having bridges with
Pandoc/Markdown, doesn't preclude/diminish any work with Pillar. I have
told this since the first thread about this subject, but you insist in a
confrontational approach. I will not engage in it when both can coexist
and be helpful for this and other communities.

I will start my exploration with PetitParser and CommonMark.

That's all I want: explore and learn. I finally have pointers to where
to start. It just be more fluent if I don't have to fight an uphill
battle anytime Markdown support is mentioned.

Cheers,

Offray

Offray Vladimir Luna Cárdenas-2 wrote
> a versatile Playground that supports several syntax
> highlighting (and some supporting behavior)

That actually could be amazing because sometimes I'm forced to use other
syntaxes (especially markups nowadays) and I'd probably be much happier
editing Markdown in Pharo than in the GH web UI! This actually just reminded
me of Lukas' Helvetia and Language Boxes. Offray, I have no idea if it's
relevant or useful (the project was from 2009), but you may want to have a
look. Here's one link: http://scg.unibe.ch/research/helvetia



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

The pipeline was not really first class in lukas work.
It was added incrementally. Now we have a real architecture.

On Sun, Dec 31, 2017 at 1:59 PM, Stephan Eggermont <[hidden email]> wrote:

Nice!

Doru

> ​Not currently updated to Pharo 6.1 or 7.0
>
> --
> Serge Stinckwich
> UMI UMMISCO 209 (IRD/UPMC/UY1)
> "Programs must be written for people to read, and only incidentally for machines to execute."
> http://www.doesnotunderstand.org/

--
www.tudorgirba.com
www.feenk.com

"Be rather willing to give than demanding to get."