Flexibility of ZnUrl (For Metacello evolution)

Hi,

In order to improve the usage of Pharo and the git integration there is
some discussions around Metacello in order to be able to:

- Load a private git project from Metacello API
- Load a project hosted on a private server from Metacello API
- Load a project hosted on a private server with non default ssh port
from Metacello API
- Depend on a private git project in a baseline
- Depend on a project hosted on a private server in a baseline
- Depend on a project hosted on a private server with non default ssh
port in a baseline

To achieve this, Metacello needs to be able to manage urls like:

- '[hidden email]:Projet/Bazard.git'
- 'ssh://[hidden email]:3456/Projet/Bazard.git'

and many others.

Dale wants to use Zinc to manage urls's in Pharo.

It works great for this url:

'ssh://[hidden email]:3456/Projet/Bazard.git' asUrl

It gives an instance of ZnUrl of the ssh scheme, gitlab.ferlicot.fr as
host, 3456 as port, git as username and #('Project' 'Bazard.git') as
segments.

Now the problem is that it does not work well for this url:

'[hidden email]:Projet/Bazard.git' asUrl

It gives an instance of ZnUrl with #('[hidden email]:Projet'
'Bazard.git') as segments.

What do you think guys about this issue? Could the flexibility of ZnUrls
be improved in order to be able to use it? Should Metacello use another way?

To see the full discussions:
- Github issue: https://github.com/Metacello/metacello/issues/474
- Metacello group discussion:
https://groups.google.com/forum/#!topic/metacello/QbpngL6Jhg8

Have a nice day :)

--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France

 ZnUrl fromString: '[hidden email]/Projet/Bazard.git' defaultScheme: #http.

 ZnUrl fromString: '[hidden email]/Projet/Bazard.git' defaultScheme: #ssh.

URI/URL are put in classes or types based on their scheme. If there is no explicit scheme in the URI/URL itself, you have to supply a default one.

That being said, the general syntax supported by ZnUrl is approximately

  scheme://username:password@host:port/segments?query#fragment

The colon right after the host is reserved for a port, a path has to start with a / (unless you are using relative URL's, but then you need a parent context). This is why I changed your example, to make it work.

There literally exist hundreds of schemes, some with very weird syntax. That is indeed the prerogative of a scheme, to define its own special syntax. However, the straightforward approach of the ZnUrl parser cannot recognise those.

The question then becomes: how to provide an extension mechanism for exotic schemes ?

Although the question has come up a couple of times, it has never been very urgent.

I know that some people have programmed around this limitation, maybe they could tell us what they did.

Why ZnURL should be flexible to treat something that is not a URI? An URI is something like

<some ascii chars>:<some arbitrary stuff>

The important part is the colon. If you split on it then the first part is addressing a name that can be resolved to get an entity that can digest the second part. That‘s it! A locator is just a special form of URI that cheats by saying that the second part can be used directly to locate the resource (that‘s the difference by identifiable resources and locatable resources meaning the ones that really exist)
So I cannot see how this can work. But nobody says you need to fold this into ZnURL. A Client is free to do something like if it starts with git I prepend that with ssh:// or git:// or git+ssh://
So the point is that if your model is the URI (it does not have to) then you should create valid URLs by adding missing stuff. Otherwise just have a model that treats URLs and your special cases.

Norbert

On 29 Dec 2017, at 10:35, Cyril Ferlicot D. [hidden email] wrote:

Hi,

In order to improve the usage of Pharo and the git integration there is
some discussions around Metacello in order to be able to:

- Load a private git project from Metacello API
- Load a project hosted on a private server from Metacello API
- Load a project hosted on a private server with non default ssh port
from Metacello API
- Depend on a private git project in a baseline
- Depend on a project hosted on a private server in a baseline
- Depend on a project hosted on a private server with non default ssh
port in a baseline

To achieve this, Metacello needs to be able to manage urls like:

- '[hidden email]'
- '<a class="moz-txt-link-freetext" href="ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git">ssh://git@...:3456/Projet/Bazard.git'

and many others.

Dale wants to use Zinc to manage urls's in Pharo.

It works great for this url:

'<a class="moz-txt-link-freetext" href="ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git">ssh://git@...:3456/Projet/Bazard.git' asUrl

It gives an instance of ZnUrl of the ssh scheme, gitlab.ferlicot.fr as
host, 3456 as port, git as username and #('Project' 'Bazard.git') as
segments.

Now the problem is that it does not work well for this url:

'[hidden email]' asUrl

 ZnUrl fromString: '[hidden email]' defaultScheme: #http.

 ZnUrl fromString: '[hidden email]' defaultScheme: #ssh.

URI/URL are put in classes or types based on their scheme. If there is no explicit scheme in the URI/URL itself, you have to supply a default one.

That being said, the general syntax supported by ZnUrl is approximately

  scheme://username:password@host:port/segments?query#fragment

The colon right after the host is reserved for a port, a path has to start with a / (unless you are using relative URL's, but then you need a parent context). This is why I changed your example, to make it work.

There literally exist hundreds of schemes, some with very weird syntax. That is indeed the prerogative of a scheme, to define its own special syntax. However, the straightforward approach of the ZnUrl parser cannot recognise those.

The question then becomes: how to provide an extension mechanism for exotic schemes ?

Although the question has come up a couple of times, it has never been very urgent.

I know that some people have programmed around this limitation, maybe they could tell us what they did.

Thanks Sven.

As Cyril points out we are considering adding a new batch of urls for repository descriptions and the first order of business is to solve the parsing issue --- I would prefer NOT to bring back the old Url hiearchy --- which is what I would do if forced to provide a Metacello-specific solution ...

The overall goal is to be able have Metacello execute an expression that looks like the following:

  repositoryDescriptionString asUrl asMetacelloRepository

where the repositoryDescriptionString could be one of the following:

  'http://seaside.gemtalksystems.com/ss/metacello'
  'github://dalehenrich/filetree:pharo1.1/repository'
  'filetree://$GS_HOME/shared/repos/metacello/repository'

Note that the last two repository descriptions are accepted by ZnUrl, but require additional parsing tease out the required fields.

Cyril's examples are what is used by git as a repository description:

  '<a class="moz-txt-link-freetext" href="ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git">ssh://git@...:3456/Projet/Bazard.git'
  '[hidden email]'

Note that in these two examples from Cyril, we are missing some critical information: commitish and repository path need for a complete Metacello repository description, so I don't consider these repository descriptions complete.

Thierry Goubier addressed these particular issues with his GitFileTree repository description[1]:

  'gitfiletree://gitlab.ferlicot.fr:3456:Projet/Bazard:dev/src'

My inclination is to base the custom urls on Thierry's gitfiletree model, but we are still in the process of gathering requirements and parsing technology will be an issue once we've settled on the standard set of repository descriptions.

In the past Thierry has mentioned that "I would have reused ZnUrl to do that [iceberg using regex], but maybe they wanted to avoid ZnUrl errors about unknown schemes. I had to copy/paste some of the ZnUrl code into GitFileTree for that purpose."[2]

Looking at Thierry's implementation[1] it does look like he is able to pack all of the critical information into the repository description, but as with the github:// and filetree:// schemes additional parsing is required to tease out all of the fields ... I have not seen examples of the repository descriptions used by iceberg (yet) ...

So, Sven, it would be interesting to hear from you about any ideas that you might have to allow for the handling of custom url parsing within the ZnUrl framework... ideally we'd not have to do any scheme-specific post-parsing of the url and the #asMetacelloRepository method could be implemented to just send messages to extract the various values of the custom fields ... as with the old Url hierarchy, allowing subclasses to implement custom parsers based on the scheme could be done, but it isn't the only approach.

I do want to point out that the #asMetacelloRepository is intended to return a repository instance from which packages can be loaded by Metacello... and the intent is to allow developers to be able to choose which platform tool will be used to manage the repository.

The idea is that platform tools will register to handle a particular scheme, so the a developer can choose whether to use gitfiletree or iceberg or <the next generation repository management tool> to manage the repository for the github:// scheme, etc. This would eliminate the current implementation where we have a fixed map of schemes to tools and developers to invent a custom url (and patch Metacello) for each new tool.

Dale

[1] https://github.com/dalehenrich/filetree/blob/734eed46ea57ebf5e24e5d935768bd49727fc22f/repository/MonticelloFileTree-Git.package/MCFileTreeGitRepository.class/class/basicFromUrl..st
[2] https://github.com/Metacello/metacello/issues/474#issuecomment-350871020