Git URL issues

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Git URL issues

Sven Van Caekenberghe-2
Hi,

I asked this before, but I still have trouble with this.

What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config.

Basically,

$vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_

towards a private server.

In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler):

$ uname -a
Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ curl get.pharo.org/64/70+vm | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3054  100  3054    0     0  57622      0 --:--:-- --:--:-- --:--:-- 57622
Downloading the latest 70 Image:
    http://files.pharo.org/get-files/70/pharo64.zip
Pharo.image
Downloading the latest pharoVM:
        http://files.pharo.org/get-files/70/pharo64-linux-stable.zip
pharo-vm/pharo
Creating starter scripts pharo and pharo-ui

$ ./pharo Pharo.image printVersion
[version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)'

$ ./pharo Pharo.image save build

$ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp

3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
Loading baseline of BaselineOfZTimestamp...
Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache
...finished baseline

$ ./pharo build.image eval ZTimestamp now
2019-10-25T12:28:34Z

So although it gave a warning, it worked fine.

The warning said,

using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
 
so let's try that after cleaning everything:

$ rm -rf pharo-local/ build.*

$ ./pharo Pharo.image save build

$ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp

...RETRY->BaselineOfZTimestamp
...RETRY->BaselineOfZTimestamp
...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git
MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer:

WTF ? Why does this not work (I can perfectly git clone that URL) ?

Of course, this would just be the first step, eventually I want to use my own private server with a URL like

https://deploy:secret@.../scm/xyz/xyz-pharo.git

How do you build/deploy non-public production code using the command line ?

Help would be much appreciated.

Sven


Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

Sven Van Caekenberghe-2
I am still at a loss with this issue.

When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it.

So obviously, Pharo 7 is capable of accessing it.

However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ?

This code is so complex...

> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
>
> Hi,
>
> I asked this before, but I still have trouble with this.
>
> What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config.
>
> Basically,
>
> $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_
>
> towards a private server.
>
> In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler):
>
> $ uname -a
> Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> $ curl get.pharo.org/64/70+vm | bash
>  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                 Dload  Upload   Total   Spent    Left  Speed
> 100  3054  100  3054    0     0  57622      0 --:--:-- --:--:-- --:--:-- 57622
> Downloading the latest 70 Image:
>    http://files.pharo.org/get-files/70/pharo64.zip
> Pharo.image
> Downloading the latest pharoVM:
> http://files.pharo.org/get-files/70/pharo64-linux-stable.zip
> pharo-vm/pharo
> Creating starter scripts pharo and pharo-ui
>
> $ ./pharo Pharo.image printVersion
> [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)'
>
> $ ./pharo Pharo.image save build
>
> $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>
> 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
> Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> Loading baseline of BaselineOfZTimestamp...
> Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache
> ...finished baseline
>
> $ ./pharo build.image eval ZTimestamp now
> 2019-10-25T12:28:34Z
>
> So although it gave a warning, it worked fine.
>
> The warning said,
>
> using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
>
> so let's try that after cleaning everything:
>
> $ rm -rf pharo-local/ build.*
>
> $ ./pharo Pharo.image save build
>
> $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp
>
> ...RETRY->BaselineOfZTimestamp
> ...RETRY->BaselineOfZTimestamp
> ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git
> MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer:
>
> WTF ? Why does this not work (I can perfectly git clone that URL) ?
>
> Of course, this would just be the first step, eventually I want to use my own private server with a URL like
>
> https://deploy:secret@.../scm/xyz/xyz-pharo.git
>
> How do you build/deploy non-public production code using the command line ?
>
> Help would be much appreciated.
>
> Sven
>


Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

EstebanLM
Hi Sven,

You are facing two different problems at the same time :)
1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle). 
If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier.

If you execute first: 

./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl”

Then your execution: 

./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp

Will work without warnings. 

Now, you have another problem: 
2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git
And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time. 
The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol).
But if you still want/need to use HTTPS, then you need to provide your plaintext credentials.
For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them).
To add them programatically you need to do something like this: 

./pharo build.image eval --save "IceCredentialStore current
storeCredential: (IcePlaintextCredentials new
username: 'esteban';
password: 'secret';
yourself)
forHostname: ‘github.com’"

Then you full script will be like this: 

./pharo build.image eval --save “
    Iceberg remoteTypeSelector: #httpsUrl”.
    IceCredentialStore current
storeCredential: (IcePlaintextCredentials new
username: 'esteban';
password: 'secret';
yourself)
forHostname: 'github.com'
./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp

I hope this solves your problem :)

Esteban

On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote:
I am still at a loss with this issue.

When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it.

So obviously, Pharo 7 is capable of accessing it.

However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ?

This code is so complex...

> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
>
> Hi,
>
> I asked this before, but I still have trouble with this.
>
> What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config.
>
> Basically,
>
> $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_
>
> towards a private server.
>
> In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler):
>
> $ uname -a
> Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> $ curl get.pharo.org/64/70+vm | bash
>  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                 Dload  Upload   Total   Spent    Left  Speed
> 100  3054  100  3054    0     0  57622      0 --:--:-- --:--:-- --:--:-- 57622
> Downloading the latest 70 Image:
>    http://files.pharo.org/get-files/70/pharo64.zip
> Pharo.image
> Downloading the latest pharoVM:
>       http://files.pharo.org/get-files/70/pharo64-linux-stable.zip
> pharo-vm/pharo
> Creating starter scripts pharo and pharo-ui
>
> $ ./pharo Pharo.image printVersion
> [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)'
>
> $ ./pharo Pharo.image save build
>
> $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>
> 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
> Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> Loading baseline of BaselineOfZTimestamp...
> Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache
> ...finished baseline
>
> $ ./pharo build.image eval ZTimestamp now
> 2019-10-25T12:28:34Z
>
> So although it gave a warning, it worked fine.
>
> The warning said,
>
> using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
>
> so let's try that after cleaning everything:
>
> $ rm -rf pharo-local/ build.*
>
> $ ./pharo Pharo.image save build
>
> $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp
>
> ...RETRY->BaselineOfZTimestamp
> ...RETRY->BaselineOfZTimestamp
> ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git
> MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer:
>
> WTF ? Why does this not work (I can perfectly git clone that URL) ?
>
> Of course, this would just be the first step, eventually I want to use my own private server with a URL like
>
> https://deploy:secret@.../scm/xyz/xyz-pharo.git
>
> How do you build/deploy non-public production code using the command line ?
>
> Help would be much appreciated.
>
> Sven
>


Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

Sven Van Caekenberghe-2
Hi Esteban,

Thanks for the reply, I will test all this as soon as I can and report back.

However, in your final script, are you sure the host (or port) can be specified (so no defaulting to GitHub.com).

Also, I was reading the implementation (hard), and it seemed as if the only real solution would be to make a new type (scheme), for this case, like

  githttps://user1:[hidden email]/scm/bla/foo-pharo.git

which would then preserve the username/password/host/port and do the proper thing directly. IIUC, https git URLs are described as the most common ones.

It might be that there is then still a problem with branches/tags, but maybe they are not needed.

Sven

> On 29 Oct 2019, at 11:07, Esteban Lorenzano <[hidden email]> wrote:
>
> Hi Sven,
>
> You are facing two different problems at the same time :)
> 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle).
> If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier.
>
> If you execute first:
>
> ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl”
>
> Then your execution:
>
> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>
> Will work without warnings.
>
> Now, you have another problem:
> 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git
> And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time.
> The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol).
> But if you still want/need to use HTTPS, then you need to provide your plaintext credentials.
> For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them).
> To add them programatically you need to do something like this:
>
> ./pharo build.image eval --save "IceCredentialStore current
> storeCredential: (IcePlaintextCredentials new
> username: 'esteban';
> password: 'secret';
> yourself)
> forHostname: ‘github.com’"
>
> Then you full script will be like this:
>
> ./pharo build.image eval --save “
>     Iceberg remoteTypeSelector: #httpsUrl”.
>     IceCredentialStore current
> storeCredential: (IcePlaintextCredentials new
> username: 'esteban';
> password: 'secret';
> yourself)
> forHostname: 'github.com'
> “
> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>
> I hope this solves your problem :)
>
> Esteban
>
> On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote:
> I am still at a loss with this issue.
>
> When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it.
>
> So obviously, Pharo 7 is capable of accessing it.
>
> However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ?
>
> This code is so complex...
>
> > On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
> >
> > Hi,
> >
> > I asked this before, but I still have trouble with this.
> >
> > What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config.
> >
> > Basically,
> >
> > $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_
> >
> > towards a private server.
> >
> > In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler):
> >
> > $ uname -a
> > Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ curl get.pharo.org/64/70+vm | bash
> >  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
> >                                 Dload  Upload   Total   Spent    Left  Speed
> > 100  3054  100  3054    0     0  57622      0 --:--:-- --:--:-- --:--:-- 57622
> > Downloading the latest 70 Image:
> >    http://files.pharo.org/get-files/70/pharo64.zip
> > Pharo.image
> > Downloading the latest pharoVM:
> >       http://files.pharo.org/get-files/70/pharo64-linux-stable.zip
> > pharo-vm/pharo
> > Creating starter scripts pharo and pharo-ui
> >
> > $ ./pharo Pharo.image printVersion
> > [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)'
> >
> > $ ./pharo Pharo.image save build
> >
> > $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
> >
> > 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
> > Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> > Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> > Loading baseline of BaselineOfZTimestamp...
> > Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> > Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache
> > ...finished baseline
> >
> > $ ./pharo build.image eval ZTimestamp now
> > 2019-10-25T12:28:34Z
> >
> > So although it gave a warning, it worked fine.
> >
> > The warning said,
> >
> > using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
> >
> > so let's try that after cleaning everything:
> >
> > $ rm -rf pharo-local/ build.*
> >
> > $ ./pharo Pharo.image save build
> >
> > $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp
> >
> > ...RETRY->BaselineOfZTimestamp
> > ...RETRY->BaselineOfZTimestamp
> > ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git
> > MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer:
> >
> > WTF ? Why does this not work (I can perfectly git clone that URL) ?
> >
> > Of course, this would just be the first step, eventually I want to use my own private server with a URL like
> >
> > https://deploy:secret@.../scm/xyz/xyz-pharo.git
> >
> > How do you build/deploy non-public production code using the command line ?
> >
> > Help would be much appreciated.
> >
> > Sven
> >
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

EstebanLM


> On 29 Oct 2019, at 11:24, Sven Van Caekenberghe <[hidden email]> wrote:
>
> Hi Esteban,
>
> Thanks for the reply, I will test all this as soon as I can and report back.
>
> However, in your final script, are you sure the host (or port) can be specified (so no defaulting to GitHub.com).

Indeed it Is not needed, but I recommend to do it :)

> Also, I was reading the implementation (hard), and it seemed as if the only real solution would be to make a new type (scheme), for this case, like
>
>  githttps://user1:[hidden email]/scm/bla/foo-pharo.git

And that’s why we use another solution more generic. Otherwise we will need to add GitHubHTTPS, GitLabHTTPS, etc., etc.

>
> which would then preserve the username/password/host/port and do the proper thing directly. IIUC, https git URLs are described as the most common ones.
>
> It might be that there is then still a problem with branches/tags, but maybe they are not needed.
>
> Sven
>
>> On 29 Oct 2019, at 11:07, Esteban Lorenzano <[hidden email]> wrote:
>>
>> Hi Sven,
>>
>> You are facing two different problems at the same time :)
>> 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle).
>> If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier.
>>
>> If you execute first:
>>
>> ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl”
>>
>> Then your execution:
>>
>> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>>
>> Will work without warnings.
>>
>> Now, you have another problem:
>> 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git
>> And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time.
>> The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol).
>> But if you still want/need to use HTTPS, then you need to provide your plaintext credentials.
>> For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them).
>> To add them programatically you need to do something like this:
>>
>> ./pharo build.image eval --save "IceCredentialStore current
>> storeCredential: (IcePlaintextCredentials new
>> username: 'esteban';
>> password: 'secret';
>> yourself)
>> forHostname: ‘github.com’"
>>
>> Then you full script will be like this:
>>
>> ./pharo build.image eval --save “
>>    Iceberg remoteTypeSelector: #httpsUrl”.
>>    IceCredentialStore current
>> storeCredential: (IcePlaintextCredentials new
>> username: 'esteban';
>> password: 'secret';
>> yourself)
>> forHostname: 'github.com'
>> “
>> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>>
>> I hope this solves your problem :)
>>
>> Esteban
>>
>> On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote:
>> I am still at a loss with this issue.
>>
>> When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it.
>>
>> So obviously, Pharo 7 is capable of accessing it.
>>
>> However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ?
>>
>> This code is so complex...
>>
>>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I asked this before, but I still have trouble with this.
>>>
>>> What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config.
>>>
>>> Basically,
>>>
>>> $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_
>>>
>>> towards a private server.
>>>
>>> In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler):
>>>
>>> $ uname -a
>>> Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> $ curl get.pharo.org/64/70+vm | bash
>>> % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>>                                Dload  Upload   Total   Spent    Left  Speed
>>> 100  3054  100  3054    0     0  57622      0 --:--:-- --:--:-- --:--:-- 57622
>>> Downloading the latest 70 Image:
>>>   http://files.pharo.org/get-files/70/pharo64.zip
>>> Pharo.image
>>> Downloading the latest pharoVM:
>>>      http://files.pharo.org/get-files/70/pharo64-linux-stable.zip
>>> pharo-vm/pharo
>>> Creating starter scripts pharo and pharo-ui
>>>
>>> $ ./pharo Pharo.image printVersion
>>> [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)'
>>>
>>> $ ./pharo Pharo.image save build
>>>
>>> $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>>>
>>> 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
>>> Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
>>> Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
>>> Loading baseline of BaselineOfZTimestamp...
>>> Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
>>> Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache
>>> ...finished baseline
>>>
>>> $ ./pharo build.image eval ZTimestamp now
>>> 2019-10-25T12:28:34Z
>>>
>>> So although it gave a warning, it worked fine.
>>>
>>> The warning said,
>>>
>>> using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
>>>
>>> so let's try that after cleaning everything:
>>>
>>> $ rm -rf pharo-local/ build.*
>>>
>>> $ ./pharo Pharo.image save build
>>>
>>> $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp
>>>
>>> ...RETRY->BaselineOfZTimestamp
>>> ...RETRY->BaselineOfZTimestamp
>>> ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git
>>> MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer:
>>>
>>> WTF ? Why does this not work (I can perfectly git clone that URL) ?
>>>
>>> Of course, this would just be the first step, eventually I want to use my own private server with a URL like
>>>
>>> https://deploy:secret@.../scm/xyz/xyz-pharo.git
>>>
>>> How do you build/deploy non-public production code using the command line ?
>>>
>>> Help would be much appreciated.
>>>
>>> Sven
>>>
>>
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

Christopher Fuhrman-4

On Tue, 29 Oct 2019 at 06:34, Esteban Lorenzano <[hidden email]> wrote:
> On 29 Oct 2019, at 11:24, Sven Van Caekenberghe <[hidden email]> wrote:
> Also, I was reading the implementation (hard), and it seemed as if the only real solution would be to make a new type (scheme), for this case, like
>
>  githttps://user1:passw1@.../scm/bla/foo-pharo.git

And that’s why we use another solution more generic. Otherwise we will need to add GitHubHTTPS, GitLabHTTPS, etc., etc.

I'm someone using 2FA on many systems (I use Authy for a key generator on my android phone) and I've been using it on GitHub now for years. I am forced to use the SSH approach since there is no viable Iceberg solution with HTTPS (there is actually a work-around with a GitHub application-defined key, but that is just a special password that isn't really 2FA IMO).  If there's a redesign of the Iceberg API, it would be good to consider the 2FA dimension, too.

Cris
 
>
> which would then preserve the username/password/host/port and do the proper thing directly. IIUC, https git URLs are described as the most common ones.
>
> It might be that there is then still a problem with branches/tags, but maybe they are not needed.
>
> Sven
>
>> On 29 Oct 2019, at 11:07, Esteban Lorenzano <[hidden email]> wrote:
>>
>> Hi Sven,
>>
>> You are facing two different problems at the same time :)
>> 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle).
>> If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier.
>>
>> If you execute first:
>>
>> ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl”
>>
>> Then your execution:
>>
>> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>>
>> Will work without warnings.
>>
>> Now, you have another problem:
>> 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git
>> And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time.
>> The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol).
>> But if you still want/need to use HTTPS, then you need to provide your plaintext credentials.
>> For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them).
>> To add them programatically you need to do something like this:
>>
>> ./pharo build.image eval --save "IceCredentialStore current
>>      storeCredential: (IcePlaintextCredentials new
>>              username: 'esteban';
>>              password: 'secret';
>>              yourself)
>>      forHostname: ‘github.com’"
>>
>> Then you full script will be like this:
>>
>> ./pharo build.image eval --save “
>>    Iceberg remoteTypeSelector: #httpsUrl”.
>>    IceCredentialStore current
>>      storeCredential: (IcePlaintextCredentials new
>>              username: 'esteban';
>>              password: 'secret';
>>              yourself)
>>      forHostname: 'github.com'
>> “
>> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>>
>> I hope this solves your problem :)
>>
>> Esteban
>>
>> On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote:
>> I am still at a loss with this issue.
>>
>> When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it.
>>
>> So obviously, Pharo 7 is capable of accessing it.
>>
>> However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ?
>>
>> This code is so complex...
>>
>>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I asked this before, but I still have trouble with this.
>>>
>>> What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config.
>>>
>>> Basically,
>>>
>>> $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_
>>>
>>> towards a private server.
>>>
>>> In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler):
>>>
>>> $ uname -a
>>> Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> $ curl get.pharo.org/64/70+vm | bash
>>> % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>>                                Dload  Upload   Total   Spent    Left  Speed
>>> 100  3054  100  3054    0     0  57622      0 --:--:-- --:--:-- --:--:-- 57622
>>> Downloading the latest 70 Image:
>>>   http://files.pharo.org/get-files/70/pharo64.zip
>>> Pharo.image
>>> Downloading the latest pharoVM:
>>>      http://files.pharo.org/get-files/70/pharo64-linux-stable.zip
>>> pharo-vm/pharo
>>> Creating starter scripts pharo and pharo-ui
>>>
>>> $ ./pharo Pharo.image printVersion
>>> [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)'
>>>
>>> $ ./pharo Pharo.image save build
>>>
>>> $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>>>
>>> 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
>>> Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
>>> Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
>>> Loading baseline of BaselineOfZTimestamp...
>>> Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
>>> Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache
>>> ...finished baseline
>>>
>>> $ ./pharo build.image eval ZTimestamp now
>>> 2019-10-25T12:28:34Z
>>>
>>> So although it gave a warning, it worked fine.
>>>
>>> The warning said,
>>>
>>> using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
>>>
>>> so let's try that after cleaning everything:
>>>
>>> $ rm -rf pharo-local/ build.*
>>>
>>> $ ./pharo Pharo.image save build
>>>
>>> $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp
>>>
>>> ...RETRY->BaselineOfZTimestamp
>>> ...RETRY->BaselineOfZTimestamp
>>> ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git
>>> MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer:
>>>
>>> WTF ? Why does this not work (I can perfectly git clone that URL) ?
>>>
>>> Of course, this would just be the first step, eventually I want to use my own private server with a URL like
>>>
>>> https://deploy:secret@.../scm/xyz/xyz-pharo.git
>>>
>>> How do you build/deploy non-public production code using the command line ?
>>>
>>> Help would be much appreciated.
>>>
>>> Sven
>>>
>>
>>
>
>




--
Christopher Fuhrman, P.Eng., PhD
Professeur au Département de génie logiciel et des technologies de l'information
ÉTS (École de technologie supérieure)
Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

Sven Van Caekenberghe-2
In reply to this post by EstebanLM
Hi Esteban,

I tried your snippets, see comments below.

> On 29 Oct 2019, at 11:07, Esteban Lorenzano <[hidden email]> wrote:
>
> Hi Sven,
>
> You are facing two different problems at the same time :)
> 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle).
> If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier.
>
> If you execute first:
>
> ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl”
>
> Then your execution:
>
> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>
> Will work without warnings.

Yes, this now removes the warning (not that that was a real problem).
I do not think I understand the implications of changing the remoteTypeSelector though.

> Now, you have another problem:
> 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git
> And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time.
> The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol).
> But if you still want/need to use HTTPS, then you need to provide your plaintext credentials.
> For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them).
> To add them programatically you need to do something like this:
>
> ./pharo build.image eval --save "IceCredentialStore current
> storeCredential: (IcePlaintextCredentials new
> username: 'esteban';
> password: 'secret';
> yourself)
> forHostname: ‘github.com’"
>
> Then you full script will be like this:
>
> ./pharo build.image eval --save “
>     Iceberg remoteTypeSelector: #httpsUrl”.
>     IceCredentialStore current
> storeCredential: (IcePlaintextCredentials new
> username: 'esteban';
> password: 'secret';
> yourself)
> forHostname: 'github.com'
> “
> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>
> I hope this solves your problem :)

This worked as well, but since ZTimestamp is a public GitHub.com project, I am not sure the authentication is really used. I tried with my GitHub.com credentials (username/password) and it loaded, but when I altered the password for a wrong one, it still worked ;-)

Now, the problem is: how do I access the private, password protected repository of my company ?

I tried to add a host to the github:// URL but I do not have the impression that this works.

Since making this more concrete involves some sensitive information, I will send you a private email with the details of a test project/repository/user that I made.

Sven

PS: I want to stress that for me Iceberg works really well on my local development machine: I created the test project/code/baseline without encountering any issue (Pharo 7).

> Esteban
>
> On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote:
> I am still at a loss with this issue.
>
> When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it.
>
> So obviously, Pharo 7 is capable of accessing it.
>
> However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ?
>
> This code is so complex...
>
> > On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
> >
> > Hi,
> >
> > I asked this before, but I still have trouble with this.
> >
> > What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config.
> >
> > Basically,
> >
> > $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_
> >
> > towards a private server.
> >
> > In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler):
> >
> > $ uname -a
> > Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ curl get.pharo.org/64/70+vm | bash
> >  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
> >                                 Dload  Upload   Total   Spent    Left  Speed
> > 100  3054  100  3054    0     0  57622      0 --:--:-- --:--:-- --:--:-- 57622
> > Downloading the latest 70 Image:
> >    http://files.pharo.org/get-files/70/pharo64.zip
> > Pharo.image
> > Downloading the latest pharoVM:
> >       http://files.pharo.org/get-files/70/pharo64-linux-stable.zip
> > pharo-vm/pharo
> > Creating starter scripts pharo and pharo-ui
> >
> > $ ./pharo Pharo.image printVersion
> > [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)'
> >
> > $ ./pharo Pharo.image save build
> >
> > $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
> >
> > 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
> > Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> > Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> > Loading baseline of BaselineOfZTimestamp...
> > Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
> > Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache
> > ...finished baseline
> >
> > $ ./pharo build.image eval ZTimestamp now
> > 2019-10-25T12:28:34Z
> >
> > So although it gave a warning, it worked fine.
> >
> > The warning said,
> >
> > using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
> >
> > so let's try that after cleaning everything:
> >
> > $ rm -rf pharo-local/ build.*
> >
> > $ ./pharo Pharo.image save build
> >
> > $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp
> >
> > ...RETRY->BaselineOfZTimestamp
> > ...RETRY->BaselineOfZTimestamp
> > ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git
> > MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer:
> >
> > WTF ? Why does this not work (I can perfectly git clone that URL) ?
> >
> > Of course, this would just be the first step, eventually I want to use my own private server with a URL like
> >
> > https://deploy:secret@.../scm/xyz/xyz-pharo.git
> >
> > How do you build/deploy non-public production code using the command line ?
> >
> > Help would be much appreciated.
> >
> > Sven
> >
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

NorbertHartl


> Am 29.10.2019 um 17:16 schrieb Sven Van Caekenberghe <[hidden email]>:
>
> Hi Esteban,
>
> I tried your snippets, see comments below.
>
>> On 29 Oct 2019, at 11:07, Esteban Lorenzano <[hidden email]> wrote:
>>
>> Hi Sven,
>>
>> You are facing two different problems at the same time :)
>> 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle).
>> If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier.
>>
>> If you execute first:
>>
>> ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl”
>>
>> Then your execution:
>>
>> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>>
>> Will work without warnings.
>
> Yes, this now removes the warning (not that that was a real problem).
> I do not think I understand the implications of changing the remoteTypeSelector though.
>
>> Now, you have another problem:
>> 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git
>> And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time.
>> The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol).
>> But if you still want/need to use HTTPS, then you need to provide your plaintext credentials.
>> For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them).
>> To add them programatically you need to do something like this:
>>
>> ./pharo build.image eval --save "IceCredentialStore current
>>    storeCredential: (IcePlaintextCredentials new
>>        username: 'esteban';
>>        password: 'secret';
>>        yourself)
>>    forHostname: ‘github.com’"
>>
>> Then you full script will be like this:
>>
>> ./pharo build.image eval --save “
>>    Iceberg remoteTypeSelector: #httpsUrl”.
>>    IceCredentialStore current
>>    storeCredential: (IcePlaintextCredentials new
>>        username: 'esteban';
>>        password: 'secret';
>>        yourself)
>>    forHostname: 'github.com'
>> “
>> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>>
>> I hope this solves your problem :)
>
> This worked as well, but since ZTimestamp is a public GitHub.com project, I am not sure the authentication is really used. I tried with my GitHub.com credentials (username/password) and it loaded, but when I altered the password for a wrong one, it still worked ;-)
>
> Now, the problem is: how do I access the private, password protected repository of my company ?
>
> I tried to add a host to the github:// URL but I do not have the impression that this works.
>
> Since making this more concrete involves some sensitive information, I will send you a private email with the details of a test project/repository/user that I made.
>
> Sven
>
> PS: I want to stress that for me Iceberg works really well on my local development machine: I created the test project/code/baseline without encountering any issue (Pharo 7).
>
I like to second that. Working with iceberg is mostly ok. Where it is not it is often related more to git than to iceberg. But yes the deployment side has weaknesses. So my position on this that for production I want to reproducible load code which you cannot be sure when using iceberg with git and metacello (repos do not get updated automatically). This is the main reason I use https. Furthermore iceberg is no lightweight. So I want to have a process that I can use for small images as well where iceberg should be absent. Please read this as something to consider for future enhancement. Managing credentials is something that needs to be improved. In order to be able to load a dependent project from a private repo I recompile url methods to
inject credentials. So yes a better way is highly welcome ;)

Norbert

>> Esteban
>>
>> On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote:
>> I am still at a loss with this issue.
>>
>> When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it.
>>
>> So obviously, Pharo 7 is capable of accessing it.
>>
>> However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ?
>>
>> This code is so complex...
>>
>>>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I asked this before, but I still have trouble with this.
>>>
>>> What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config.
>>>
>>> Basically,
>>>
>>> $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_
>>>
>>> towards a private server.
>>>
>>> In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler):
>>>
>>> $ uname -a
>>> Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> $ curl get.pharo.org/64/70+vm | bash
>>> % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>>                                Dload  Upload   Total   Spent    Left  Speed
>>> 100  3054  100  3054    0     0  57622      0 --:--:-- --:--:-- --:--:-- 57622
>>> Downloading the latest 70 Image:
>>>   http://files.pharo.org/get-files/70/pharo64.zip
>>> Pharo.image
>>> Downloading the latest pharoVM:
>>>      http://files.pharo.org/get-files/70/pharo64-linux-stable.zip
>>> pharo-vm/pharo
>>> Creating starter scripts pharo and pharo-ui
>>>
>>> $ ./pharo Pharo.image printVersion
>>> [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)'
>>>
>>> $ ./pharo Pharo.image save build
>>>
>>> $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp
>>>
>>> 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
>>> Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
>>> Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
>>> Loading baseline of BaselineOfZTimestamp...
>>> Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master]
>>> Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache
>>> ...finished baseline
>>>
>>> $ ./pharo build.image eval ZTimestamp now
>>> 2019-10-25T12:28:34Z
>>>
>>> So although it gave a warning, it worked fine.
>>>
>>> The warning said,
>>>
>>> using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant.
>>>
>>> so let's try that after cleaning everything:
>>>
>>> $ rm -rf pharo-local/ build.*
>>>
>>> $ ./pharo Pharo.image save build
>>>
>>> $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp
>>>
>>> ...RETRY->BaselineOfZTimestamp
>>> ...RETRY->BaselineOfZTimestamp
>>> ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git
>>> MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer:
>>>
>>> WTF ? Why does this not work (I can perfectly git clone that URL) ?
>>>
>>> Of course, this would just be the first step, eventually I want to use my own private server with a URL like
>>>
>>> https://deploy:secret@.../scm/xyz/xyz-pharo.git
>>>
>>> How do you build/deploy non-public production code using the command line ?
>>>
>>> Help would be much appreciated.
>>>
>>> Sven
>>>
>>
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

Sven Van Caekenberghe-2
In reply to this post by Sven Van Caekenberghe-2


> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
>
> How do you build/deploy non-public production code using the command line ?

I just learned about the following technique:

Using SSH agent forwarding

https://developer.github.com/v3/guides/using-ssh-agent-forwarding/

This certainly makes using certificates on production servers much easier since you do no longer have to manage or install special ones on the deployment servers, you can just use your standard developer certificates.

Sven
Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

NorbertHartl


> Am 31.10.2019 um 12:12 schrieb Sven Van Caekenberghe <[hidden email]>:
>
>
>
>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>> How do you build/deploy non-public production code using the command line ?
>
> I just learned about the following technique:
>
> Using SSH agent forwarding
>
> https://developer.github.com/v3/guides/using-ssh-agent-forwarding/
>
> This certainly makes using certificates on production servers much easier since you do no longer have to manage or install special ones on the deployment servers, you can just use your standard developer certificates.
>
That is quite late :P I always use this as it also chains through all the machins you log in. We use jenkins for building our software there it does not work. We just have a deployment key for that. That is a key pair without passphrase that is registered in the git repository and installed on the jenkins server.

Norbert



Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

Sven Van Caekenberghe-2


> On 31 Oct 2019, at 12:18, Norbert Hartl <[hidden email]> wrote:
>
>
>
>> Am 31.10.2019 um 12:12 schrieb Sven Van Caekenberghe <[hidden email]>:
>>
>>
>>
>>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> How do you build/deploy non-public production code using the command line ?
>>
>> I just learned about the following technique:
>>
>> Using SSH agent forwarding
>>
>> https://developer.github.com/v3/guides/using-ssh-agent-forwarding/
>>
>> This certainly makes using certificates on production servers much easier since you do no longer have to manage or install special ones on the deployment servers, you can just use your standard developer certificates.
>>
> That is quite late :P I always use this as it also chains through all the machins you log in. We use jenkins for building our software there it does not work. We just have a deployment key for that. That is a key pair without passphrase that is registered in the git repository and installed on the jenkins server.
>
> Norbert

Indeed, the SSH forwarding cannot be used for independent CI builds.

I am curious though: how do you do the initial checkout (clone) from your private repository ? Using regular git command line tools or using Iceberg/Metacello ?

If the latter, how do you specify your URL, exactly ?

The first case works for me too, using a gitlocal:// URL, but that is cheating a bit, I feel like we should be able to do this directly in Pharo.

Sven



Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

NorbertHartl


> Am 01.11.2019 um 17:44 schrieb Sven Van Caekenberghe <[hidden email]>:
>
>
>
>> On 31 Oct 2019, at 12:18, Norbert Hartl <[hidden email]> wrote:
>>
>>
>>
>>> Am 31.10.2019 um 12:12 schrieb Sven Van Caekenberghe <[hidden email]>:
>>>
>>>
>>>
>>>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:
>>>>
>>>> How do you build/deploy non-public production code using the command line ?
>>>
>>> I just learned about the following technique:
>>>
>>> Using SSH agent forwarding
>>>
>>> https://developer.github.com/v3/guides/using-ssh-agent-forwarding/
>>>
>>> This certainly makes using certificates on production servers much easier since you do no longer have to manage or install special ones on the deployment servers, you can just use your standard developer certificates.
>>>
>> That is quite late :P I always use this as it also chains through all the machins you log in. We use jenkins for building our software there it does not work. We just have a deployment key for that. That is a key pair without passphrase that is registered in the git repository and installed on the jenkins server.
>>
>> Norbert
>
> Indeed, the SSH forwarding cannot be used for independent CI builds.
>
> I am curious though: how do you do the initial checkout (clone) from your private repository ? Using regular git command line tools or using Iceberg/Metacello ?
>
> If the latter, how do you specify your URL, exactly ?
>
> The first case works for me too, using a gitlocal:// URL, but that is cheating a bit, I feel like we should be able to do this directly in Pharo.

If checking out from private repo your problem is as deep as you reference private repos. My project (1) is in a private repository, it contains a baseline that references a project in a private repository (2) and this references a project in a private repo (3).

* Project (1) is checked out by jenkins. I have a deployment key pair without passphrase where the public key is registered in the bitbucket project and the private key is deployed on jenkins.

* I try to avoid iceberg as much as possible for deployment builds because later I want to have an image without iceberg. And it is not reliable to use iceberg because metacello won't update git repos automatically and the odds are high that you load old content. So I load the project with the eval commandline handler with

Metacello new
   repository: 'filetree://source';
   baseline …
   …

* In order to load the private projects in the the baseline of the main project I run a script before build that hacks the image in a way that

MCBitbucketRepository compile: 'httpsUrl
        ^ ''https://USER:PASS@.../'', projectPath, ''.git'''.

MCBitbucketRepository compile: 'projectZipUrlFor: projectPath versionString: versionString
  ^ ''https://USER:PASS@.../'' , projectPath , ''/get/'' , versionString , ''.zip'''.

MCBitbucketRepository class compile: 'projectZipUrlFor: projectPath versionString: versionString
  ^ ''https://USER:PASS@.../'' , projectPath , ''/get/'' , versionString , ''.zip'''.

This works only because all private projects are on bitbucket and no public one.

* Project (3) I made public because I did not see how to make that work

Hope this helps. I feel this to be far too complicated. Together with the confusion between pharo5/pharo6/pharo7 with iceberg, tonel and iceberg and/or tonel support killed two weeks of my time at least. So I would welcome if there would be a better way to manage credentials

Norbert
Reply | Threaded
Open this post in threaded view
|

Re: Git URL issues

EstebanLM
I made a PR to allow “generic” git repositories to be referenced: 


But… problem is more complicated than you may think: Metacello “vanilla” relies on download a zip file with the sources, it cannot have a “generic” way to act. 
This means the PR will work just if you use iceberg. And maybe you do not want to declare your baselines relying on iceberg.

In that case, you have just one workaround possible: 

1) clone locally the remove private sources outside the image.
2) load them using plain metacello and filetree://.
3) “lock” those repositories.

Then you can run your installation.

Esteban

On 4 Nov 2019, at 12:52, Norbert Hartl <[hidden email]> wrote:



Am 01.11.2019 um 17:44 schrieb Sven Van Caekenberghe <[hidden email]>:



On 31 Oct 2019, at 12:18, Norbert Hartl <[hidden email]> wrote:



Am 31.10.2019 um 12:12 schrieb Sven Van Caekenberghe <[hidden email]>:



On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote:

How do you build/deploy non-public production code using the command line ?

I just learned about the following technique:

Using SSH agent forwarding

https://developer.github.com/v3/guides/using-ssh-agent-forwarding/

This certainly makes using certificates on production servers much easier since you do no longer have to manage or install special ones on the deployment servers, you can just use your standard developer certificates.

That is quite late :P I always use this as it also chains through all the machins you log in. We use jenkins for building our software there it does not work. We just have a deployment key for that. That is a key pair without passphrase that is registered in the git repository and installed on the jenkins server.

Norbert

Indeed, the SSH forwarding cannot be used for independent CI builds.

I am curious though: how do you do the initial checkout (clone) from your private repository ? Using regular git command line tools or using Iceberg/Metacello ?

If the latter, how do you specify your URL, exactly ?

The first case works for me too, using a gitlocal:// URL, but that is cheating a bit, I feel like we should be able to do this directly in Pharo.

If checking out from private repo your problem is as deep as you reference private repos. My project (1) is in a private repository, it contains a baseline that references a project in a private repository (2) and this references a project in a private repo (3).

* Project (1) is checked out by jenkins. I have a deployment key pair without passphrase where the public key is registered in the bitbucket project and the private key is deployed on jenkins. 

* I try to avoid iceberg as much as possible for deployment builds because later I want to have an image without iceberg. And it is not reliable to use iceberg because metacello won't update git repos automatically and the odds are high that you load old content. So I load the project with the eval commandline handler with 

Metacello new
  repository: '<a href="filetree://source" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">filetree://source';
  baseline …
  …

* In order to load the private projects in the the baseline of the main project I run a script before build that hacks the image in a way that

MCBitbucketRepository compile: 'httpsUrl
^ ''https://USER:PASS@.../'', projectPath, ''.git'''.

MCBitbucketRepository compile: 'projectZipUrlFor: projectPath versionString: versionString
 ^ ''https://USER:PASS@.../'' , projectPath , ''/get/'' , versionString , ''.zip'''.

MCBitbucketRepository class compile: 'projectZipUrlFor: projectPath versionString: versionString
 ^ ''https://USER:PASS@.../'' , projectPath , ''/get/'' , versionString , ''.zip'''.

This works only because all private projects are on bitbucket and no public one.

* Project (3) I made public because I did not see how to make that work

Hope this helps. I feel this to be far too complicated. Together with the confusion between pharo5/pharo6/pharo7 with iceberg, tonel and iceberg and/or tonel support killed two weeks of my time at least. So I would welcome if there would be a better way to manage credentials

Norbert