Git URL issues
Locked
 
|
Hi,
I asked this before, but I still have trouble with this. What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config. Basically, $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_ towards a private server. In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler): $ uname -a Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux $ curl get.pharo.org/64/70+vm | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3054 100 3054 0 0 57622 0 --:--:-- --:--:-- --:--:-- 57622 Downloading the latest 70 Image: http://files.pharo.org/get-files/70/pharo64.zip Pharo.image Downloading the latest pharoVM: http://files.pharo.org/get-files/70/pharo64-linux-stable.zip pharo-vm/pharo Creating starter scripts pharo and pharo-ui $ ./pharo Pharo.image printVersion [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)' $ ./pharo Pharo.image save build $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] Loading baseline of BaselineOfZTimestamp... Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache ...finished baseline $ ./pharo build.image eval ZTimestamp now 2019-10-25T12:28:34Z So although it gave a warning, it worked fine. The warning said, using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. so let's try that after cleaning everything: $ rm -rf pharo-local/ build.* $ ./pharo Pharo.image save build $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp ...RETRY->BaselineOfZTimestamp ...RETRY->BaselineOfZTimestamp ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: WTF ? Why does this not work (I can perfectly git clone that URL) ? Of course, this would just be the first step, eventually I want to use my own private server with a URL like https://deploy:secret@.../scm/xyz/xyz-pharo.git How do you build/deploy non-public production code using the command line ? Help would be much appreciated. Sven |
|
|
I am still at a loss with this issue.
When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it. So obviously, Pharo 7 is capable of accessing it. However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ? This code is so complex... > On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote: > > Hi, > > I asked this before, but I still have trouble with this. > > What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config. > > Basically, > > $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_ > > towards a private server. > > In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler): > > $ uname -a > Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux > > $ curl get.pharo.org/64/70+vm | bash > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 3054 100 3054 0 0 57622 0 --:--:-- --:--:-- --:--:-- 57622 > Downloading the latest 70 Image: > http://files.pharo.org/get-files/70/pharo64.zip > Pharo.image > Downloading the latest pharoVM: > http://files.pharo.org/get-files/70/pharo64-linux-stable.zip > pharo-vm/pharo > Creating starter scripts pharo and pharo-ui > > $ ./pharo Pharo.image printVersion > [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)' > > $ ./pharo Pharo.image save build > > $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp > > 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. > Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] > Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] > Loading baseline of BaselineOfZTimestamp... > Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] > Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache > ...finished baseline > > $ ./pharo build.image eval ZTimestamp now > 2019-10-25T12:28:34Z > > So although it gave a warning, it worked fine. > > The warning said, > > using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. > > so let's try that after cleaning everything: > > $ rm -rf pharo-local/ build.* > > $ ./pharo Pharo.image save build > > $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp > > ...RETRY->BaselineOfZTimestamp > ...RETRY->BaselineOfZTimestamp > ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git > MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: > > WTF ? Why does this not work (I can perfectly git clone that URL) ? > > Of course, this would just be the first step, eventually I want to use my own private server with a URL like > > https://deploy:secret@.../scm/xyz/xyz-pharo.git > > How do you build/deploy non-public production code using the command line ? > > Help would be much appreciated. > > Sven > |
|
|
Hi Sven, You are facing two different problems at the same time :) 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle). If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier. If you execute first: ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl” Then your execution: ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp Will work without warnings. Now, you have another problem: 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time. The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol). But if you still want/need to use HTTPS, then you need to provide your plaintext credentials. For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them). To add them programatically you need to do something like this: ./pharo build.image eval --save "IceCredentialStore current storeCredential: (IcePlaintextCredentials new username: 'esteban'; password: 'secret'; yourself) forHostname: ‘github.com’" Then you full script will be like this: ./pharo build.image eval --save “ Iceberg remoteTypeSelector: #httpsUrl”. IceCredentialStore current storeCredential: (IcePlaintextCredentials new username: 'esteban'; password: 'secret'; yourself) forHostname: 'github.com' “ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp I hope this solves your problem :) Esteban On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote: I am still at a loss with this issue. |
|
|
Hi Esteban,
Thanks for the reply, I will test all this as soon as I can and report back. However, in your final script, are you sure the host (or port) can be specified (so no defaulting to GitHub.com). Also, I was reading the implementation (hard), and it seemed as if the only real solution would be to make a new type (scheme), for this case, like githttps://user1:[hidden email]/scm/bla/foo-pharo.git which would then preserve the username/password/host/port and do the proper thing directly. IIUC, https git URLs are described as the most common ones. It might be that there is then still a problem with branches/tags, but maybe they are not needed. Sven > On 29 Oct 2019, at 11:07, Esteban Lorenzano <[hidden email]> wrote: > > Hi Sven, > > You are facing two different problems at the same time :) > 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle). > If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier. > > If you execute first: > > ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl” > > Then your execution: > > ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp > > Will work without warnings. > > Now, you have another problem: > 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git > And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time. > The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol). > But if you still want/need to use HTTPS, then you need to provide your plaintext credentials. > For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them). > To add them programatically you need to do something like this: > > ./pharo build.image eval --save "IceCredentialStore current > storeCredential: (IcePlaintextCredentials new > username: 'esteban'; > password: 'secret'; > yourself) > forHostname: ‘github.com’" > > Then you full script will be like this: > > ./pharo build.image eval --save “ > Iceberg remoteTypeSelector: #httpsUrl”. > IceCredentialStore current > storeCredential: (IcePlaintextCredentials new > username: 'esteban'; > password: 'secret'; > yourself) > forHostname: 'github.com' > “ > ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp > > I hope this solves your problem :) > > Esteban > > On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote: > I am still at a loss with this issue. > > When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it. > > So obviously, Pharo 7 is capable of accessing it. > > However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ? > > This code is so complex... > > > On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote: > > > > Hi, > > > > I asked this before, but I still have trouble with this. > > > > What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config. > > > > Basically, > > > > $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_ > > > > towards a private server. > > > > In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler): > > > > $ uname -a > > Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux > > > > $ curl get.pharo.org/64/70+vm | bash > > % Total % Received % Xferd Average Speed Time Time Time Current > > Dload Upload Total Spent Left Speed > > 100 3054 100 3054 0 0 57622 0 --:--:-- --:--:-- --:--:-- 57622 > > Downloading the latest 70 Image: > > http://files.pharo.org/get-files/70/pharo64.zip > > Pharo.image > > Downloading the latest pharoVM: > > http://files.pharo.org/get-files/70/pharo64-linux-stable.zip > > pharo-vm/pharo > > Creating starter scripts pharo and pharo-ui > > > > $ ./pharo Pharo.image printVersion > > [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)' > > > > $ ./pharo Pharo.image save build > > > > $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp > > > > 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. > > Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] > > Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] > > Loading baseline of BaselineOfZTimestamp... > > Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] > > Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache > > ...finished baseline > > > > $ ./pharo build.image eval ZTimestamp now > > 2019-10-25T12:28:34Z > > > > So although it gave a warning, it worked fine. > > > > The warning said, > > > > using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. > > > > so let's try that after cleaning everything: > > > > $ rm -rf pharo-local/ build.* > > > > $ ./pharo Pharo.image save build > > > > $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp > > > > ...RETRY->BaselineOfZTimestamp > > ...RETRY->BaselineOfZTimestamp > > ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git > > MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: > > > > WTF ? Why does this not work (I can perfectly git clone that URL) ? > > > > Of course, this would just be the first step, eventually I want to use my own private server with a URL like > > > > https://deploy:secret@.../scm/xyz/xyz-pharo.git > > > > How do you build/deploy non-public production code using the command line ? > > > > Help would be much appreciated. > > > > Sven > > > > |
|
|
> On 29 Oct 2019, at 11:24, Sven Van Caekenberghe <[hidden email]> wrote: > > Hi Esteban, > > Thanks for the reply, I will test all this as soon as I can and report back. > > However, in your final script, are you sure the host (or port) can be specified (so no defaulting to GitHub.com). Indeed it Is not needed, but I recommend to do it :) > Also, I was reading the implementation (hard), and it seemed as if the only real solution would be to make a new type (scheme), for this case, like > > githttps://user1:[hidden email]/scm/bla/foo-pharo.git And that’s why we use another solution more generic. Otherwise we will need to add GitHubHTTPS, GitLabHTTPS, etc., etc. > > which would then preserve the username/password/host/port and do the proper thing directly. IIUC, https git URLs are described as the most common ones. > > It might be that there is then still a problem with branches/tags, but maybe they are not needed. > > Sven > >> On 29 Oct 2019, at 11:07, Esteban Lorenzano <[hidden email]> wrote: >> >> Hi Sven, >> >> You are facing two different problems at the same time :) >> 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle). >> If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier. >> >> If you execute first: >> >> ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl” >> >> Then your execution: >> >> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp >> >> Will work without warnings. >> >> Now, you have another problem: >> 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git >> And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time. >> The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol). >> But if you still want/need to use HTTPS, then you need to provide your plaintext credentials. >> For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them). >> To add them programatically you need to do something like this: >> >> ./pharo build.image eval --save "IceCredentialStore current >> storeCredential: (IcePlaintextCredentials new >> username: 'esteban'; >> password: 'secret'; >> yourself) >> forHostname: ‘github.com’" >> >> Then you full script will be like this: >> >> ./pharo build.image eval --save “ >> Iceberg remoteTypeSelector: #httpsUrl”. >> IceCredentialStore current >> storeCredential: (IcePlaintextCredentials new >> username: 'esteban'; >> password: 'secret'; >> yourself) >> forHostname: 'github.com' >> “ >> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp >> >> I hope this solves your problem :) >> >> Esteban >> >> On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote: >> I am still at a loss with this issue. >> >> When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it. >> >> So obviously, Pharo 7 is capable of accessing it. >> >> However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ? >> >> This code is so complex... >> >>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote: >>> >>> Hi, >>> >>> I asked this before, but I still have trouble with this. >>> >>> What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config. >>> >>> Basically, >>> >>> $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_ >>> >>> towards a private server. >>> >>> In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler): >>> >>> $ uname -a >>> Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>> >>> $ curl get.pharo.org/64/70+vm | bash >>> % Total % Received % Xferd Average Speed Time Time Time Current >>> Dload Upload Total Spent Left Speed >>> 100 3054 100 3054 0 0 57622 0 --:--:-- --:--:-- --:--:-- 57622 >>> Downloading the latest 70 Image: >>> http://files.pharo.org/get-files/70/pharo64.zip >>> Pharo.image >>> Downloading the latest pharoVM: >>> http://files.pharo.org/get-files/70/pharo64-linux-stable.zip >>> pharo-vm/pharo >>> Creating starter scripts pharo and pharo-ui >>> >>> $ ./pharo Pharo.image printVersion >>> [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)' >>> >>> $ ./pharo Pharo.image save build >>> >>> $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp >>> >>> 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. >>> Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] >>> Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] >>> Loading baseline of BaselineOfZTimestamp... >>> Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] >>> Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache >>> ...finished baseline >>> >>> $ ./pharo build.image eval ZTimestamp now >>> 2019-10-25T12:28:34Z >>> >>> So although it gave a warning, it worked fine. >>> >>> The warning said, >>> >>> using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. >>> >>> so let's try that after cleaning everything: >>> >>> $ rm -rf pharo-local/ build.* >>> >>> $ ./pharo Pharo.image save build >>> >>> $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp >>> >>> ...RETRY->BaselineOfZTimestamp >>> ...RETRY->BaselineOfZTimestamp >>> ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git >>> MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: >>> >>> WTF ? Why does this not work (I can perfectly git clone that URL) ? >>> >>> Of course, this would just be the first step, eventually I want to use my own private server with a URL like >>> >>> https://deploy:secret@.../scm/xyz/xyz-pharo.git >>> >>> How do you build/deploy non-public production code using the command line ? >>> >>> Help would be much appreciated. >>> >>> Sven >>> >> >> > > |
|
|
On Tue, 29 Oct 2019 at 06:34, Esteban Lorenzano <[hidden email]> wrote: > On 29 Oct 2019, at 11:24, Sven Van Caekenberghe <[hidden email]> wrote: I'm someone using 2FA on many systems (I use Authy for a key generator on my android phone) and I've been using it on GitHub now for years. I am forced to use the SSH approach since there is no viable Iceberg solution with HTTPS (there is actually a work-around with a GitHub application-defined key, but that is just a special password that isn't really 2FA IMO). If there's a redesign of the Iceberg API, it would be good to consider the 2FA dimension, too. Cris > Christopher Fuhrman, P.Eng., PhD Professeur au Département de génie logiciel et des technologies de l'informationÉTS (École de technologie supérieure) |
|
|
In reply to this post by EstebanLM
Hi Esteban,
I tried your snippets, see comments below. > On 29 Oct 2019, at 11:07, Esteban Lorenzano <[hidden email]> wrote: > > Hi Sven, > > You are facing two different problems at the same time :) > 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle). > If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier. > > If you execute first: > > ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl” > > Then your execution: > > ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp > > Will work without warnings. Yes, this now removes the warning (not that that was a real problem). I do not think I understand the implications of changing the remoteTypeSelector though. > Now, you have another problem: > 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git > And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time. > The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol). > But if you still want/need to use HTTPS, then you need to provide your plaintext credentials. > For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them). > To add them programatically you need to do something like this: > > ./pharo build.image eval --save "IceCredentialStore current > storeCredential: (IcePlaintextCredentials new > username: 'esteban'; > password: 'secret'; > yourself) > forHostname: ‘github.com’" > > Then you full script will be like this: > > ./pharo build.image eval --save “ > Iceberg remoteTypeSelector: #httpsUrl”. > IceCredentialStore current > storeCredential: (IcePlaintextCredentials new > username: 'esteban'; > password: 'secret'; > yourself) > forHostname: 'github.com' > “ > ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp > > I hope this solves your problem :) This worked as well, but since ZTimestamp is a public GitHub.com project, I am not sure the authentication is really used. I tried with my GitHub.com credentials (username/password) and it loaded, but when I altered the password for a wrong one, it still worked ;-) Now, the problem is: how do I access the private, password protected repository of my company ? I tried to add a host to the github:// URL but I do not have the impression that this works. Since making this more concrete involves some sensitive information, I will send you a private email with the details of a test project/repository/user that I made. Sven PS: I want to stress that for me Iceberg works really well on my local development machine: I created the test project/code/baseline without encountering any issue (Pharo 7). > Esteban > > On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote: > I am still at a loss with this issue. > > When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it. > > So obviously, Pharo 7 is capable of accessing it. > > However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ? > > This code is so complex... > > > On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote: > > > > Hi, > > > > I asked this before, but I still have trouble with this. > > > > What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config. > > > > Basically, > > > > $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_ > > > > towards a private server. > > > > In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler): > > > > $ uname -a > > Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux > > > > $ curl get.pharo.org/64/70+vm | bash > > % Total % Received % Xferd Average Speed Time Time Time Current > > Dload Upload Total Spent Left Speed > > 100 3054 100 3054 0 0 57622 0 --:--:-- --:--:-- --:--:-- 57622 > > Downloading the latest 70 Image: > > http://files.pharo.org/get-files/70/pharo64.zip > > Pharo.image > > Downloading the latest pharoVM: > > http://files.pharo.org/get-files/70/pharo64-linux-stable.zip > > pharo-vm/pharo > > Creating starter scripts pharo and pharo-ui > > > > $ ./pharo Pharo.image printVersion > > [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)' > > > > $ ./pharo Pharo.image save build > > > > $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp > > > > 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. > > Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] > > Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] > > Loading baseline of BaselineOfZTimestamp... > > Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] > > Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache > > ...finished baseline > > > > $ ./pharo build.image eval ZTimestamp now > > 2019-10-25T12:28:34Z > > > > So although it gave a warning, it worked fine. > > > > The warning said, > > > > using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. > > > > so let's try that after cleaning everything: > > > > $ rm -rf pharo-local/ build.* > > > > $ ./pharo Pharo.image save build > > > > $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp > > > > ...RETRY->BaselineOfZTimestamp > > ...RETRY->BaselineOfZTimestamp > > ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git > > MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: > > > > WTF ? Why does this not work (I can perfectly git clone that URL) ? > > > > Of course, this would just be the first step, eventually I want to use my own private server with a URL like > > > > https://deploy:secret@.../scm/xyz/xyz-pharo.git > > > > How do you build/deploy non-public production code using the command line ? > > > > Help would be much appreciated. > > > > Sven > > > > |
|
|
> Am 29.10.2019 um 17:16 schrieb Sven Van Caekenberghe <[hidden email]>: > > Hi Esteban, > > I tried your snippets, see comments below. > >> On 29 Oct 2019, at 11:07, Esteban Lorenzano <[hidden email]> wrote: >> >> Hi Sven, >> >> You are facing two different problems at the same time :) >> 1) a problem with Metacello api. There is a mismatch between git and metacello because the apis are not equivalent (specially having iceberg in the middle). >> If you want to use HTTPS in metacello you have to explicit you want to use HTTPS, but you still has to use the GitHub:// identifier. >> >> If you execute first: >> >> ./pharo build.image eval --save "Iceberg remoteTypeSelector: #httpsUrl” >> >> Then your execution: >> >> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp >> >> Will work without warnings. > > Yes, this now removes the warning (not that that was a real problem). > I do not think I understand the implications of changing the remoteTypeSelector though. > >> Now, you have another problem: >> 2) You cannot use an url as this with iceberg/metacello: https://deploy:secret@.../scm/xyz/xyz-pharo.git >> And yes, we could do a parse of it and use it (also solving your metacello command problem), but this is not easily doable and it would take time. >> The usual way of doing this is not using HTTPS but distributing public/private keys (then you use the regular SSH protocol). >> But if you still want/need to use HTTPS, then you need to provide your plaintext credentials. >> For iceberg, your credentials are stored in your preferences directory, at credentials.fuel (so you could add them to your CI server, if you have them). >> To add them programatically you need to do something like this: >> >> ./pharo build.image eval --save "IceCredentialStore current >> storeCredential: (IcePlaintextCredentials new >> username: 'esteban'; >> password: 'secret'; >> yourself) >> forHostname: ‘github.com’" >> >> Then you full script will be like this: >> >> ./pharo build.image eval --save “ >> Iceberg remoteTypeSelector: #httpsUrl”. >> IceCredentialStore current >> storeCredential: (IcePlaintextCredentials new >> username: 'esteban'; >> password: 'secret'; >> yourself) >> forHostname: 'github.com' >> “ >> ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp >> >> I hope this solves your problem :) > > This worked as well, but since ZTimestamp is a public GitHub.com project, I am not sure the authentication is really used. I tried with my GitHub.com credentials (username/password) and it loaded, but when I altered the password for a wrong one, it still worked ;-) > > Now, the problem is: how do I access the private, password protected repository of my company ? > > I tried to add a host to the github:// URL but I do not have the impression that this works. > > Since making this more concrete involves some sensitive information, I will send you a private email with the details of a test project/repository/user that I made. > > Sven > > PS: I want to stress that for me Iceberg works really well on my local development machine: I created the test project/code/baseline without encountering any issue (Pharo 7). > inject credentials. So yes a better way is highly welcome ;) Norbert >> Esteban >> >> On Mon, Oct 28, 2019 at 11:58 AM Sven Van Caekenberghe <[hidden email]> wrote: >> I am still at a loss with this issue. >> >> When loading from github://svenvc/ztimestamp using Metacello (see original mail) the code in MCGitBasedNetworkRepository>>#createIcebergRepositoryWithFallbackFor:url: clearly switches to using https://github.com/svenvc/ztimestamp.git - I traced it. >> >> So obviously, Pharo 7 is capable of accessing it. >> >> However, starting directly with https://github.com/svenvc/ztimestamp.git I end up in a totally different place. Why ? >> >> This code is so complex... >> >>>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote: >>> >>> Hi, >>> >>> I asked this before, but I still have trouble with this. >>> >>> What I ultimately want is what I used for years now: use the zero config tools to point to my private repo and install my code and all its dependencies from a single Metacello config. >>> >>> Basically, >>> >>> $vm $image config _CONFIG_REPO_ _CONFIG_NAME_ --username=_CONFIG_USER_ --password=_CONFIG_PASS_ --install=_CONFIG_VERSION_ --group=_CONFIG_GROUP_ >>> >>> towards a private server. >>> >>> In the new world of git(hub)/tonel/iceberg I am trying the following (not using the 'config' but the new 'metacello' command line handler): >>> >>> $ uname -a >>> Linux audio359 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>> >>> $ curl get.pharo.org/64/70+vm | bash >>> % Total % Received % Xferd Average Speed Time Time Time Current >>> Dload Upload Total Spent Left Speed >>> 100 3054 100 3054 0 0 57622 0 --:--:-- --:--:-- --:--:-- 57622 >>> Downloading the latest 70 Image: >>> http://files.pharo.org/get-files/70/pharo64.zip >>> Pharo.image >>> Downloading the latest pharoVM: >>> http://files.pharo.org/get-files/70/pharo64-linux-stable.zip >>> pharo-vm/pharo >>> Creating starter scripts pharo and pharo-ui >>> >>> $ ./pharo Pharo.image printVersion >>> [version] 'Pharo7.0.4' 'Pharo-7.0.4+build.168.sha.ccd1f6489120f58ddeacb2cac77cd3a0f0dcfbe6 (64 Bit)' >>> >>> $ ./pharo Pharo.image save build >>> >>> $ ./pharo build.image metacello install github://svenvc/ztimestamp BaselineOfZTimestamp >>> >>> 3. using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. >>> Fetched -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] >>> Loaded -> BaselineOfZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] >>> Loading baseline of BaselineOfZTimestamp... >>> Fetched -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- https://github.com/svenvc/ztimestamp.git[master] >>> Loaded -> ZTimestamp-CompatibleUserName.1554735117 --- https://github.com/svenvc/ztimestamp.git[master] --- cache >>> ...finished baseline >>> >>> $ ./pharo build.image eval ZTimestamp now >>> 2019-10-25T12:28:34Z >>> >>> So although it gave a warning, it worked fine. >>> >>> The warning said, >>> >>> using HTTPS instead SSH (Just use an url in the form HTTPS://etc.git). I will try to clone the HTTPS variant. >>> >>> so let's try that after cleaning everything: >>> >>> $ rm -rf pharo-local/ build.* >>> >>> $ ./pharo Pharo.image save build >>> >>> $ ./pharo build.image metacello install https://github.com/svenvc/ztimestamp.git BaselineOfZTimestamp >>> >>> ...RETRY->BaselineOfZTimestamp >>> ...RETRY->BaselineOfZTimestamp >>> ...FAILED->BaselineOfZTimestampCould not resolve: BaselineOfZTimestamp [BaselineOfZTimestamp] in /home/sven/pharo7/pharo-local/package-cache https://github.com/svenvc/ztimestamp.git >>> MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>>retryingResolvePackageSpecReferences:gofer: >>> >>> WTF ? Why does this not work (I can perfectly git clone that URL) ? >>> >>> Of course, this would just be the first step, eventually I want to use my own private server with a URL like >>> >>> https://deploy:secret@.../scm/xyz/xyz-pharo.git >>> >>> How do you build/deploy non-public production code using the command line ? >>> >>> Help would be much appreciated. >>> >>> Sven >>> >> >> > > |
|
|
In reply to this post by Sven Van Caekenberghe-2
> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote: > > How do you build/deploy non-public production code using the command line ? I just learned about the following technique: Using SSH agent forwarding https://developer.github.com/v3/guides/using-ssh-agent-forwarding/ This certainly makes using certificates on production servers much easier since you do no longer have to manage or install special ones on the deployment servers, you can just use your standard developer certificates. Sven |
|
|
> Am 31.10.2019 um 12:12 schrieb Sven Van Caekenberghe <[hidden email]>: > > > >> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote: >> >> How do you build/deploy non-public production code using the command line ? > > I just learned about the following technique: > > Using SSH agent forwarding > > https://developer.github.com/v3/guides/using-ssh-agent-forwarding/ > > This certainly makes using certificates on production servers much easier since you do no longer have to manage or install special ones on the deployment servers, you can just use your standard developer certificates. > Norbert |
|
|
> On 31 Oct 2019, at 12:18, Norbert Hartl <[hidden email]> wrote: > > > >> Am 31.10.2019 um 12:12 schrieb Sven Van Caekenberghe <[hidden email]>: >> >> >> >>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote: >>> >>> How do you build/deploy non-public production code using the command line ? >> >> I just learned about the following technique: >> >> Using SSH agent forwarding >> >> https://developer.github.com/v3/guides/using-ssh-agent-forwarding/ >> >> This certainly makes using certificates on production servers much easier since you do no longer have to manage or install special ones on the deployment servers, you can just use your standard developer certificates. >> > That is quite late :P I always use this as it also chains through all the machins you log in. We use jenkins for building our software there it does not work. We just have a deployment key for that. That is a key pair without passphrase that is registered in the git repository and installed on the jenkins server. > > Norbert Indeed, the SSH forwarding cannot be used for independent CI builds. I am curious though: how do you do the initial checkout (clone) from your private repository ? Using regular git command line tools or using Iceberg/Metacello ? If the latter, how do you specify your URL, exactly ? The first case works for me too, using a gitlocal:// URL, but that is cheating a bit, I feel like we should be able to do this directly in Pharo. Sven |
|
|
> Am 01.11.2019 um 17:44 schrieb Sven Van Caekenberghe <[hidden email]>: > > > >> On 31 Oct 2019, at 12:18, Norbert Hartl <[hidden email]> wrote: >> >> >> >>> Am 31.10.2019 um 12:12 schrieb Sven Van Caekenberghe <[hidden email]>: >>> >>> >>> >>>> On 25 Oct 2019, at 14:49, Sven Van Caekenberghe <[hidden email]> wrote: >>>> >>>> How do you build/deploy non-public production code using the command line ? >>> >>> I just learned about the following technique: >>> >>> Using SSH agent forwarding >>> >>> https://developer.github.com/v3/guides/using-ssh-agent-forwarding/ >>> >>> This certainly makes using certificates on production servers much easier since you do no longer have to manage or install special ones on the deployment servers, you can just use your standard developer certificates. >>> >> That is quite late :P I always use this as it also chains through all the machins you log in. We use jenkins for building our software there it does not work. We just have a deployment key for that. That is a key pair without passphrase that is registered in the git repository and installed on the jenkins server. >> >> Norbert > > Indeed, the SSH forwarding cannot be used for independent CI builds. > > I am curious though: how do you do the initial checkout (clone) from your private repository ? Using regular git command line tools or using Iceberg/Metacello ? > > If the latter, how do you specify your URL, exactly ? > > The first case works for me too, using a gitlocal:// URL, but that is cheating a bit, I feel like we should be able to do this directly in Pharo. If checking out from private repo your problem is as deep as you reference private repos. My project (1) is in a private repository, it contains a baseline that references a project in a private repository (2) and this references a project in a private repo (3). * Project (1) is checked out by jenkins. I have a deployment key pair without passphrase where the public key is registered in the bitbucket project and the private key is deployed on jenkins. * I try to avoid iceberg as much as possible for deployment builds because later I want to have an image without iceberg. And it is not reliable to use iceberg because metacello won't update git repos automatically and the odds are high that you load old content. So I load the project with the eval commandline handler with Metacello new repository: 'filetree://source'; baseline … … * In order to load the private projects in the the baseline of the main project I run a script before build that hacks the image in a way that MCBitbucketRepository compile: 'httpsUrl ^ ''https://USER:PASS@.../'', projectPath, ''.git'''. MCBitbucketRepository compile: 'projectZipUrlFor: projectPath versionString: versionString ^ ''https://USER:PASS@.../'' , projectPath , ''/get/'' , versionString , ''.zip'''. MCBitbucketRepository class compile: 'projectZipUrlFor: projectPath versionString: versionString ^ ''https://USER:PASS@.../'' , projectPath , ''/get/'' , versionString , ''.zip'''. This works only because all private projects are on bitbucket and no public one. * Project (3) I made public because I did not see how to make that work Hope this helps. I feel this to be far too complicated. Together with the confusion between pharo5/pharo6/pharo7 with iceberg, tonel and iceberg and/or tonel support killed two weeks of my time at least. So I would welcome if there would be a better way to manage credentials Norbert |
|
|
I made a PR to allow “generic” git repositories to be referenced:
But… problem is more complicated than you may think: Metacello “vanilla” relies on download a zip file with the sources, it cannot have a “generic” way to act. This means the PR will work just if you use iceberg. And maybe you do not want to declare your baselines relying on iceberg. In that case, you have just one workaround possible: 1) clone locally the remove private sources outside the image. 2) load them using plain metacello and filetree://. 3) “lock” those repositories. Then you can run your installation. Esteban
|
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |