missing dependency ftp://.../libpng-1.6.29.tar.gz

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

missing dependency ftp://.../libpng-1.6.29.tar.gz

Ben Coman
 
It funny when the system seems to fight back.
Just as I think I resolve my freetype build issues, 
my Windows build started failing as follows...
> wget -q --no-check-certificate  -O ../../.thirdparty-cache/libpng-1.6.29.tar.gz ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/libpng-1.6.29.tar.gz
> make: *** [../third-party/Makefile.libpng:19: ../../.thirdparty-cache/libpng-1.6.29.tar.gz] Error 8

What I see at ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/
is...
Name                           Size   Date Modified
libpng-1.6.34-LICENSE.txt    4.9 kB   29/09/2017, 08:00:00
libpng-1.6.34-README.txt      96 3B   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz         1.4 MB   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz.asc      819 B   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz         975 kB   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz.asc      819 B   29/09/2017, 08:00:00
lpng1634.7z                  734 kB   29/09/2017, 08:00:00
lpng1634.7z.asc               819 B   29/09/2017, 08:00:00
lpng1634.zip                 1.3 MB   29/09/2017, 08:00:00
lpng1634.zip.asc              819 B   29/09/2017, 08:00:00

So did version 1.6.29 exist yesterday but not today? 
or am I going crazy?

Anyway, should our builds avoid referencing external resources not under our control?  Or would it be be better to copy the required tar files to a server our community controls, (either files.opensmalltalk.org or file.pharo.org)?

Plus btw our makefiles should verify sha1 checksums of these downloaded files via a field in the third-party/*spec files.

cheers -ben


Reply | Threaded
Open this post in threaded view
|

Re: missing dependency ftp://.../libpng-1.6.29.tar.gz

Eliot Miranda-2
 
Hi Ben,

On Dec 22, 2018, at 5:09 AM, Ben Coman <[hidden email]> wrote:

It funny when the system seems to fight back.
Just as I think I resolve my freetype build issues, 
my Windows build started failing as follows...
> wget -q --no-check-certificate  -O ../../.thirdparty-cache/libpng-1.6.29.tar.gz ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/libpng-1.6.29.tar.gz
> make: *** [../third-party/Makefile.libpng:19: ../../.thirdparty-cache/libpng-1.6.29.tar.gz] Error 8

What I see at ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/
is...
Name                           Size   Date Modified
libpng-1.6.34-LICENSE.txt    4.9 kB   29/09/2017, 08:00:00
libpng-1.6.34-README.txt      96 3B   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz         1.4 MB   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz.asc      819 B   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz         975 kB   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz.asc      819 B   29/09/2017, 08:00:00
lpng1634.7z                  734 kB   29/09/2017, 08:00:00
lpng1634.7z.asc               819 B   29/09/2017, 08:00:00
lpng1634.zip                 1.3 MB   29/09/2017, 08:00:00
lpng1634.zip.asc              819 B   29/09/2017, 08:00:00

So did version 1.6.29 exist yesterday but not today? 
or am I going crazy?

:-(

Anyway, should our builds avoid referencing external resources not under our control?  Or would it be be better to copy the required tar files to a server our community controls, (either files.opensmalltalk.org or file.pharo.org)?

I don’t see how the former is possible :-(. Do some variation of the latter is necessary.  I like your suggestion of files.pharo.org; files.opensmalltalk.org doesn’t exist and do would need funding.  I think adding the tar files to opensmalltalk-vm itself would be a huge mistake; it would cause the repository to bloat quite quickly right?  So what are the logistics of using files.pharo.org?

Plus btw our makefiles should verify sha1 checksums of these downloaded files via a field in the third-party/*spec files.

+1

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: missing dependency ftp://.../libpng-1.6.29.tar.gz

Nicolas Cellier
 
Submodule is also a possibility. If the repo is not on github, we could clone it under opensmalltalk in order to reduce web failures.

Le sam. 22 déc. 2018 à 14:45, Eliot Miranda <[hidden email]> a écrit :
 
Hi Ben,

On Dec 22, 2018, at 5:09 AM, Ben Coman <[hidden email]> wrote:

It funny when the system seems to fight back.
Just as I think I resolve my freetype build issues, 
my Windows build started failing as follows...
> wget -q --no-check-certificate  -O ../../.thirdparty-cache/libpng-1.6.29.tar.gz ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/libpng-1.6.29.tar.gz
> make: *** [../third-party/Makefile.libpng:19: ../../.thirdparty-cache/libpng-1.6.29.tar.gz] Error 8

What I see at ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/
is...
Name                           Size   Date Modified
libpng-1.6.34-LICENSE.txt    4.9 kB   29/09/2017, 08:00:00
libpng-1.6.34-README.txt      96 3B   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz         1.4 MB   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz.asc      819 B   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz         975 kB   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz.asc      819 B   29/09/2017, 08:00:00
lpng1634.7z                  734 kB   29/09/2017, 08:00:00
lpng1634.7z.asc               819 B   29/09/2017, 08:00:00
lpng1634.zip                 1.3 MB   29/09/2017, 08:00:00
lpng1634.zip.asc              819 B   29/09/2017, 08:00:00

So did version 1.6.29 exist yesterday but not today? 
or am I going crazy?

:-(

Anyway, should our builds avoid referencing external resources not under our control?  Or would it be be better to copy the required tar files to a server our community controls, (either files.opensmalltalk.org or file.pharo.org)?

I don’t see how the former is possible :-(. Do some variation of the latter is necessary.  I like your suggestion of files.pharo.org; files.opensmalltalk.org doesn’t exist and do would need funding.  I think adding the tar files to opensmalltalk-vm itself would be a huge mistake; it would cause the repository to bloat quite quickly right?  So what are the logistics of using files.pharo.org?

Plus btw our makefiles should verify sha1 checksums of these downloaded files via a field in the third-party/*spec files.

+1

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: missing dependency ftp://.../libpng-1.6.29.tar.gz

fniephaus
 
Hi all,

On Sat, Dec 22, 2018 at 6:28 PM Nicolas Cellier
<[hidden email]> wrote:

>
>
> Submodule is also a possibility. If the repo is not on github, we could clone it under opensmalltalk in order to reduce web failures.
>
> Le sam. 22 déc. 2018 à 14:45, Eliot Miranda <[hidden email]> a écrit :
>>
>>
>> Hi Ben,
>>
>> On Dec 22, 2018, at 5:09 AM, Ben Coman <[hidden email]> wrote:
>>
>> It funny when the system seems to fight back.
>> Just as I think I resolve my freetype build issues,
>> my Windows build started failing as follows...
>> > wget -q --no-check-certificate  -O ../../.thirdparty-cache/libpng-1.6.29.tar.gz ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/libpng-1.6.29.tar.gz
>> > make: *** [../third-party/Makefile.libpng:19: ../../.thirdparty-cache/libpng-1.6.29.tar.gz] Error 8
>>
>> What I see at ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/
>> is...
>> Name                           Size   Date Modified
>> libpng-1.6.34-LICENSE.txt    4.9 kB   29/09/2017, 08:00:00
>> libpng-1.6.34-README.txt      96 3B   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.gz         1.4 MB   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.gz.asc      819 B   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.xz         975 kB   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.xz.asc      819 B   29/09/2017, 08:00:00
>> lpng1634.7z                  734 kB   29/09/2017, 08:00:00
>> lpng1634.7z.asc               819 B   29/09/2017, 08:00:00
>> lpng1634.zip                 1.3 MB   29/09/2017, 08:00:00
>> lpng1634.zip.asc              819 B   29/09/2017, 08:00:00
>>
>> So did version 1.6.29 exist yesterday but not today?
>> or am I going crazy?
>>
>>
>> :-(
>>
>> Anyway, should our builds avoid referencing external resources not under our control?  Or would it be be better to copy the required tar files to a server our community controls, (either files.opensmalltalk.org or file.pharo.org)?
>>
>>
>> I don’t see how the former is possible :-(. Do some variation of the latter is necessary.  I like your suggestion of files.pharo.org; files.opensmalltalk.org doesn’t exist and do would need funding.  I think adding the tar files to opensmalltalk-vm itself would be a huge mistake; it would cause the repository to bloat quite quickly right?  So what are the logistics of using files.pharo.org?
>>

How about something like this:
https://github.com/OpenSmalltalk/opensmalltalk-vm-dependencies

We basically maintain a second repository with links to all
dependencies and mirror corresponding files as part of a single GitHub
release (see [1] for limitations). Advantages: free plus TravisCI and
AppVeyor optimize their network connection to GitHub.

Example release:
https://github.com/OpenSmalltalk/opensmalltalk-vm-dependencies/releases

Example download link:
https://github.com/OpenSmalltalk/opensmalltalk-vm-dependencies/releases/download/latest/libpng-1.6.29.tar.gz

Cheers,
Fabio

[1] https://help.github.com/articles/about-releases/#limitations-on-binary-files

>> Plus btw our makefiles should verify sha1 checksums of these downloaded files via a field in the third-party/*spec files.
>>
>>
>> +1
>>
>> cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: missing dependency ftp://.../libpng-1.6.29.tar.gz

Ben Coman
 


On Sun, 23 Dec 2018 at 01:56, Fabio Niephaus <[hidden email]> wrote:
 
Hi all,

On Sat, Dec 22, 2018 at 6:28 PM Nicolas Cellier
<[hidden email]> wrote:
>
>
> Submodule is also a possibility. If the repo is not on github, we could clone it under opensmalltalk in order to reduce web failures.
>
> Le sam. 22 déc. 2018 à 14:45, Eliot Miranda <[hidden email]> a écrit :
>>
>>
>> Hi Ben,
>>
>> On Dec 22, 2018, at 5:09 AM, Ben Coman <[hidden email]> wrote:
>>
>> It funny when the system seems to fight back.
>> Just as I think I resolve my freetype build issues,
>> my Windows build started failing as follows...
>> > wget -q --no-check-certificate  -O ../../.thirdparty-cache/libpng-1.6.29.tar.gz ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/libpng-1.6.29.tar.gz
>> > make: *** [../third-party/Makefile.libpng:19: ../../.thirdparty-cache/libpng-1.6.29.tar.gz] Error 8
>>
>> What I see at ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/
>> is...
>> Name                           Size   Date Modified
>> libpng-1.6.34-LICENSE.txt    4.9 kB   29/09/2017, 08:00:00
>> libpng-1.6.34-README.txt      96 3B   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.gz         1.4 MB   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.gz.asc      819 B   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.xz         975 kB   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.xz.asc      819 B   29/09/2017, 08:00:00
>> lpng1634.7z                  734 kB   29/09/2017, 08:00:00
>> lpng1634.7z.asc               819 B   29/09/2017, 08:00:00
>> lpng1634.zip                 1.3 MB   29/09/2017, 08:00:00
>> lpng1634.zip.asc              819 B   29/09/2017, 08:00:00
>>
>> So did version 1.6.29 exist yesterday but not today?
>> or am I going crazy?
>>
>>
>> :-(
>>
>> Anyway, should our builds avoid referencing external resources not under our control?  Or would it be be better to copy the required tar files to a server our community controls, (either files.opensmalltalk.org or file.pharo.org)?
>>
>>
>> I don’t see how the former is possible :-(. Do some variation of the latter is necessary.  I like your suggestion of files.pharo.org; files.opensmalltalk.org doesn’t exist and do would need funding.  I think adding the tar files to opensmalltalk-vm itself would be a huge mistake; it would cause the repository to bloat quite quickly right? 

Yes. I agree.
 
So what are the logistics of using files.pharo.org?

That would be up to Esteban, et al,
but Fabio's suggestion might be a better way to go...
 
How about something like this:
https://github.com/OpenSmalltalk/opensmalltalk-vm-dependencies

We basically maintain a second repository with links to all
dependencies and mirror corresponding files as part of a single GitHub
release (see [1] for limitations). Advantages: free plus TravisCI and
AppVeyor optimize their network connection to GitHub. 

Example release:
https://github.com/OpenSmalltalk/opensmalltalk-vm-dependencies/releases

Example download link:
https://github.com/OpenSmalltalk/opensmalltalk-vm-dependencies/releases/download/latest/libpng-1.6.29.tar.gz

... but actually, like Nicolas suggests I've often thought keeping a local fork of dependencies 
 to be a pragmatic safety measure (e.g. under a branch called "mirror").  But since we end up with dll and so files
I wonder if there is another path than using submodules.  When a newcomer freshly clones opensmalltalk-vm
the majority of build time for pharo.cog.spur is the third-party libraries.  This seems a bit of a waste
when those dependencies don't change often.  If we have a repo per dependency a CI build 
could "release" built dependent dll and so files.  So these could be downloaded rather than built each time.

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: missing dependency ftp://.../libpng-1.6.29.tar.gz

Ben Coman
In reply to this post by Ben Coman
 


On Sat, 22 Dec 2018 at 21:09, Ben Coman <[hidden email]> wrote:
It funny when the system seems to fight back.
Just as I think I resolve my freetype build issues, 
my Windows build started failing as follows...
> wget -q --no-check-certificate  -O ../../.thirdparty-cache/libpng-1.6.29.tar.gz ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/libpng-1.6.29.tar.gz
> make: *** [../third-party/Makefile.libpng:19: ../../.thirdparty-cache/libpng-1.6.29.tar.gz] Error 8

What I see at ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/
is...
Name                           Size   Date Modified
libpng-1.6.34-LICENSE.txt    4.9 kB   29/09/2017, 08:00:00
libpng-1.6.34-README.txt      96 3B   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz         1.4 MB   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz.asc      819 B   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz         975 kB   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz.asc      819 B   29/09/2017, 08:00:00
lpng1634.7z                  734 kB   29/09/2017, 08:00:00
lpng1634.7z.asc               819 B   29/09/2017, 08:00:00
lpng1634.zip                 1.3 MB   29/09/2017, 08:00:00
lpng1634.zip.asc              819 B   29/09/2017, 08:00:00

So did version 1.6.29 exist yesterday but not today? 
or am I going crazy?

Not to reduce the significance of the point I raise, but just sharing that I discovered that I had built
from an old branch, so the server for this missing tar.gz file had already been corrected 11 months ago.
 
cheers -ben


Anyway, should our builds avoid referencing external resources not under our control?  Or would it be be better to copy the required tar files to a server our community controls, (either files.opensmalltalk.org or file.pharo.org)?

Plus btw our makefiles should verify sha1 checksums of these downloaded files via a field in the third-party/*spec files.

cheers -ben


Reply | Threaded
Open this post in threaded view
|

Re: missing dependency ftp://.../libpng-1.6.29.tar.gz

fniephaus
In reply to this post by Ben Coman
 


On Sun, 23 Dec 2018 at 1:34 pm, Ben Coman <[hidden email]> wrote:
 


On Sun, 23 Dec 2018 at 01:56, Fabio Niephaus <[hidden email]> wrote:
 
Hi all,

On Sat, Dec 22, 2018 at 6:28 PM Nicolas Cellier
<[hidden email]> wrote:
>
>
> Submodule is also a possibility. If the repo is not on github, we could clone it under opensmalltalk in order to reduce web failures.
>
> Le sam. 22 déc. 2018 à 14:45, Eliot Miranda <[hidden email]> a écrit :
>>
>>
>> Hi Ben,
>>
>> On Dec 22, 2018, at 5:09 AM, Ben Coman <[hidden email]> wrote:
>>
>> It funny when the system seems to fight back.
>> Just as I think I resolve my freetype build issues,
>> my Windows build started failing as follows...
>> > wget -q --no-check-certificate  -O ../../.thirdparty-cache/libpng-1.6.29.tar.gz ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/libpng-1.6.29.tar.gz
>> > make: *** [../third-party/Makefile.libpng:19: ../../.thirdparty-cache/libpng-1.6.29.tar.gz] Error 8
>>
>> What I see at ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/
>> is...
>> Name                           Size   Date Modified
>> libpng-1.6.34-LICENSE.txt    4.9 kB   29/09/2017, 08:00:00
>> libpng-1.6.34-README.txt      96 3B   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.gz         1.4 MB   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.gz.asc      819 B   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.xz         975 kB   29/09/2017, 08:00:00
>> libpng-1.6.34.tar.xz.asc      819 B   29/09/2017, 08:00:00
>> lpng1634.7z                  734 kB   29/09/2017, 08:00:00
>> lpng1634.7z.asc               819 B   29/09/2017, 08:00:00
>> lpng1634.zip                 1.3 MB   29/09/2017, 08:00:00
>> lpng1634.zip.asc              819 B   29/09/2017, 08:00:00
>>
>> So did version 1.6.29 exist yesterday but not today?
>> or am I going crazy?
>>
>>
>> :-(
>>
>> Anyway, should our builds avoid referencing external resources not under our control?  Or would it be be better to copy the required tar files to a server our community controls, (either files.opensmalltalk.org or file.pharo.org)?
>>
>>
>> I don’t see how the former is possible :-(. Do some variation of the latter is necessary.  I like your suggestion of files.pharo.org; files.opensmalltalk.org doesn’t exist and do would need funding.  I think adding the tar files to opensmalltalk-vm itself would be a huge mistake; it would cause the repository to bloat quite quickly right? 

Yes. I agree.

Agreed. Git is not designed to store large files and removing large files from the history was the most annoying part when we migrated from SVN to Git.

 
So what are the logistics of using files.pharo.org?

That would be up to Esteban, et al,
but Fabio's suggestion might be a better way to go...

I'm in favor of keeping the VM as independent as possible, so I'd like a solution everyone can easily contribute to.

 
How about something like this:
https://github.com/OpenSmalltalk/opensmalltalk-vm-dependencies

We basically maintain a second repository with links to all
dependencies and mirror corresponding files as part of a single GitHub
release (see [1] for limitations). Advantages: free plus TravisCI and
AppVeyor optimize their network connection to GitHub. 

Example release:
https://github.com/OpenSmalltalk/opensmalltalk-vm-dependencies/releases

Example download link:
https://github.com/OpenSmalltalk/opensmalltalk-vm-dependencies/releases/download/latest/libpng-1.6.29.tar.gz

... but actually, like Nicolas suggests I've often thought keeping a local fork of dependencies 
 to be a pragmatic safety measure (e.g. under a branch called "mirror").  But since we end up with dll and so files
I wonder if there is another path than using submodules.  When a newcomer freshly clones opensmalltalk-vm
the majority of build time for pharo.cog.spur is the third-party libraries.  This seems a bit of a waste
when those dependencies don't change often.  If we have a repo per dependency a CI build 
could "release" built dependent dll and so files.  So these could be downloaded rather than built each time

I hope third party libs only need to be compiled once after a fresh checkout. To avoid this overhead, we cache all of this on CI. But I agree, the real problem is that some external plugins require third party libs and are part of the OSVM repo. Why do we have a VM which is easy to extend if we don't use that mechanism? So +1 for externalizing external plugins into other repos under the OpenSmallalk umbrella. This way, breaking a plugin doesn't break the rest of the VM.

Fabio

.

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: missing dependency ftp://.../libpng-1.6.29.tar.gz

fniephaus
In reply to this post by Ben Coman
 


On Sun, 23 Dec 2018 at 1:47 pm, Ben Coman <[hidden email]> wrote:
 


On Sat, 22 Dec 2018 at 21:09, Ben Coman <[hidden email]> wrote:
It funny when the system seems to fight back.
Just as I think I resolve my freetype build issues, 
my Windows build started failing as follows...
> wget -q --no-check-certificate  -O ../../.thirdparty-cache/libpng-1.6.29.tar.gz ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/libpng-1.6.29.tar.gz
> make: *** [../third-party/Makefile.libpng:19: ../../.thirdparty-cache/libpng-1.6.29.tar.gz] Error 8

What I see at ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng16/
is...
Name                           Size   Date Modified
libpng-1.6.34-LICENSE.txt    4.9 kB   29/09/2017, 08:00:00
libpng-1.6.34-README.txt      96 3B   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz         1.4 MB   29/09/2017, 08:00:00
libpng-1.6.34.tar.gz.asc      819 B   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz         975 kB   29/09/2017, 08:00:00
libpng-1.6.34.tar.xz.asc      819 B   29/09/2017, 08:00:00
lpng1634.7z                  734 kB   29/09/2017, 08:00:00
lpng1634.7z.asc               819 B   29/09/2017, 08:00:00
lpng1634.zip                 1.3 MB   29/09/2017, 08:00:00
lpng1634.zip.asc              819 B   29/09/2017, 08:00:00

So did version 1.6.29 exist yesterday but not today? 
or am I going crazy?

Not to reduce the significance of the point I raise, but just sharing that I discovered that I had built
from an old branch, so the server for this missing tar.gz file had already been corrected 11 months ago.

I think the problem here is that we link against external files that are not under our control. Although URLs are not suppose to change or become unavailable, they break way too often. I think the best way for us going forward would be to maintain a mirror for such dependencies.

Fabio

 
cheers -ben


Anyway, should our builds avoid referencing external resources not under our control?  Or would it be be better to copy the required tar files to a server our community controls, (either files.opensmalltalk.org or file.pharo.org)?

Plus btw our makefiles should verify sha1 checksums of these downloaded files via a field in the third-party/*spec files.

cheers -ben