StandardFileStream size limit?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

StandardFileStream size limit?

Göran Krampe
Hey all!

I have just added large file (chunkwise upload and download) support to
Kom/Networking/Iliad in Pharo 1.4 and while testing it I tried uploading
a 2.9Gb file.

I haven't looked at the exact error message, and I haven't tried writing
a trivial test - but the "half written" file on disk is almost EXACTLY
2Gb (2147483647) so I am wondering - do we still have a 2Gb file size limit?

This is "one click 1.4" on Ubuntu 64 with ext4.

regards, Göran

Reply | Threaded
Open this post in threaded view
|

Cog issue? Re: StandardFileStream size limit?

Göran Krampe
Hey!

Ok, so the plot thickens:

If I run this in a "oneclick 1.4" I will get a file write error on 2Gb:

| f b |
f := StandardFileStream newFileNamed: 'test'.
b := ByteArray new: 1024*1024*100. "100Mb"
[30 timesRepeat: [f nextPutAll: b]] ensure: [f close] "3Gb"


...but it worked fine using a self built "standard" VM from squeakvm.org
(4.4.7.2357)! Same image btw.

Now, does it work with the bleeding edge Cog? Tried r2556 and nope, same
problem.

So I "guess" it is a Cog thing? I haven't tried building Cog from source.

regards, Göran

Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Cog issue? Re: StandardFileStream size limit?

David T. Lewis
On Wed, Jun 13, 2012 at 04:08:14PM +0200, G?ran Krampe wrote:

>
> Hey!
>
> Ok, so the plot thickens:
>
> If I run this in a "oneclick 1.4" I will get a file write error on 2Gb:
>
> | f b |
> f := StandardFileStream newFileNamed: 'test'.
> b := ByteArray new: 1024*1024*100. "100Mb"
> [30 timesRepeat: [f nextPutAll: b]] ensure: [f close] "3Gb"
>
>
> ...but it worked fine using a self built "standard" VM from squeakvm.org
> (4.4.7.2357)! Same image btw.
>
> Now, does it work with the bleeding edge Cog? Tried r2556 and nope, same
> problem.
>
> So I "guess" it is a Cog thing? I haven't tried building Cog from source.
>
> regards, G?ran

Background on large file support, see especially Bert's summary:

http://forum.world.st/Re-squeak-dev-filesize-reporting-0-for-very-large-files-td4483646.html
http://bugs.squeak.org/view.php?id=7522
http://www.suse.de/~aj/linux_lfs.html

I'm not sure if the Cog VMs are being compiled with the the LFS option, although
I expect that if you compile it yourself with the build options that Bert explains,
then it should start working.

Dave


Reply | Threaded
Open this post in threaded view
|

Re: StandardFileStream size limit?

Eliot Miranda-2
In reply to this post by Göran Krampe


On Wed, Jun 13, 2012 at 12:35 AM, Göran Krampe <[hidden email]> wrote:
Hey all!

I have just added large file (chunkwise upload and download) support to Kom/Networking/Iliad in Pharo 1.4 and while testing it I tried uploading a 2.9Gb file.

I haven't looked at the exact error message, and I haven't tried writing a trivial test - but the "half written" file on disk is almost EXACTLY 2Gb <a href="tel:%282147483647" value="+12147483647" target="_blank">(2147483647) so I am wondering - do we still have a 2Gb file size limit?

I think so.  Still need to add the relevant define (LARGEFILE_SOURCE or some such) to the linux build to enable 64-bit file support. 

This is "one click 1.4" on Ubuntu 64 with ext4.

regards, Göran

--
best,
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Cog issue? Re: StandardFileStream size limit?

Eliot Miranda-2
In reply to this post by David T. Lewis


On Wed, Jun 13, 2012 at 7:40 AM, David T. Lewis <[hidden email]> wrote:

On Wed, Jun 13, 2012 at 04:08:14PM +0200, G?ran Krampe wrote:
>
> Hey!
>
> Ok, so the plot thickens:
>
> If I run this in a "oneclick 1.4" I will get a file write error on 2Gb:
>
> | f b |
> f := StandardFileStream newFileNamed: 'test'.
> b := ByteArray new: 1024*1024*100. "100Mb"
> [30 timesRepeat: [f nextPutAll: b]] ensure: [f close] "3Gb"
>
>
> ...but it worked fine using a self built "standard" VM from squeakvm.org
> (4.4.7.2357)! Same image btw.
>
> Now, does it work with the bleeding edge Cog? Tried r2556 and nope, same
> problem.
>
> So I "guess" it is a Cog thing? I haven't tried building Cog from source.
>
> regards, G?ran

Background on large file support, see especially Bert's summary:

http://forum.world.st/Re-squeak-dev-filesize-reporting-0-for-very-large-files-td4483646.html
http://bugs.squeak.org/view.php?id=7522
http://www.suse.de/~aj/linux_lfs.html

I'm not sure if the Cog VMs are being compiled with the the LFS option, although
I expect that if you compile it yourself with the build options that Bert explains,
then it should start working.

I just committed the necessary changes for my branch.  Will rebuild soon.  Göran, if you're in a hurry build your own?  You'd need to be in http://www.squeakvm.org/svn/squeak/branches/Cog/unixbuild/bld and run ./mvm.
--
best,
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Cog issue? Re: StandardFileStream size limit?

Jimmie Houchin-5
This still fails three and a half years later with a default Pharo 4 vm
and image.
:(

I spent a few hours trying to diagnose why Pharo was telling me that I
file I knew existed didn't exist. Very frustrating trying to debug this
and then to find it is a vm and not an image issue. I then to search the
archives to find this is a known issue for this long and still broken.

The latest linux vm works fine.
http://files.pharo.org/vm/pharo/linux/Pharo-VM-linux-latest.zip

Is there a reason why the "latest" vm isn't the stable vm? Or at least
this bugfix applied to the stable vm?

Thanks.

Jimmie



On 06/13/2012 05:00 PM, eliot.miranda at gmail.com (Eliot Miranda) wrote:

> On Wed, Jun 13, 2012 at 7:40 AM, David T. Lewis <lewis at mail.msen.com> wrote:
>
>> On Wed, Jun 13, 2012 at 04:08:14PM +0200, G?ran Krampe wrote:
>>> Hey!
>>>
>>> Ok, so the plot thickens:
>>>
>>> If I run this in a "oneclick 1.4" I will get a file write error on 2Gb:
>>>
>>> | f b |
>>> f := StandardFileStream newFileNamed: 'test'.
>>> b := ByteArray new: 1024*1024*100. "100Mb"
>>> [30 timesRepeat: [f nextPutAll: b]] ensure: [f close] "3Gb"
>>>
>>>
>>> ...but it worked fine using a self built "standard" VM from squeakvm.org
>>> (4.4.7.2357)! Same image btw.
>>>
>>> Now, does it work with the bleeding edge Cog? Tried r2556 and nope, same
>>> problem.
>>>
>>> So I "guess" it is a Cog thing? I haven't tried building Cog from source.
>>>
>>> regards, G?ran
>> Background on large file support, see especially Bert's summary:
>>
>>
>> http://forum.world.st/Re-squeak-dev-filesize-reporting-0-for-very-large-files-td4483646.html
>> http://bugs.squeak.org/view.php?id=7522
>> http://www.suse.de/~aj/linux_lfs.html
>>
>> I'm not sure if the Cog VMs are being compiled with the the LFS option,
>> although
>> I expect that if you compile it yourself with the build options that Bert
>> explains,
>> then it should start working.
>>
> I just committed the necessary changes for my branch.  Will rebuild soon.
>   G?ran, if you're in a hurry build your own?  You'd need to be in
> http://www.squeakvm.org/svn/squeak/branches/Cog/unixbuild/bld and run ./mvm.


Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Cog issue? Re: StandardFileStream size limit?

Nicolai Hess-3-2


2015-11-12 6:14 GMT+01:00 Jimmie Houchin <[hidden email]>:
This still fails three and a half years later with a default Pharo 4 vm and image.
:(

I spent a few hours trying to diagnose why Pharo was telling me that I file I knew existed didn't exist. Very frustrating trying to debug this and then to find it is a vm and not an image issue. I then to search the archives to find this is a known issue for this long and still broken.

The latest linux vm works fine.
http://files.pharo.org/vm/pharo/linux/Pharo-VM-linux-latest.zip

Yes, the latest vm is pretty stable. But I think the problem is, there are to many changes for to many
different projects:
- I added a change to the windows keyevent handling and created a small  regression (can not use ctrl+space anymore)
- changes for OSWindow /SDL
- spur (?) or any other changes from the main squeak vm
- ...

I think we should have create release candidate branch and make sure more people testing that version and all
newer changes go to the master/development branch only.

nicolai
 

Is there a reason why the "latest" vm isn't the stable vm? Or at least this bugfix applied to the stable vm?

Thanks.

Jimmie



On 06/13/2012 05:00 PM, eliot.miranda at gmail.com (Eliot Miranda) wrote:
On Wed, Jun 13, 2012 at 7:40 AM, David T. Lewis <lewis at mail.msen.com> wrote:

On Wed, Jun 13, 2012 at 04:08:14PM +0200, G?ran Krampe wrote:
Hey!

Ok, so the plot thickens:

If I run this in a "oneclick 1.4" I will get a file write error on 2Gb:

| f b |
f := StandardFileStream newFileNamed: 'test'.
b := ByteArray new: 1024*1024*100. "100Mb"
[30 timesRepeat: [f nextPutAll: b]] ensure: [f close] "3Gb"


...but it worked fine using a self built "standard" VM from squeakvm.org
(4.4.7.2357)! Same image btw.

Now, does it work with the bleeding edge Cog? Tried r2556 and nope, same
problem.

So I "guess" it is a Cog thing? I haven't tried building Cog from source.

regards, G?ran
Background on large file support, see especially Bert's summary:


http://forum.world.st/Re-squeak-dev-filesize-reporting-0-for-very-large-files-td4483646.html
http://bugs.squeak.org/view.php?id=7522
http://www.suse.de/~aj/linux_lfs.html

I'm not sure if the Cog VMs are being compiled with the the LFS option,
although
I expect that if you compile it yourself with the build options that Bert
explains,
then it should start working.

I just committed the necessary changes for my branch.  Will rebuild soon.
  G?ran, if you're in a hurry build your own?  You'd need to be in
http://www.squeakvm.org/svn/squeak/branches/Cog/unixbuild/bld and run ./mvm.



Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Cog issue? Re: StandardFileStream size limit?

kilon.alios
latest vm has issues with OSProcess stdout the last time I checked on Yosemite. I would not call it stable unless we talk only crashes here.

On Thu, Nov 12, 2015 at 2:55 PM Nicolai Hess <[hidden email]> wrote:
2015-11-12 6:14 GMT+01:00 Jimmie Houchin <[hidden email]>:
This still fails three and a half years later with a default Pharo 4 vm and image.
:(

I spent a few hours trying to diagnose why Pharo was telling me that I file I knew existed didn't exist. Very frustrating trying to debug this and then to find it is a vm and not an image issue. I then to search the archives to find this is a known issue for this long and still broken.

The latest linux vm works fine.
http://files.pharo.org/vm/pharo/linux/Pharo-VM-linux-latest.zip

Yes, the latest vm is pretty stable. But I think the problem is, there are to many changes for to many
different projects:
- I added a change to the windows keyevent handling and created a small  regression (can not use ctrl+space anymore)
- changes for OSWindow /SDL
- spur (?) or any other changes from the main squeak vm
- ...

I think we should have create release candidate branch and make sure more people testing that version and all
newer changes go to the master/development branch only.

nicolai
 

Is there a reason why the "latest" vm isn't the stable vm? Or at least this bugfix applied to the stable vm?

Thanks.

Jimmie



On 06/13/2012 05:00 PM, eliot.miranda at gmail.com (Eliot Miranda) wrote:
On Wed, Jun 13, 2012 at 7:40 AM, David T. Lewis <lewis at mail.msen.com> wrote:

On Wed, Jun 13, 2012 at 04:08:14PM +0200, G?ran Krampe wrote:
Hey!

Ok, so the plot thickens:

If I run this in a "oneclick 1.4" I will get a file write error on 2Gb:

| f b |
f := StandardFileStream newFileNamed: 'test'.
b := ByteArray new: 1024*1024*100. "100Mb"
[30 timesRepeat: [f nextPutAll: b]] ensure: [f close] "3Gb"


...but it worked fine using a self built "standard" VM from squeakvm.org
(4.4.7.2357)! Same image btw.

Now, does it work with the bleeding edge Cog? Tried r2556 and nope, same
problem.

So I "guess" it is a Cog thing? I haven't tried building Cog from source.

regards, G?ran
Background on large file support, see especially Bert's summary:


http://forum.world.st/Re-squeak-dev-filesize-reporting-0-for-very-large-files-td4483646.html
http://bugs.squeak.org/view.php?id=7522
http://www.suse.de/~aj/linux_lfs.html

I'm not sure if the Cog VMs are being compiled with the the LFS option,
although
I expect that if you compile it yourself with the build options that Bert
explains,
then it should start working.

I just committed the necessary changes for my branch.  Will rebuild soon.
  G?ran, if you're in a hurry build your own?  You'd need to be in
http://www.squeakvm.org/svn/squeak/branches/Cog/unixbuild/bld and run ./mvm.


Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Cog issue? Re: StandardFileStream size limit?

Nicolai Hess-3-2


2015-11-12 14:10 GMT+01:00 Dimitris Chloupis <[hidden email]>:
latest vm has issues with OSProcess stdout the last time I checked on Yosemite. I would not call it stable unless we talk only crashes here.

Ah, I didn't know this was a vm issue. Does this means it is working with the "official" stable vm ?
 

On Thu, Nov 12, 2015 at 2:55 PM Nicolai Hess <[hidden email]> wrote:
2015-11-12 6:14 GMT+01:00 Jimmie Houchin <[hidden email]>:
This still fails three and a half years later with a default Pharo 4 vm and image.
:(

I spent a few hours trying to diagnose why Pharo was telling me that I file I knew existed didn't exist. Very frustrating trying to debug this and then to find it is a vm and not an image issue. I then to search the archives to find this is a known issue for this long and still broken.

The latest linux vm works fine.
http://files.pharo.org/vm/pharo/linux/Pharo-VM-linux-latest.zip

Yes, the latest vm is pretty stable. But I think the problem is, there are to many changes for to many
different projects:
- I added a change to the windows keyevent handling and created a small  regression (can not use ctrl+space anymore)
- changes for OSWindow /SDL
- spur (?) or any other changes from the main squeak vm
- ...

I think we should have create release candidate branch and make sure more people testing that version and all
newer changes go to the master/development branch only.

nicolai
 

Is there a reason why the "latest" vm isn't the stable vm? Or at least this bugfix applied to the stable vm?

Thanks.

Jimmie



On 06/13/2012 05:00 PM, eliot.miranda at gmail.com (Eliot Miranda) wrote:
On Wed, Jun 13, 2012 at 7:40 AM, David T. Lewis <lewis at mail.msen.com> wrote:

On Wed, Jun 13, 2012 at 04:08:14PM +0200, G?ran Krampe wrote:
Hey!

Ok, so the plot thickens:

If I run this in a "oneclick 1.4" I will get a file write error on 2Gb:

| f b |
f := StandardFileStream newFileNamed: 'test'.
b := ByteArray new: 1024*1024*100. "100Mb"
[30 timesRepeat: [f nextPutAll: b]] ensure: [f close] "3Gb"


...but it worked fine using a self built "standard" VM from squeakvm.org
(4.4.7.2357)! Same image btw.

Now, does it work with the bleeding edge Cog? Tried r2556 and nope, same
problem.

So I "guess" it is a Cog thing? I haven't tried building Cog from source.

regards, G?ran
Background on large file support, see especially Bert's summary:


http://forum.world.st/Re-squeak-dev-filesize-reporting-0-for-very-large-files-td4483646.html
http://bugs.squeak.org/view.php?id=7522
http://www.suse.de/~aj/linux_lfs.html

I'm not sure if the Cog VMs are being compiled with the the LFS option,
although
I expect that if you compile it yourself with the build options that Bert
explains,
then it should start working.

I just committed the necessary changes for my branch.  Will rebuild soon.
  G?ran, if you're in a hurry build your own?  You'd need to be in
http://www.squeakvm.org/svn/squeak/branches/Cog/unixbuild/bld and run ./mvm.



Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Cog issue? Re: StandardFileStream size limit?

Nicolai Hess-3-2


2015-11-12 14:15 GMT+01:00 Nicolai Hess <[hidden email]>:


2015-11-12 14:10 GMT+01:00 Dimitris Chloupis <[hidden email]>:
latest vm has issues with OSProcess stdout the last time I checked on Yosemite. I would not call it stable unless we talk only crashes here.

Ah, I didn't know this was a vm issue. Does this means it is working with the "official" stable vm ?

Can you check if OSProcess is working with the stable vm? This may help to fix this issue.

Thanks
nicolai

 
 

On Thu, Nov 12, 2015 at 2:55 PM Nicolai Hess <[hidden email]> wrote:
2015-11-12 6:14 GMT+01:00 Jimmie Houchin <[hidden email]>:
This still fails three and a half years later with a default Pharo 4 vm and image.
:(

I spent a few hours trying to diagnose why Pharo was telling me that I file I knew existed didn't exist. Very frustrating trying to debug this and then to find it is a vm and not an image issue. I then to search the archives to find this is a known issue for this long and still broken.

The latest linux vm works fine.
http://files.pharo.org/vm/pharo/linux/Pharo-VM-linux-latest.zip

Yes, the latest vm is pretty stable. But I think the problem is, there are to many changes for to many
different projects:
- I added a change to the windows keyevent handling and created a small  regression (can not use ctrl+space anymore)
- changes for OSWindow /SDL
- spur (?) or any other changes from the main squeak vm
- ...

I think we should have create release candidate branch and make sure more people testing that version and all
newer changes go to the master/development branch only.

nicolai
 

Is there a reason why the "latest" vm isn't the stable vm? Or at least this bugfix applied to the stable vm?

Thanks.

Jimmie



On 06/13/2012 05:00 PM, eliot.miranda at gmail.com (Eliot Miranda) wrote:
On Wed, Jun 13, 2012 at 7:40 AM, David T. Lewis <lewis at mail.msen.com> wrote:

On Wed, Jun 13, 2012 at 04:08:14PM +0200, G?ran Krampe wrote:
Hey!

Ok, so the plot thickens:

If I run this in a "oneclick 1.4" I will get a file write error on 2Gb:

| f b |
f := StandardFileStream newFileNamed: 'test'.
b := ByteArray new: 1024*1024*100. "100Mb"
[30 timesRepeat: [f nextPutAll: b]] ensure: [f close] "3Gb"


...but it worked fine using a self built "standard" VM from squeakvm.org
(4.4.7.2357)! Same image btw.

Now, does it work with the bleeding edge Cog? Tried r2556 and nope, same
problem.

So I "guess" it is a Cog thing? I haven't tried building Cog from source.

regards, G?ran
Background on large file support, see especially Bert's summary:


http://forum.world.st/Re-squeak-dev-filesize-reporting-0-for-very-large-files-td4483646.html
http://bugs.squeak.org/view.php?id=7522
http://www.suse.de/~aj/linux_lfs.html

I'm not sure if the Cog VMs are being compiled with the the LFS option,
although
I expect that if you compile it yourself with the build options that Bert
explains,
then it should start working.

I just committed the necessary changes for my branch.  Will rebuild soon.
  G?ran, if you're in a hurry build your own?  You'd need to be in
http://www.squeakvm.org/svn/squeak/branches/Cog/unixbuild/bld and run ./mvm.




Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Cog issue? Re: StandardFileStream size limit?

Thierry Goubier


2015-11-13 16:24 GMT+01:00 Nicolai Hess <[hidden email]>:


2015-11-12 14:15 GMT+01:00 Nicolai Hess <[hidden email]>:


2015-11-12 14:10 GMT+01:00 Dimitris Chloupis <[hidden email]>:
latest vm has issues with OSProcess stdout the last time I checked on Yosemite. I would not call it stable unless we talk only crashes here.

Ah, I didn't know this was a vm issue. Does this means it is working with the "official" stable vm ?

Can you check if OSProcess is working with the stable vm? This may help to fix this issue.

OSProcess works on the stable vm on Mac OS X, and on the stable and on the latest vm on Linux.

Thierry
 

Thanks
nicolai

 
 

On Thu, Nov 12, 2015 at 2:55 PM Nicolai Hess <[hidden email]> wrote:
2015-11-12 6:14 GMT+01:00 Jimmie Houchin <[hidden email]>:
This still fails three and a half years later with a default Pharo 4 vm and image.
:(

I spent a few hours trying to diagnose why Pharo was telling me that I file I knew existed didn't exist. Very frustrating trying to debug this and then to find it is a vm and not an image issue. I then to search the archives to find this is a known issue for this long and still broken.

The latest linux vm works fine.
http://files.pharo.org/vm/pharo/linux/Pharo-VM-linux-latest.zip

Yes, the latest vm is pretty stable. But I think the problem is, there are to many changes for to many
different projects:
- I added a change to the windows keyevent handling and created a small  regression (can not use ctrl+space anymore)
- changes for OSWindow /SDL
- spur (?) or any other changes from the main squeak vm
- ...

I think we should have create release candidate branch and make sure more people testing that version and all
newer changes go to the master/development branch only.

nicolai
 

Is there a reason why the "latest" vm isn't the stable vm? Or at least this bugfix applied to the stable vm?

Thanks.

Jimmie



On 06/13/2012 05:00 PM, eliot.miranda at gmail.com (Eliot Miranda) wrote:
On Wed, Jun 13, 2012 at 7:40 AM, David T. Lewis <lewis at mail.msen.com> wrote:

On Wed, Jun 13, 2012 at 04:08:14PM +0200, G?ran Krampe wrote:
Hey!

Ok, so the plot thickens:

If I run this in a "oneclick 1.4" I will get a file write error on 2Gb:

| f b |
f := StandardFileStream newFileNamed: 'test'.
b := ByteArray new: 1024*1024*100. "100Mb"
[30 timesRepeat: [f nextPutAll: b]] ensure: [f close] "3Gb"


...but it worked fine using a self built "standard" VM from squeakvm.org
(4.4.7.2357)! Same image btw.

Now, does it work with the bleeding edge Cog? Tried r2556 and nope, same
problem.

So I "guess" it is a Cog thing? I haven't tried building Cog from source.

regards, G?ran
Background on large file support, see especially Bert's summary:


http://forum.world.st/Re-squeak-dev-filesize-reporting-0-for-very-large-files-td4483646.html
http://bugs.squeak.org/view.php?id=7522
http://www.suse.de/~aj/linux_lfs.html

I'm not sure if the Cog VMs are being compiled with the the LFS option,
although
I expect that if you compile it yourself with the build options that Bert
explains,
then it should start working.

I just committed the necessary changes for my branch.  Will rebuild soon.
  G?ran, if you're in a hurry build your own?  You'd need to be in
http://www.squeakvm.org/svn/squeak/branches/Cog/unixbuild/bld and run ./mvm.