Smalltalk › Squeak › Squeak - Dev

FileStream size limitation?

‹ Previous Topic Next Topic ›
 
classic Classic list List threaded Threaded
♦
♦
 7 messages Options
Squeak List
Reply | Threaded
Open this post in threaded view
♦
♦
|

FileStream size limitation?

Squeak List
Hello,

I have an OrderedCollection with 24,576 elements.

A bunch of musical notes basically: each of the 24,576 elements is like:
#('eo6' 0.25 500)
for example.

Goal: I would like to turn that *entire* OrderedCollection into a single MIDI file.

However, there seems to be some sort of size limitation that I do not see how to get around.

1) Even before getting to the MIDI part, I tried:

FileStream fileNamed: '1-24576.rtf'
do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:24576) asString)].

which creates a nice file of 48.8KB size.
And opening the file, it ends with: 
 #('c#o6' 0.25 500) #('d...etc...
which is not what I expected.

2) If I do:

FileStream fileNamed: '1-2729.rtf'
do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2729) asString)].

it works fine: ending tidily with the expected: 
#('c#o6' 0.25 500))

3) But,

FileStream fileNamed: '1-2730.rtf'
do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2730) asString)].

is where this starts:
 #('c#o6' 0.25 500) #('d...etc...

This exact same problem happens when making MIDI files: there seems to be a size limitation around 48.8KB.

Aside from MIDI, is FileStream just the wrong class to use for anything larger than creating a 48.8kb text file (.txt or .rtf)? Or how can I get around this restriction?

Any suggestions are most welcome.

Thanks,
ken

Windows 7 - 64 bit

Squeak4.4
latest update: #12319
Current Change Set: Unnamed
Image format 6505 (32 bit)

Squeak.exe
Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.331]
Win32 built on Aug 22 2013 10:20:54 Compiler: 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)
platform sources revision VM: r2776



Hannes Hirzel
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: FileStream size limitation?

Hannes Hirzel
Hello

      http://stackoverflow.com/questions/tagged/squeak

You may post it there as well without disclosing your name. It is
easier to work out the solution with you.


-- Hannes

On 8/31/13, Squeak List <[hidden email]> wrote:

> Hello,
>
> I have an OrderedCollection with 24,576 elements.
>
> A bunch of musical notes basically: each of the 24,576 elements is like:
>
> #('eo6' 0.25 500)
>
> for example.
>
>
> Goal: I would like to turn that *entire* OrderedCollection into a single
> MIDI file.
>
>
> However, there seems to be some sort of size limitation that I do not see
> how to get around.
>
> 1) Even before getting to the MIDI part, I tried:
>
> FileStream fileNamed: '1-24576.rtf'
> do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:24576) asString)].
>
> which creates a nice file of 48.8KB size.
> And opening the file, it ends with:
>
> #('c#o6' 0.25 500) #('d...etc...
> which is not what I expected.
>
> 2) If I do:
>
> FileStream fileNamed: '1-2729.rtf'
> do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2729) asString)].
>
> it works fine: ending tidily with the expected:
> #('c#o6' 0.25 500))
>
> 3) But,
>
> FileStream fileNamed: '1-2730.rtf'
> do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2730) asString)].
>
> is where this starts:
>
> #('c#o6' 0.25 500) #('d...etc...
>
>
> This exact same problem happens when making MIDI files: there seems to be a
> size limitation around 48.8KB.
>
> Aside from MIDI, is FileStream just the wrong class to use for anything
> larger than creating a 48.8kb text file (.txt or .rtf)? Or how can I get
> around this restriction?
>
> Any suggestions are most welcome.
>
>
> Thanks,
> ken
>
> Windows 7 - 64 bit
>
>
> Squeak4.4
> latest update: #12319
> Current Change Set: Unnamed
> Image format 6505 (32 bit)
>
> Squeak.exe
> Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.331]
> Win32 built on Aug 22 2013 10:20:54 Compiler: 3.4.4 (cygming special, gdc
> 0.12, using dmd 0.125)
> platform sources revision VM: r2776

Bob Arning-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: FileStream size limitation?

Bob Arning-2
In reply to this post by Squeak List
The "...etc..." part is a limitation in the implementation of #printString, which get called by #asString here. The goal is *not* to put huge amounts of data into the Transcript or other simple UI elements where it might be unwieldy. If you really want the full thing printed out, use

myBigOrderedCollection printStringLimitedTo: 99999999

or

myBigOrderedCollection fullPrintString

Cheers,
Bob
On 8/31/13 4:49 AM, Squeak List wrote:
Hello,

I have an OrderedCollection with 24,576 elements.

A bunch of musical notes basically: each of the 24,576 elements is like:
#('eo6' 0.25 500)
for example.

Goal: I would like to turn that *entire* OrderedCollection into a single MIDI file.

However, there seems to be some sort of size limitation that I do not see how to get around.

1) Even before getting to the MIDI part, I tried:

FileStream fileNamed: '1-24576.rtf'
do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:24576) asString)].

which creates a nice file of 48.8KB size.
And opening the file, it ends with: 
 #('c#o6' 0.25 500) #('d...etc...
which is not what I expected.

2) If I do:

FileStream fileNamed: '1-2729.rtf'
do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2729) asString)].

it works fine: ending tidily with the expected: 
#('c#o6' 0.25 500))

3) But,

FileStream fileNamed: '1-2730.rtf'
do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2730) asString)].

is where this starts:
 #('c#o6' 0.25 500) #('d...etc...

This exact same problem happens when making MIDI files: there seems to be a size limitation around 48.8KB.

Aside from MIDI, is FileStream just the wrong class to use for anything larger than creating a 48.8kb text file (.txt or .rtf)? Or how can I get around this restriction?

Any suggestions are most welcome.

Thanks,
ken

Windows 7 - 64 bit

Squeak4.4
latest update: #12319
Current Change Set: Unnamed
Image format 6505 (32 bit)

Squeak.exe
Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.331]
Win32 built on Aug 22 2013 10:20:54 Compiler: 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)
platform sources revision VM: r2776 




    

    

  





	
	
	
	

						
			

		

	

	

		

			

				

					Squeak List
				

				

					Reply |
					Threaded
	

		Open this post in threaded view
		

			
♦

			
♦

		

	

	 |
					
	
				

				

					
					
		
	
					

		Re: FileStream size limitation? SOLVED
	

				

			

		

		

			

				
					

		

			Squeak List

	
	
		

	

	
	
	

	
						
				
					
	
	

		
Thanks Bob!

fullPrintString  was the solution. 

MIDI files working full size also
 :)

TYVM
ken

  
 
 
 
   From: Bob Arning <[hidden email]>
 To: [hidden email]
 
 Sent: Saturday, August 31, 2013 2:44 AM
 Subject: Re: [squeak-dev] FileStream size limitation?
  
 




  

    
  
  

    The "...etc..." part is a limitation in the
      implementation of #printString, which get called by #asString here.
      The goal is *not* to put huge amounts of data into the Transcript
      or other simple UI elements where it might be unwieldy. If you
      really want the full thing printed out, use

      

      myBigOrderedCollection printStringLimitedTo: 99999999

      

      or

    

    myBigOrderedCollection fullPrintString

        

      Cheers,

      Bob

      

    
    
On 8/31/13 4:49 AM, Squeak List wrote:

    

    

      

        
Hello,

        


        

        
I have an OrderedCollection with 24,576
          elements.

        
 

        

        
A bunch of musical notes basically: each
          of the 24,576 elements is like: 

        

        
#('eo6' 0.25 500)

        

        
for example.

        

        


        

        
Goal: I would like to turn that *entire*
          OrderedCollection into a single MIDI file. 

        

        


        

        
However, there seems to be some sort of
          size limitation that I do not see how to get around.

        


        

        
1) Even before getting to the MIDI part,
          I tried:

        


        

        
FileStream fileNamed: '1-24576.rtf'

          do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:24576)
          asString)].

        


        

        
which creates a nice file of 48.8KB size.

        
And opening the file, it ends with: 

        

        

        

        
 #('c#o6' 0.25 500) #('d...etc...

        
which is not what I expected.

        


        

        
2) If I do:

        


        

        
FileStream fileNamed: '1-2729.rtf'

          do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2729)
          asString)].

        


        

        
it works fine: ending tidily with the
          expected: 

        
#('c#o6' 0.25 500))

        


        

        
3) But,

        


        

        
FileStream fileNamed: '1-2730.rtf'

          do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2730)
          asString)].

        


        

        
is where this starts:

        

        

        

        
 #('c#o6' 0.25 500) #('d...etc...


        


        

        
This exact same problem happens when
          making MIDI files: there seems to be a size limitation around
          48.8KB.

        


        

        
Aside from MIDI, is FileStream just the
          wrong class to use for anything larger than creating a 48.8kb
          text file (.txt or .rtf)? Or how can I get around this
          restriction?

        


        

        
Any suggestions are most welcome. 

        

        


        

        
Thanks,

        
ken

        


        

        
Windows 7 - 64 bit

        

        


        

        
Squeak4.4

          latest update: #12319

          Current Change Set: Unnamed

          Image format 6505 (32 bit)

        


        

        
Squeak.exe

          Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.331]

          Win32 built on Aug 22 2013 10:20:54 Compiler: 3.4.4 (cygming
          special, gdc 0.12, using dmd 0.125)

          platform sources revision VM: r2776 

        

        


        

      

      

      

      

      

    

    

  







 
 
  




	
	
	
	

						
			

		

	

	

		

			

				

					Benoit St-Jean
				

				

					Reply |
					Threaded
	

		Open this post in threaded view
		

			
♦

			
♦

		

	

	 |
					
	
				

				

					
					
		
	
					

		Re: FileStream size limitation?
	

				

			

		

		

			

				
					

		

			Benoit St-Jean

	
	
		

	

	
	
	

	
						
				
					
	

				In reply to this post by Squeak List
			

	

		
You could have gone the other way around!  Like this:

| fs |
fs := FileStream fileNamed: '1-2729.rtf'.
oc do: [:each | fs nextPutAll: each asString].
fs close.


That way, regardless of the size of your collection, it will always work!


 
-----------------
Benoit St-Jean
Yahoo! Messenger: bstjean
Blogue: endormitoire.wordpress.com
A standpoint is an intellectual horizon of radius zero.
(Albert Einstein)

  
 
 
 
   From: Squeak List <[hidden email]>
 To: The general-purpose Squeak developers list <[hidden email]> 
 Sent: Saturday, August 31, 2013 4:49:02 AM
 Subject: [squeak-dev] FileStream size limitation?
  
 

Hello,

I have an OrderedCollection with 24,576 elements.
 
A bunch of musical notes basically: each of the 24,576 elements is like: 
#('eo6' 0.25 500)
for example.

Goal: I would like to turn that *entire* OrderedCollection into a single MIDI file. 

However, there seems to be some sort of size limitation that I do not see how to get around.

1) Even before getting to the MIDI part, I tried:

FileStream fileNamed: '1-24576.rtf'
do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:24576) asString)].

which creates a nice file of 48.8KB size.
And opening the file, it ends with: 


 #('c#o6' 0.25 500) #('d...etc...


which is not what I expected.

2) If I do:

FileStream fileNamed: '1-2729.rtf'
do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2729) asString)].

it works fine: ending tidily with the expected: 

#('c#o6' 0.25 500))



3) But,

FileStream fileNamed: '1-2730.rtf'
do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2730) asString)].

is where this starts:


 #('c#o6' 0.25 500) #('d...etc...



This exact same problem happens when making MIDI files: there seems to be a size limitation around 48.8KB.

Aside from MIDI, is FileStream just the wrong class to use for anything larger than creating a 48.8kb text file (.txt or .rtf)? Or how can I get around this restriction?

Any suggestions are most welcome. 

Thanks,
ken

Windows 7 - 64 bit

Squeak4.4
latest update: #12319
Current Change Set: Unnamed
Image format 6505 (32 bit)

Squeak.exe
Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.331]
Win32 built on Aug 22 2013 10:20:54 Compiler: 3.4.4 (cygming special, gdc 0.12, using dmd
 0.125)
platform sources revision VM: r2776 





 
 
  




	
	
	
	

						
			

		

	

	

		

			

				

					Levente Uzonyi-2
				

				

					Reply |
					Threaded
	

		Open this post in threaded view
		

			
♦

			
♦

		

	

	 |
					
	
				

				

					
					
		
	
					

		Re: FileStream size limitation?
	

				

			

		

		

			

				
					

		

			Levente Uzonyi-2

	
	
		

	

	
	
	

	
						
				
					
	
	

		On Sat, 31 Aug 2013, Benoit St-Jean wrote:


> You could have gone the other way around!  Like this:

> 

> | fs |

> fs := FileStream fileNamed: '1-2729.rtf'.

> oc do: [:each | fs nextPutAll: each asString].

> fs close.

> 

> 

> That way, regardless of the size of your collection, it will always work!


There are two reasons why this is not the best way to do it:

1) #asString might cause truncation, which was the original problem. If 

one element's #printString (sent by #asString) were longer than 50000 

characters, then it would be truncated. The proper solution is to use 

#printOn: (but see below why it's not the best idea either).

2) FileStreams are not write buffered. Each write (#nextPutAll:) will 

call a primitive, which can be rather slow.



Levente


> 

> 

>  

> -----------------

> Benoit St-Jean

> Yahoo! Messenger: bstjean

> Blogue: endormitoire.wordpress.com

> A standpoint is an intellectual horizon of radius zero.

> (Albert Einstein)

> 

> __________________________________________________________________________________________________________________________________________________

> From: Squeak List <[hidden email]>

> To: The general-purpose Squeak developers list <[hidden email]>

> Sent: Saturday, August 31, 2013 4:49:02 AM

> Subject: [squeak-dev] FileStream size limitation?

> 

> Hello,

> 

> I have an OrderedCollection with 24,576 elements.

> 

> A bunch of musical notes basically: each of the 24,576 elements is like:

> #('eo6' 0.25 500)

> for example.

> 

> Goal: I would like to turn that *entire* OrderedCollection into a single MIDI file.

> 

> However, there seems to be some sort of size limitation that I do not see how to get around.

> 

> 1) Even before getting to the MIDI part, I tried:

> 

> FileStream fileNamed: '1-24576.rtf'

> do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:24576) asString)].

> 

> which creates a nice file of 48.8KB size.

> And opening the file, it ends with:

>

>  #('c#o6' 0.25 500) #('d...etc...

> which is not what I expected.

> 

> 2) If I do:

> 

> FileStream fileNamed: '1-2729.rtf'

> do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2729) asString)].

> 

> it works fine: ending tidily with the expected:

> 

> #('c#o6' 0.25 500))

> 

> 3) But,

> 

> FileStream fileNamed: '1-2730.rtf'

> do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2730) asString)].

> 

> is where this starts:

>

>  #('c#o6' 0.25 500) #('d...etc...

> 

> This exact same problem happens when making MIDI files: there seems to be a size limitation around 48.8KB.

> 

> Aside from MIDI, is FileStream just the wrong class to use for anything larger than creating a 48.8kb text file (.txt or .rtf)? Or how can I get

> around this restriction?

> 

> Any suggestions are most welcome.

> 

> Thanks,

> ken

> 

> Windows 7 - 64 bit

> 

> Squeak4.4

> latest update: #12319

> Current Change Set: Unnamed

> Image format 6505 (32 bit)

> 

> Squeak.exe

> Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.331]

> Win32 built on Aug 22 2013 10:20:54 Compiler: 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)

> platform sources revision VM: r2776

> 

> 

> 

> 

> 

> 

>



	
	
	
	

						
			

		

	

	

		

			

				

					Benoit St-Jean
				

				

					Reply |
					Threaded
	

		Open this post in threaded view
		

			
♦

			
♦

		

	

	 |
					
	
				

				

					
					
		
	
					

		Re: FileStream size limitation?
	

				

			

		

		

			

				
					

		

			Benoit St-Jean

	
	
		

	

	
	
	

	
						
				
					
	
	

		
Well,

1) The #asString is not necessary since, from what I understood, the elements of the OrderedCollection are Strings already!  Besides, if that is not the case, the items in that collection are limited to less than 20 character strings so there wouldn't be any truncation problem there!
2) Since it was a "one time" conversion (and not something called repeatedly (if I understood correctly), being somewhat slow ain't a major concern here.

 
-----------------
Benoit St-Jean
Yahoo! Messenger: bstjean
Blogue: endormitoire.wordpress.com
A standpoint is an intellectual horizon of radius zero.
(Albert Einstein)

  
 
 
 
   From: Levente Uzonyi <[hidden email]>
 To: Benoit St-Jean <[hidden email]>; The general-purpose Squeak developers list <[hidden email]> 
Cc: Squeak List <[hidden email]> 
 Sent: Sunday, September 1, 2013 8:39:41 PM
 Subject: Re: [squeak-dev] FileStream size limitation?
  
 

On Sat, 31 Aug 2013, Benoit St-Jean wrote:

> You could have gone the other way around!  Like this:
> 
> | fs |
> fs := FileStream fileNamed: '1-2729.rtf'.
> oc do: [:each | fs nextPutAll: each asString].
> fs
 close.
> 
> 
> That way, regardless of the size of your collection, it will always work!

There are two reasons why this is not the best way to do it:
1) #asString might cause truncation, which was the original problem. If 
one element's #printString (sent by #asString) were longer than 50000 
characters, then it would be truncated. The proper solution is to use 
#printOn: (but see below why it's not the best idea either).
2) FileStreams are not write buffered. Each write (#nextPutAll:) will 
call a primitive, which can be rather slow.


Levente

> 
> 
>  
> -----------------
> Benoit St-Jean
> Yahoo! Messenger: bstjean
> Blogue: endormitoire.wordpress.com
> A standpoint is an intellectual horizon of radius zero.
> (Albert Einstein)
> 
>
 __________________________________________________________________________________________________________________________________________________
> From: Squeak List <[hidden email]>
> To: The general-purpose Squeak developers list <[hidden email]>
> Sent: Saturday, August 31, 2013 4:49:02 AM
> Subject: [squeak-dev] FileStream size limitation?
> 
> Hello,
> 
> I have an OrderedCollection with 24,576 elements.
> 
> A bunch of musical notes basically: each of the 24,576 elements is like:
> #('eo6' 0.25 500)
> for example.
> 
> Goal: I would like to turn that *entire* OrderedCollection into a single MIDI file.
> 
> However, there
 seems to be some sort of size limitation that I do not see how to get around.
> 
> 1) Even before getting to the MIDI part, I tried:
> 
> FileStream fileNamed: '1-24576.rtf'
> do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:24576) asString)].
> 
> which creates a nice file of 48.8KB size.
> And opening the file, it ends with:
>
>  #('c#o6' 0.25 500) #('d...etc...
> which is not what I expected.
> 
> 2) If I do:
> 
> FileStream fileNamed: '1-2729.rtf'
> do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2729) asString)].
> 
> it works fine: ending tidily with the expected:
> 
> #('c#o6' 0.25 500))
> 
> 3) But,
> 
> FileStream fileNamed: '1-2730.rtf'
> do: [:stream | stream nextPutAll: ((oc copyFrom: 1 to:2730) asString)].
> 
> is where this starts:
>
>  #('c#o6'
 0.25 500) #('d...etc...
> 
> This exact same problem happens when making MIDI files: there seems to be a size limitation around 48.8KB.
> 
> Aside from MIDI, is FileStream just the wrong class to use for anything larger than creating a 48.8kb text file (.txt or .rtf)? Or how can I get
> around this restriction?
> 
> Any suggestions are most welcome.
> 
> Thanks,
> ken
> 
> Windows 7 - 64 bit
> 
> Squeak4.4
> latest update: #12319
> Current Change Set: Unnamed
> Image format 6505 (32 bit)
> 
> Squeak.exe
> Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.331]
> Win32 built on Aug 22 2013 10:20:54 Compiler: 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)
> platform sources revision VM: r2776
> 
> 
> 
> 
> 
> 
>

 
 
  




	
	
	
	

						
			

		

	

	
		
	

	

			
				
			

		

			
				Free forum by Nabble
			
			
				
				Edit this page