CompiledMethodTrailers ready for use

> Hello,
> i finished this stuff, and its ready for adoption.
>
> See http://bugs.squeak.org/view.php?id=7428
>
> Anyone wants to help pushing it into trunk update stream (using MC configs)?
>
> It works fine on recent trunk image,
> on pharo however i had some problems installing changes, because of
> some differencies.
>
> Tried on PharoCore-1.1-11106-ALPHA.image
>
> phase2.1.cs
> - do not filein the TextEditor changes, since pharo-core don't have it.
> - do not filein the last line (reorganizing)..
>
> - tests failing because pharo String class does not implements
> #squeakToUtf8
> nor
> #utf8ToSqueak
>
> Do we having an uniform way how to encode  ANY String -> ByteString(utf8)
> and back? What ANSI standard saying about it? Maybe i'm using wrong methods?
>

> Hello,
> i finished this stuff, and its ready for adoption.
>
> See http://bugs.squeak.org/view.php?id=7428
>
> Anyone wants to help pushing it into trunk update stream (using MC configs)?
>
> It works fine on recent trunk image,
> on pharo however i had some problems installing changes, because of
> some differencies.
>
> Tried on PharoCore-1.1-11106-ALPHA.image
>
> phase2.1.cs
> - do not filein the TextEditor changes, since pharo-core don't have it.
> - do not filein the last line (reorganizing)..
>
> - tests failing because pharo String class does not implements
> #squeakToUtf8
> nor
> #utf8ToSqueak
>
> Do we having an uniform way how to encode  ANY String -> ByteString(utf8)
> and back? What ANSI standard saying about it? Maybe i'm using wrong methods?
>
> Still, i think we need this thing standartized and be common for all
> dialects (not just Pharo/Squeak).
>

> Cheers,
>  - Andreas
>
> Igor Stasenko wrote:
>>
>> Hello,
>> i finished this stuff, and its ready for adoption.
>>
>> See http://bugs.squeak.org/view.php?id=7428
>>
>> Anyone wants to help pushing it into trunk update stream (using MC
>> configs)?
>>
>> It works fine on recent trunk image,
>> on pharo however i had some problems installing changes, because of
>> some differencies.
>>
>> Tried on PharoCore-1.1-11106-ALPHA.image
>>
>> phase2.1.cs
>> - do not filein the TextEditor changes, since pharo-core don't have it.
>> - do not filein the last line (reorganizing)..
>>
>> - tests failing because pharo String class does not implements
>> #squeakToUtf8
>> nor
>> #utf8ToSqueak
>>
>> Do we having an uniform way how to encode  ANY String -> ByteString(utf8)
>> and back? What ANSI standard saying about it? Maybe i'm using wrong
>> methods?
>>
>> Still, i think we need this thing standartized and be common for all
>> dialects (not just Pharo/Squeak).
>>
>
>
>

> On Tue, Dec 22, 2009 at 01:59:34PM +0100, Andreas Raab wrote:
>> Folks -
>>
>> We're all clear now after the CompiledMethodTrailer changes.
>> Can I have a 2GB changes file now? And a pony? :)
>>
>
> Here's the 2GB changes file patch. File it in, run tests, install
> with ExpandedSourceFileArray install.
>

> You will want to disable the changes size check in #checkOKToAdd:at:
> and also arrange for the ExpandedSourceFileArray to be installed
> at image startup.
>
> I was able to grow a changes file beyond 32M, and verified that
> the image has large sourcePointer values beyond the range previously
> allowed.
>
>  (CompiledMethod allInstances collect: [:e | e sourcePointer]) max hex ==> '6054DAA'
>
> I'm not so sure about the pony, but Randal promised to send one ;)
>
> Cheers,
> Dave
>
>
>
>
>

> 2009/12/23 David T. Lewis <[hidden email]>:
>> On Tue, Dec 22, 2009 at 01:59:34PM +0100, Andreas Raab wrote:
>>> Folks -
>>>
>>> We're all clear now after the CompiledMethodTrailer changes.
>>> Can I have a 2GB changes file now? And a pony? :)
>>>
>>
>> Here's the 2GB changes file patch. File it in, run tests, install
>> with ExpandedSourceFileArray install.
>>
>
> so, we're now bound to VM file-size handling limitation, or its using
> 64-bits for file pointers?
> 2GB is better than 32M.. but still small :)
>
>> You will want to disable the changes size check in #checkOKToAdd:at:
>> and also arrange for the ExpandedSourceFileArray to be installed
>> at image startup.
>>
>> I was able to grow a changes file beyond 32M, and verified that
>> the image has large sourcePointer values beyond the range previously
>> allowed.
>>
>>  (CompiledMethod allInstances collect: [:e | e sourcePointer]) max hex ==>
>> '6054DAA'
>>
>> I'm not so sure about the pony, but Randal promised to send one ;)
>>
>> Cheers,
>> Dave
>>
>>
>>
>>
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>
>

> On 12/23/09, Igor Stasenko <[hidden email]> wrote:
>> 2009/12/23 David T. Lewis <[hidden email]>:
>>> On Tue, Dec 22, 2009 at 01:59:34PM +0100, Andreas Raab wrote:
>>>> Folks -
>>>>
>>>> We're all clear now after the CompiledMethodTrailer changes.
>>>> Can I have a 2GB changes file now? And a pony? :)
>>>>
>>>
>>> Here's the 2GB changes file patch. File it in, run tests, install
>>> with ExpandedSourceFileArray install.
>>>
>>
>> so, we're now bound to VM file-size handling limitation, or its using
>> 64-bits for file pointers?
>> 2GB is better than 32M.. but still small :)
>>
>>> You will want to disable the changes size check in #checkOKToAdd:at:
>>> and also arrange for the ExpandedSourceFileArray to be installed
>>> at image startup.
>>>
>>> I was able to grow a changes file beyond 32M, and verified that
>>> the image has large sourcePointer values beyond the range previously
>>> allowed.
>>>
>>>  (CompiledMethod allInstances collect: [:e | e sourcePointer]) max hex ==>
>>> '6054DAA'
>>>
>>> I'm not so sure about the pony, but Randal promised to send one ;)
>>>
>>> Cheers,
>>> Dave
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>>
>>
>
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>

> 2009/12/23 David T. Lewis <[hidden email]>:
> > On Tue, Dec 22, 2009 at 01:59:34PM +0100, Andreas Raab wrote:
> >> Folks -
> >>
> >> We're all clear now after the CompiledMethodTrailer changes.
> >> Can I have a 2GB changes file now? And a pony? :)
> >>
> >
> > Here's the 2GB changes file patch. File it in, run tests, install
> > with ExpandedSourceFileArray install.
> >
>
> so, we're now bound to VM file-size handling limitation, or its using
> 64-bits for file pointers?
> 2GB is better than 32M.. but still small :)
>

> On Wed, Dec 23, 2009 at 10:27:42AM -0800, Eliot Miranda wrote:
>> On Wed, Dec 23, 2009 at 6:43 AM, David T. Lewis <[hidden email]> wrote:
>>
>> > On Wed, Dec 23, 2009 at 10:27:29AM +0200, Igor Stasenko wrote:
>> > > 2009/12/23 David T. Lewis <[hidden email]>:
>> > > > On Tue, Dec 22, 2009 at 01:59:34PM +0100, Andreas Raab wrote:
>> > > >> Folks -
>> > > >>
>> > > >> We're all clear now after the CompiledMethodTrailer changes.
>> > > >> Can I have a 2GB changes file now? And a pony? :)
>> > > >>
>> > > >
>> > > > Here's the 2GB changes file patch. File it in, run tests, install
>> > > > with ExpandedSourceFileArray install.
>> > > >
>> > >
>> > > so, we're now bound to VM file-size handling limitation, or its using
>> > > 64-bits for file pointers?
>> > > 2GB is better than 32M.. but still small :)
>> > >
>> >
>> > Now that CompiledMethod can address unlimited address space, all
>> > that was required was to is to implement three small methods in
>> > a subclass of SourceFileArray (*). So if somebody invents a file
>> > system that maps to all the particles in the universe, we can
>> > easily expand the changes file to make use if it. And as you point
>> > out in another message, it should now be easier to map to things
>> > that are not traditional file systems at all, which does open up
>> > some interesting possibilities.
>> >
>>
>> I'm not up-to-date with the changes.  Are you making sure that the new
>> source manager includes read-only copies of the source files so that
>> fetching source does not involve creating a new file?
>
> The class that I posted affects only the address mapping and would
> not affect any other aspects of source file mapping.
>
> That said, performance is poor so this is not ready for prime time.
> I'm attaching an update that is somewhat less bad, but still nowhere
> near good enough (3 or 4 times slower than StandardSourceFileArray).
>

> On Wed, Dec 23, 2009 at 11:07:54PM +0100, Levente Uzonyi wrote:
>> On Wed, 23 Dec 2009, David T. Lewis wrote:
>>>
>>> That said, performance is poor so this is not ready for prime time.
>>> I'm attaching an update that is somewhat less bad, but still nowhere
>>> near good enough (3 or 4 times slower than StandardSourceFileArray).
>>>
>>
>> Which part is 3-4 times slower? Can you show us your benchmark?
>>
>
> I'm glad you asked. There was a typo in my test, and performance
> seems to be pretty good after all. Here is what I was checking:
>
>  Time millisecondsToRun: [
> sf := StandardSourceFileArray new.
> (16r1000000 to: 16r1FFFFFF by: 11) do: [:e |
> i := sf fileIndexFromSourcePointer: e.
> p := sf filePositionFromSourcePointer: e.
> a := sf sourcePointerFromFileIndex: i andPosition: p.
> a = e ifFalse: [self error: 'snafu']]
> ] ==> 3391
>
>  Time millisecondsToRun: [
> sf := ExpandedSourceFileArray new.
> (16r1000000 to: 16r1FFFFFF by: 11) do: [:e |
> i := sf fileIndexFromSourcePointer: e.
> p := sf filePositionFromSourcePointer: e.
> a := sf sourcePointerFromFileIndex: i andPosition: p.
> a = e ifFalse: [self error: 'snafu']]
> ] ==> 3220
>

> On Wed, 23 Dec 2009, David T. Lewis wrote:
>
> >On Wed, Dec 23, 2009 at 11:07:54PM +0100, Levente Uzonyi wrote:
> >>On Wed, 23 Dec 2009, David T. Lewis wrote:
> >>>
> >>>That said, performance is poor so this is not ready for prime time.
> >>>I'm attaching an update that is somewhat less bad, but still nowhere
> >>>near good enough (3 or 4 times slower than StandardSourceFileArray).
> >>>
> >>
> >>Which part is 3-4 times slower? Can you show us your benchmark?
> >>
> >
> >I'm glad you asked. There was a typo in my test, and performance
> >seems to be pretty good after all. Here is what I was checking:
> >
> > Time millisecondsToRun: [
> > sf := StandardSourceFileArray new.
> > (16r1000000 to: 16r1FFFFFF by: 11) do: [:e |
> > i := sf fileIndexFromSourcePointer: e.
> > p := sf filePositionFromSourcePointer: e.
> > a := sf sourcePointerFromFileIndex: i andPosition: p.
> > a = e ifFalse: [self error: 'snafu']]
> > ] ==> 3391
> >
> > Time millisecondsToRun: [
> > sf := ExpandedSourceFileArray new.
> > (16r1000000 to: 16r1FFFFFF by: 11) do: [:e |
> > i := sf fileIndexFromSourcePointer: e.
> > p := sf filePositionFromSourcePointer: e.
> > a := sf sourcePointerFromFileIndex: i andPosition: p.
> > a = e ifFalse: [self error: 'snafu']]
> > ] ==> 3220
> >
>
> My slightly modified (but semantically the same) version:
>
> { StandardSourceFileArray. ExpandedSourceFileArray } collect: [ :class |
> [
> | sf |
> sf := class new.
> 16r1000000 to: 16r1FFFFFF by: 11 do: [:e |
> | i p a |
> i := sf fileIndexFromSourcePointer: e.
> p := sf filePositionFromSourcePointer: e.
> a := sf sourcePointerFromFileIndex: i andPosition: p.
> a = e ifFalse: [self error: 'snafu' ] ] ] timeToRun ]
>
> gives #(766 682) on my pc.
>
> So I think the performance is "solved". :)

> On Wed, Dec 23, 2009 at 01:19:43AM -0800, John McIntosh wrote:
>> It should be 64bit since I made that change years back. However most
>> (all?) file systems don't support all 64 bit of space. Still it was
>> intersting to learn I could make terabyte sparse file on my freebsd
>> system.
>
> Let's see (counting on fingers) I think that a 64 bit address
> space is about a trillion times bigger than the current changes
> files limit, so that should be sufficient for the next few years.
>
>>
>> So is it 2gb or 4gb? Is that a limit due to 32bit signed numbers?
>
> Various FAT and Linux file systems are limited to 2GB. Very likely
> it's due to the use of signed ints for addressing. I'm just guessing
> here, but if you use a signed int, you can easily tell if the size
> exceeds 2GB by testing for file pointer less than zero. Besides,
> who would ever dream of trying to address more than 2GB of space
> for a single file? I mean, how may floppy disks is that, anyway?
>
> If we ever get to the point where this matters for a Squeak changes
> file, then it will absolutely, positively be time for someone to
> burn the disk packs. It will be a very large bonfire, and we can
> all roast marshmellows :)
>
> Dave
>
>
>