New source code subsystem

> Let's be more pragmatic and have a small incremental bottom up approach.
>
> I see two solutions:
> - store source code and history in image... I do not believe we should  
> go in this direction even in compressed forms. We are rather making  
> efforts to make images shorter... But you can have other opinion of  
> course.
> - continue storing source in external repositories (files, URL, NET  
> servers...)
>
> In the later case, we could have many files instead of two (sources  
> changes) and we should be able to have large files: conclusion: let's  
> just change source code pointer encoding.

> Using SmallIntegers (30 bits?) we can address 1GByte of space. More if  
> bits are not encoded as file offset but as Oop on a server...
>
> Using two SmallInteger, we have at least 60 bits, 1 ExaByte of data!  
> Let's go for it (count how many compiledMethod are in your image... and  
> you'll see this is not a drama).
>
> Maybe we can have an encoding supporting 1 single SmallInteger for  
> certain patterns (for example it can be sufficient for code stored in  
> 'sources' file, 'sources' file could also be organised with an  
> indirection table instead of using byte offsets...Given average source  
> code length, that can save a factor between 10 and 100 i presume).
>
> We should be able to extend source code being in several places in the  
> future... (several files or URL or ...). So let use change encoding by  
> splitting bits : 20 bits for sourceRegistry offset, 40 bits for file  
> offset, or 30bits/30bits, or both (pattern starting with bits 0001 will  
> be 20/40, all other 30/30 for example).
>
> Then we have to create the SourceRegistry with objects able to handle  
> source code stored in files, in compressed files, on the Net, with all  
> necessary caching to reduce number of connections and design of the  
> source code servers...

> List,
>
> I put together what I think is relevant for a new source code  
> subsystem and, if I'm mistaken or have overlooked one of your  
> recent suggestions and/or requirements, please do not hesitate and  
> say so.
>
> What follows is not intended as a replacement for MC or SM, rather  
> the new subsystem should be as independent as possible. If you load  
> something with MC, the new subsystem can treat it like any other  
> contribution (more about contributions later).
>
> Trying to keep preliminaries as short as possible, we have as  
> organizational elements (for the sake of discussion): system  
> categories, classes, message categories and methods. The unit of  
> source code "entity" we consider to be a unit of change belongs to  
> one of these elements. We can add, replace, rename and delete. We  
> have not only methods, but also DoIt's (definitions,  
> organizational, etc) to take into account. And let's say that the  
> browser is our tool, just to keep it simple here.

> -------------------------------
>
> At the time an image is distributed, it is supposed to be in sync  
> (not necessarily *full* sync) with its respective source code  
> subsystem. So when you open a browser, all entities are colored  
> black on white, and my story begins.
>
> When you want to *observe* a certain area of the source code  
> subsytem, you put the respective entity (item from here on) onto  
> your watchlist (cmd+ if you like) from where the item gets the  
> color blue. After putting a class onto your watchlist you get all  
> its message categories and methods blue; un-watch those (sub)items  
> which are of no interest to you; etc for other organizational levels.
>
> When you do a *change* to a blue colored item, its new source is  
> contributed to the source code subsystem (from here on scs for  
> short) and the item is colored green in your browser. If later  
> somebody else contributes a change to the same blue item, it gets  
> the color yellow in both browsers.
>
> And if an entity you have in your image cannot be *matched* with  
> the scs, it is colored red.
>
> As is tradition, if you don't save your image before the next  
> crash, all this coloring information is lost, and you are back to  
> black and white. The scs does NOT know the state of your image.
>
> -------------------------------
>
> A contribution to the scs either has a predecessor or else it is  
> new (to the scs). Let's say three developers make contributions to  
> the same blue item, then we arrive at three branches (and the three  
> browsers color the respective item yellow). It is up to you to  
> accept one of these branches, but you are no forced to do so. An  
> accepted branch is colored green (it will turn yellow when the next  
> contribution relative to it arrives).
>
> -------------------------------
>
> By chasing the predecessor it is always possible to roll-back an  
> item (or a group of items), this is independent of the coloring I  
> used for illustration. The same "backwards" information is readily  
> available for analyzing the complete history of an item, including  
> all unaccepted branches.
>
> -------------------------------
>
> Doing this into the other direction (roll-forward) depends on which  
> branches where accepted. At any point in time, accepted branches  
> represent a "cut" in the scs.
>
> -------------------------------
>
> If you work alone on a project, you'll never see any yellow colored  
> item, unless you revert a change (because the dismissed becomes an  
> unaccepted branch). For people who are used to make small  
> incremental changes, a preference tells whether or not every change  
> is treated as a branch or else is to be consolidated until you do a  
> "big save" to the scs. The latter requires, I think, a local  
> buffering scs on your harddisk (or on your local area server).
>
> -------------------------------
>
> The amount of conversation between browser and scs is small, the  
> browser only has to ask for colors (one byte answer per item) for  
> those items it shows *and* which are on your watch list. But if you  
> want to see more (like, for example, all branches of an item), more  
> data will have to travel.
>
> -------------------------------
>
> It's now time for me to sit back and relax :-)
>
> /Klaus
>
>

> Klaus, i would say this is another source version control system.
>
> This bottom up approach is interesting but will not work. Squeakers  
> will continue to use their preferred tools, not your specific browser.
>
> Let's be more pragmatic and have a small incremental bottom up  
> approach.
>
> I see two solutions:
> - store source code and history in image... I do not believe we  
> should go in this direction even in compressed forms. We are rather  
> making efforts to make images shorter... But you can have other  
> opinion of course.
> - continue storing source in external repositories (files, URL,  
> NET  servers...)
>
> In the later case, we could have many files instead of two (sources  
> changes) and we should be able to have large files: conclusion:  
> let's just change source code pointer encoding.
>
> Using SmallIntegers (30 bits?) we can address 1GByte of space. More  
> if bits are not encoded as file offset but as Oop on a server...
>
> Using two SmallInteger, we have at least 60 bits, 1 ExaByte of  
> data! Let's go for it (count how many compiledMethod are in your  
> image... and you'll see this is not a drama).
>
> Maybe we can have an encoding supporting 1 single SmallInteger for  
> certain patterns (for example it can be sufficient for code stored  
> in 'sources' file, 'sources' file could also be organised with an  
> indirection table instead of using byte offsets...Given average  
> source code length, that can save a factor between 10 and 100 i  
> presume).
>
> We should be able to extend source code being in several places in  
> the future... (several files or URL or ...). So let use change  
> encoding by splitting bits : 20 bits for sourceRegistry offset, 40  
> bits for file offset, or 30bits/30bits, or both (pattern starting  
> with bits 0001 will be 20/40, all other 30/30 for example).
>
> Then we have to create the SourceRegistry with objects able to  
> handle source code stored in files, in compressed files, on the  
> Net, with all necessary caching to reduce number of connections and  
> design of the source code servers...
>
> However, i would keep the principle of copying changes in the  
> 'changes' file to recover easily in case of image crash, and  
> commiting to another source would still be the responsibility of  
> user, as is now.
>
> Small details: make sure the FileSystem primitives can deal with  
> huge offsets (and also underlying OS of course).
>
> You see, nothing really new, nothing really difficult.
> With just a little bit more reflexion we can go in this direction  
> and that should be enough for a few years of development and future  
> enhancements...
>
> Nicolas
>
> Klaus D. Witzel:
>> List,
>>
>> I put together what I think is relevant for a new source code  
>> subsystem
>> and, if I'm mistaken or have overlooked one of your recent  
>> suggestions
>> and/or requirements, please do not hesitate and say so.
>>
>> What follows is not intended as a replacement for MC or SM, rather  
>> the new
>> subsystem should be as independent as possible. If you load  
>> something with
>> MC, the new subsystem can treat it like any other contribution  
>> (more about
>> contributions later).
>>
>> Trying to keep preliminaries as short as possible, we have as
>> organizational elements (for the sake of discussion): system  
>> categories,
>> classes, message categories and methods. The unit of source code  
>> "entity"
>> we consider to be a unit of change belongs to one of these  
>> elements. We
>> can add, replace, rename and delete. We have not only methods, but  
>> also
>> DoIt's (definitions, organizational, etc) to take into account.  
>> And let's
>> say that the browser is our tool, just to keep it simple here.
>>
>> -------------------------------
>>
>> At the time an image is distributed, it is supposed to be in sync  
>> (not
>> necessarily *full* sync) with its respective source code  
>> subsystem. So
>> when you open a browser, all entities are colored black on white,  
>> and my
>> story begins.
>>
>> When you want to *observe* a certain area of the source code  
>> subsytem, you
>> put the respective entity (item from here on) onto your watchlist  
>> (cmd+ if
>> you like) from where the item gets the color blue. After putting a  
>> class
>> onto your watchlist you get all its message categories and methods  
>> blue;
>> un-watch those (sub)items which are of no interest to you; etc for  
>> other
>> organizational levels.
>>
>> When you do a *change* to a blue colored item, its new source is
>> contributed to the source code subsystem (from here on scs for  
>> short) and
>> the item is colored green in your browser. If later somebody else
>> contributes a change to the same blue item, it gets the color  
>> yellow in
>> both browsers.
>>
>> And if an entity you have in your image cannot be *matched* with  
>> the scs,
>> it is colored red.
>>
>> As is tradition, if you don't save your image before the next  
>> crash, all
>> this coloring information is lost, and you are back to black and  
>> white.
>> The scs does NOT know the state of your image.
>>
>> -------------------------------
>>
>> A contribution to the scs either has a predecessor or else it is  
>> new (to
>> the scs). Let's say three developers make contributions to the  
>> same blue
>> item, then we arrive at three branches (and the three browsers  
>> color the
>> respective item yellow). It is up to you to accept one of these  
>> branches,
>> but you are no forced to do so. An accepted branch is colored  
>> green (it
>> will turn yellow when the next contribution relative to it arrives).
>>
>> -------------------------------
>>
>> By chasing the predecessor it is always possible to roll-back an  
>> item (or
>> a group of items), this is independent of the coloring I used for
>> illustration. The same "backwards" information is readily  
>> available for
>> analyzing the complete history of an item, including all unaccepted
>> branches.
>>
>> -------------------------------
>>
>> Doing this into the other direction (roll-forward) depends on which
>> branches where accepted. At any point in time, accepted branches  
>> represent
>> a "cut" in the scs.
>>
>> -------------------------------
>>
>> If you work alone on a project, you'll never see any yellow  
>> colored item,
>> unless you revert a change (because the dismissed becomes an  
>> unaccepted
>> branch). For people who are used to make small incremental changes, a
>> preference tells whether or not every change is treated as a  
>> branch or
>> else is to be consolidated until you do a "big save" to the scs. The
>> latter requires, I think, a local buffering scs on your harddisk  
>> (or on
>> your local area server).
>>
>> -------------------------------
>>
>> The amount of conversation between browser and scs is small, the  
>> browser
>> only has to ask for colors (one byte answer per item) for those  
>> items it
>> shows *and* which are on your watch list. But if you want to see more
>> (like, for example, all branches of an item), more data will have to
>> travel.
>>
>> -------------------------------
>>
>> It's now time for me to sit back and relax :-)
>>
>> /Klaus
>>
>>
>>
>
>
> ______________________________________________________________________
> __
> iFRANCE, exprimez-vous !
> http://web.ifrance.com
>

> On 27.07.2006, at 09:39, Klaus D. Witzel wrote:
>> No, I was lead by the aim for more collaborative features (hmm,
>> since there are none in Squeak, should I say for *at least* one or
>> two collaborative features).
>>
>
> What I would suggest: Do not plan to save the world, at least not at
> first. If you now set out to build "the solution" we won't have for a
> looong
> time, most likely never.
>
> What we need first is
>
>   - source offset in a property, not at the end of the bytecode. Use
> an Integer , so we get indefinite size of .source and .changens --
>     One Problem Solved.

> On Thu, 27 Jul 2006 09:57:24 +0200, Marcus Denker wrote:
>> On 27.07.2006, at 09:39, Klaus D. Witzel wrote:
>>> No, I was lead by the aim for more collaborative features (hmm,
>>> since there are none in Squeak, should I say for *at least* one or
>>> two collaborative features).
>>>
>>
>> What I would suggest: Do not plan to save the world, at least not at
>> first. If you now set out to build "the solution" we won't have for a
>> looong
>> time, most likely never.
>>
>> What we need first is
>>
>>   - source offset in a property, not at the end of the bytecode. Use
>> an Integer , so we get indefinite size of .source and .changens --
>>     One Problem Solved.
>
> Wait a minute, a quick question before that can be solved
>
> (nil systemNavigation
> allMethodsSelect: [:cm | (cm fileIndex = 0
> or: [(SourceFiles at: cm fileIndex) == nil])
> or: [cm filePosition = 0]])asArray
>
> This has size 204 in #7048, do the elements have anything to do with  
> potential changes to SourceFiles, fileIndex or filePosition?
>
> /Klaus
>
>
>

> On 27.07.2006, at 11:36, Klaus D. Witzel wrote:
>
>> The 204 methods without sourcePointer in #7048 are from
>>
>> BrowserProvider
>> WorldMenuProvider
>> PlayWithMe1
>> PlayWithMe1 class
>> Preferences class
>>
>> Marcus, can that be fixed (fileIn from some source/package) in the
>> next to come image
>
> No, these methods are generated, there was never code for them.

>
> On 27.07.2006, at 09:39, Klaus D. Witzel wrote:
>
> > Nicolas
> >
> > thank you for taking the time for a response.
> >
> > On Wed, 26 Jul 2006 20:55:44 +0200, you wrote:
> >
> >> Klaus, i would say this is another source version control system.
> >
> > I did not have version *control* in mind but yes, scs could be used
> > for such ancient concepts (used them for > 30 years, quite an
> > experience).
> >
> > No, I was lead by the aim for more collaborative features (hmm,
> > since there are none in Squeak, should I say for *at least* one or
> > two collaborative features).
> >
>
> What I would suggest: Do not plan to save the world, at least not at
> first. If you now set out to build "the solution" we won't have for a
> looong
> time, most likely never.
>
> What we need first is
>
>   - source offset in a property, not at the end of the bytecode. Use
> an Integer , so we get indefinite size of .source and .changens --
>     One Problem Solved.
> - Fix dan's source compression. This can work on the .changes, too,
> giving smaller releases or full development deployment
>    for embedded systems (dan it for the weather station) or web servers.
>
> With that, we have solved those problems that started the
> discussion... quite a step in the right direction.
>
> After that, I personally would continue by cleaning up the sources/
> changes handling and fileOut mechanism, thus then enabeling
> experiments to make something really better. But this "real better"
> thing is the last step, not the first.
>
> 1) fix the real problem
> 2) clean up / refactor / generalise
> 3) now invent the future...
>
>      Marcus
>
>
>
>
>
>

>> Trying to keep preliminaries as short as possible, we have as
>> organizational elements (for the sake of discussion): system
>> categories, classes, message categories and methods. The unit of
>> source code "entity" we consider to be a unit of change belongs to one
>> of these elements. We can add, replace, rename and delete. We have not
>> only methods, but also DoIt's (definitions, organizational, etc) to
>> take into account. And let's say that the browser is our tool, just to
>> keep it simple here.
>
> I would really like to have a distinction between Doit and definition.
> Because this way we could build much clever tools. In VW classDef are
> not doit and you can eliminate doits when you want to do a replay all.
>
> Do I understand correctly by saying that this would be a component that
> could be put between the sources/changes and MC for example.

> Klaus wrote:
>> I put together what I think is relevant for a new source code subsystem  
>> and, if I'm mistaken or have overlooked one of your recent suggestions  
>> and/or requirements, please do not hesitate and say so.
>>
>> What follows is not intended as a replacement for MC or SM, rather the  
>> new subsystem should be as independent as possible. If you load  
>> something with MC, the new subsystem can treat it like any other  
>> contribution (more about contributions later).
>
> I would really like to have a distinction between Doit and definition.  
> Because this way we could build much clever tools. In VW classDef are  
> not doit and you can eliminate doits when you want to do a replay all.