Could someone add a VMMaker Inbox project to source.squeak.org?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Could someone add a VMMaker Inbox project to source.squeak.org?

Eliot Miranda-2
 
I'd like to roll back VM Maker: VMMaker.oscog-eem.2445 from the trunk VMMaker but keep it to the side.
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Bert Freudenberg
 

On Mon, Oct 1, 2018 at 4:10 PM Eliot Miranda <[hidden email]> wrote:
 
I'd like to roll back VM Maker: VMMaker.oscog-eem.2445 from the trunk VMMaker but keep it to the side.
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

David T. Lewis
 
Brilliant! Thanks Bert and Eliot.

Dave

On Mon, Oct 01, 2018 at 06:05:36PM -0700, Bert Freudenberg wrote:

>  
> http://source.squeak.org/VMMakerInbox.html
>
> - Bert -
>
>
> On Mon, Oct 1, 2018 at 4:10 PM Eliot Miranda <[hidden email]>
> wrote:
>
> >
> > I'd like to roll back VM Maker: VMMaker.oscog-eem.2445 from the trunk
> > VMMaker but keep it to the side.
> > _,,,^..^,,,_
> > best, Eliot
> >

Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Bert Freudenberg
 
Did it just crash? And take that new repo with it?

- Bert -

On Mon, Oct 1, 2018 at 6:20 PM David T. Lewis <[hidden email]> wrote:
 
Brilliant! Thanks Bert and Eliot.

Dave

On Mon, Oct 01, 2018 at 06:05:36PM -0700, Bert Freudenberg wrote:

> http://source.squeak.org/VMMakerInbox.html
>
> - Bert -
>
>
> On Mon, Oct 1, 2018 at 4:10 PM Eliot Miranda <[hidden email]>
> wrote:
>
> >
> > I'd like to roll back VM Maker: VMMaker.oscog-eem.2445 from the trunk
> > VMMaker but keep it to the side.
> > _,,,^..^,,,_
> > best, Eliot
> >

Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Bert Freudenberg
 
On Mon, Oct 1, 2018 at 7:15 PM Bert Freudenberg <[hidden email]> wrote:
Did it just crash? And take that new repo with it?

I did get a 502 Bad gateway, but it's still there. Curious why it's not in the nav list on the left.

- Bert -

 
- Bert -

On Mon, Oct 1, 2018 at 6:20 PM David T. Lewis <[hidden email]> wrote:
 
Brilliant! Thanks Bert and Eliot.

Dave

On Mon, Oct 01, 2018 at 06:05:36PM -0700, Bert Freudenberg wrote:

> http://source.squeak.org/VMMakerInbox.html
>
> - Bert -
>
>
> On Mon, Oct 1, 2018 at 4:10 PM Eliot Miranda <[hidden email]>
> wrote:
>
> >
> > I'd like to roll back VM Maker: VMMaker.oscog-eem.2445 from the trunk
> > VMMaker but keep it to the side.
> > _,,,^..^,,,_
> > best, Eliot
> >

Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Eliot Miranda-2
 
Hi Bert,

    thanks!!

On Mon, Oct 1, 2018 at 7:19 PM Bert Freudenberg <[hidden email]> wrote:
 
On Mon, Oct 1, 2018 at 7:15 PM Bert Freudenberg <[hidden email]> wrote:
Did it just crash? And take that new repo with it?

I did get a 502 Bad gateway, but it's still there. Curious why it's not in the nav list on the left.

I was able to provoke a 504 (?) Gateway Timeout by asking for a diff.  I wonder if the diff generation is optimisable.  It always breaks updating the repo after committing too.  What and where is the code in question?

- Bert -

- Bert -

On Mon, Oct 1, 2018 at 6:20 PM David T. Lewis <[hidden email]> wrote:
 
Brilliant! Thanks Bert and Eliot.

Dave

On Mon, Oct 01, 2018 at 06:05:36PM -0700, Bert Freudenberg wrote:

> http://source.squeak.org/VMMakerInbox.html
>
> - Bert -
>
> On Mon, Oct 1, 2018 at 4:10 PM Eliot Miranda <[hidden email]>
> wrote:
>
> >
> > I'd like to roll back VM Maker: VMMaker.oscog-eem.2445 from the trunk
> > VMMaker but keep it to the side.
> > _,,,^..^,,,_
> > best, Eliot
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Levente Uzonyi
 
My impression is that these slowdowns are present since the update to the
Magma backend.
I haven't seen the code yet, but I presume the code accessing data from
Magma is responsible for this, and someone familiar with Magma can easily
fix it.

Levente
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Bert Freudenberg
 

- Bert -

On Tue, Oct 2, 2018 at 10:09 AM Levente Uzonyi <[hidden email]> wrote:
 
My impression is that these slowdowns are present since the update to the
Magma backend.
I haven't seen the code yet, but I presume the code accessing data from
Magma is responsible for this, and someone familiar with Magma can easily
fix it.

Levente
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Chris Muller-3
In reply to this post by Levente Uzonyi
 
As I'm no longer able to VNC into ss-andreas, I'm not able to do any
diagnostics on the production server.  I emailed about this a few
weeks ago but got no responses.

Remmina is suddenly refusing to connect, and I see this message in my
terminal window:

    channel 3: open failed: administratively prohibited: open failed .

I'm tunneling port 5901 to that on my localhost when I log in:

   ssh -L 5901:locahost:5901 chrismuller@andreas

and then connecting Remmina to localhost:5901.  I tried using ssh
within the tunnel, I didn't see the "administrative message" but
Remmina locked up.

I never had this problem before, and not expert enough with SSH and
haven't had time to google it.  Sorry.

 - Chris
On Tue, Oct 2, 2018 at 12:08 PM Levente Uzonyi <[hidden email]> wrote:
>
>
> My impression is that these slowdowns are present since the update to the
> Magma backend.
> I haven't seen the code yet, but I presume the code accessing data from
> Magma is responsible for this, and someone familiar with Magma can easily
> fix it.
>
> Levente
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Chris Muller-3
In reply to this post by Levente Uzonyi
 
There are a number of factors that combine to lead to occasional
timeouts depending on the package.

In this case, VMMaker is a huge package, and it takes a long time for
the server to open up the .mcz and materialize its contents.  That's
why, for example, the diffing operation, which doesn't use Magma at
all, times out.

When a new version of VMMaker is saved, once again, it must be opened
up by the server and then, all of its MCDefinitions are enumerated and
indexed into huge Dictionary's stored by Magma.  This takes a long
time and even though it occurs in a background local Smalltalk
Process, it can affect local image performance.

Another cause are the many thousands of files which the server must
constantly refresh its directory cache, again and again, as it grows
ever larger.

Yet another issue is the dual persistence -- currently since we use
both File-based AND Magma, we have to save a "data.obj" file, and so
we're forced to keep the entire model into memory instead of taking
advantage of Magma's ability to work efficiently with subgraphs of
models.  But this is the only way we can have the MC history and
origin functions.

As I run the exact same code repository on my laptops to manage my own
code as runs source.squeak.org, after Squeak 5.2, I plan to refresh
and test my own codebase, bring it up to 64-bit, latest Magma, and
make some optimizations.  I'll test it by running my own development
with it for a while and then apply the same upgrades to
source.squeak.org.

 - Chris
On Tue, Oct 2, 2018 at 12:08 PM Levente Uzonyi <[hidden email]> wrote:
>
>
> My impression is that these slowdowns are present since the update to the
> Magma backend.
> I haven't seen the code yet, but I presume the code accessing data from
> Magma is responsible for this, and someone familiar with Magma can easily
> fix it.
>
> Levente
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Chris Muller-3
 
> Yet another issue is the dual persistence -- currently since we use
> both File-based AND Magma, we have to save a "data.obj" file, and so
> we're forced to keep the entire model into memory instead of taking
> advantage of Magma's ability to work efficiently with subgraphs of
> models.  But this is the only way we can have the MC history and
> origin functions.

When I do my 5.2 upgrade, I'll switch my own repository to Magma-only
and see how well it improves performance.  If it goes well, will
consider for source.squeak.org, too.

> As I run the exact same code repository on my laptops to manage my own
> code as runs source.squeak.org, after Squeak 5.2, I plan to refresh
> and test my own codebase, bring it up to 64-bit, latest Magma, and
> make some optimizations.  I'll test it by running my own development
> with it for a while and then apply the same upgrades to
> source.squeak.org.
>
>  - Chris
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Chris Muller-3
In reply to this post by Eliot Miranda-2
 
Hi Eliot,

What are your plans for this new "VMMaker Inbox" project?

Is this an area you want to reserve for other VM Maker contributors
going forward?

Do we need to refine our normal Inbox process?

 - Chris

On Tue, Oct 2, 2018 at 7:39 AM Eliot Miranda <[hidden email]> wrote:

>
>
> Hi Bert,
>
>     thanks!!
>
> On Mon, Oct 1, 2018 at 7:19 PM Bert Freudenberg <[hidden email]> wrote:
>>
>>
>> On Mon, Oct 1, 2018 at 7:15 PM Bert Freudenberg <[hidden email]> wrote:
>>>
>>> Did it just crash? And take that new repo with it?
>>
>>
>> I did get a 502 Bad gateway, but it's still there. Curious why it's not in the nav list on the left.
>
>
> I was able to provoke a 504 (?) Gateway Timeout by asking for a diff.  I wonder if the diff generation is optimisable.  It always breaks updating the repo after committing too.  What and where is the code in question?
>
>> - Bert -
>>
>>> - Bert -
>>>
>>> On Mon, Oct 1, 2018 at 6:20 PM David T. Lewis <[hidden email]> wrote:
>>>>
>>>>
>>>> Brilliant! Thanks Bert and Eliot.
>>>>
>>>> Dave
>>>>
>>>> On Mon, Oct 01, 2018 at 06:05:36PM -0700, Bert Freudenberg wrote:
>>>> >
>>>> > http://source.squeak.org/VMMakerInbox.html
>>>> >
>>>> > - Bert -
>>>> >
>>>> > On Mon, Oct 1, 2018 at 4:10 PM Eliot Miranda <[hidden email]>
>>>> > wrote:
>>>> >
>>>> > >
>>>> > > I'd like to roll back VM Maker: VMMaker.oscog-eem.2445 from the trunk
>>>> > > VMMaker but keep it to the side.
>>>> > > _,,,^..^,,,_
>>>> > > best, Eliot
>
> _,,,^..^,,,_
> best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [Box-Admins] Could someone add a VMMaker Inbox project to source.squeak.org?

Levente Uzonyi
In reply to this post by Chris Muller-3
 
Hi Chris,

I just checked and I could create tunnels and connect to the VNC server in
the image through the tunnel.
I also checked the server configuration and didn't find any restriction on
tunnel creation which would explain the problems you experienced.
One thing I would try is to use 127.0.0.1 instead of localhost in the
command, because localhost is just a name that can resolve to whatever
address on the server:

    ssh -L 5901:127.0.0.1:5901 chrismuller@andreas

Levente

On Tue, 2 Oct 2018, Chris Muller wrote:

> As I'm no longer able to VNC into ss-andreas, I'm not able to do any
> diagnostics on the production server.  I emailed about this a few
> weeks ago but got no responses.
>
> Remmina is suddenly refusing to connect, and I see this message in my
> terminal window:
>
>    channel 3: open failed: administratively prohibited: open failed .
>
> I'm tunneling port 5901 to that on my localhost when I log in:
>
>   ssh -L 5901:locahost:5901 chrismuller@andreas
>
> and then connecting Remmina to localhost:5901.  I tried using ssh
> within the tunnel, I didn't see the "administrative message" but
> Remmina locked up.
>
> I never had this problem before, and not expert enough with SSH and
> haven't had time to google it.  Sorry.
>
> - Chris
> On Tue, Oct 2, 2018 at 12:08 PM Levente Uzonyi <[hidden email]> wrote:
>>
>>
>> My impression is that these slowdowns are present since the update to the
>> Magma backend.
>> I haven't seen the code yet, but I presume the code accessing data from
>> Magma is responsible for this, and someone familiar with Magma can easily
>> fix it.
>>
>> Levente
>
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Bert Freudenberg
In reply to this post by Chris Muller-3
 
Hi Chris,

VMMaker is independent from the Squeak Trunk process. That's why it has its own repository, and now its own inbox.

We did the same for Etoys before, there was the Etoys repo, and the Etoys Inbox for contributions.

- Bert -

On Tue, Oct 2, 2018 at 2:19 PM Chris Muller <[hidden email]> wrote:
 
Hi Eliot,

What are your plans for this new "VMMaker Inbox" project?

Is this an area you want to reserve for other VM Maker contributors
going forward?

Do we need to refine our normal Inbox process?

 - Chris

On Tue, Oct 2, 2018 at 7:39 AM Eliot Miranda <[hidden email]> wrote:
>
>
> Hi Bert,
>
>     thanks!!
>
> On Mon, Oct 1, 2018 at 7:19 PM Bert Freudenberg <[hidden email]> wrote:
>>
>>
>> On Mon, Oct 1, 2018 at 7:15 PM Bert Freudenberg <[hidden email]> wrote:
>>>
>>> Did it just crash? And take that new repo with it?
>>
>>
>> I did get a 502 Bad gateway, but it's still there. Curious why it's not in the nav list on the left.
>
>
> I was able to provoke a 504 (?) Gateway Timeout by asking for a diff.  I wonder if the diff generation is optimisable.  It always breaks updating the repo after committing too.  What and where is the code in question?
>
>> - Bert -
>>
>>> - Bert -
>>>
>>> On Mon, Oct 1, 2018 at 6:20 PM David T. Lewis <[hidden email]> wrote:
>>>>
>>>>
>>>> Brilliant! Thanks Bert and Eliot.
>>>>
>>>> Dave
>>>>
>>>> On Mon, Oct 01, 2018 at 06:05:36PM -0700, Bert Freudenberg wrote:
>>>> >
>>>> > http://source.squeak.org/VMMakerInbox.html
>>>> >
>>>> > - Bert -
>>>> >
>>>> > On Mon, Oct 1, 2018 at 4:10 PM Eliot Miranda <[hidden email]>
>>>> > wrote:
>>>> >
>>>> > >
>>>> > > I'd like to roll back VM Maker: VMMaker.oscog-eem.2445 from the trunk
>>>> > > VMMaker but keep it to the side.
>>>> > > _,,,^..^,,,_
>>>> > > best, Eliot
>
> _,,,^..^,,,_
> best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Eliot Miranda-2
In reply to this post by Chris Muller-3
 
Hi Chris,


> On Oct 2, 2018, at 2:17 PM, Chris Muller <[hidden email]> wrote:
>
>
> Hi Eliot,
>
> What are your plans for this new "VMMaker Inbox" project?

Adding to Bert’s reply, I hope the VMMaker Inbox will result in a more reliable main VMMaker.oscog process.  I am using the VMMaker Inbox for in-progress fixes and experiments, so it allows development to occur without disturbing VMMaker.oscog as much as has happened in the past.


>
> Is this an area you want to reserve for other VM Maker contributors
> going forward?
>
> Do we need to refine our normal Inbox process?
>
> - Chris
>
>> On Tue, Oct 2, 2018 at 7:39 AM Eliot Miranda <[hidden email]> wrote:
>>
>>
>> Hi Bert,
>>
>>    thanks!!
>>
>>> On Mon, Oct 1, 2018 at 7:19 PM Bert Freudenberg <[hidden email]> wrote:
>>>
>>>
>>>> On Mon, Oct 1, 2018 at 7:15 PM Bert Freudenberg <[hidden email]> wrote:
>>>>
>>>> Did it just crash? And take that new repo with it?
>>>
>>>
>>> I did get a 502 Bad gateway, but it's still there. Curious why it's not in the nav list on the left.
>>
>>
>> I was able to provoke a 504 (?) Gateway Timeout by asking for a diff.  I wonder if the diff generation is optimisable.  It always breaks updating the repo after committing too.  What and where is the code in question?
>>
>>> - Bert -
>>>
>>>> - Bert -
>>>>
>>>>> On Mon, Oct 1, 2018 at 6:20 PM David T. Lewis <[hidden email]> wrote:
>>>>>
>>>>>
>>>>> Brilliant! Thanks Bert and Eliot.
>>>>>
>>>>> Dave
>>>>>
>>>>>> On Mon, Oct 01, 2018 at 06:05:36PM -0700, Bert Freudenberg wrote:
>>>>>>
>>>>>> http://source.squeak.org/VMMakerInbox.html
>>>>>>
>>>>>> - Bert -
>>>>>>
>>>>>> On Mon, Oct 1, 2018 at 4:10 PM Eliot Miranda <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> I'd like to roll back VM Maker: VMMaker.oscog-eem.2445 from the trunk
>>>>>>> VMMaker but keep it to the side.
>>>>>>> _,,,^..^,,,_
>>>>>>> best, Eliot
>>
>> _,,,^..^,,,_
>> best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Could someone add a VMMaker Inbox project to source.squeak.org?

Levente Uzonyi
In reply to this post by Chris Muller-3
 
On Tue, 2 Oct 2018, Chris Muller wrote:

>
> There are a number of factors that combine to lead to occasional
> timeouts depending on the package.
>
> In this case, VMMaker is a huge package, and it takes a long time for
> the server to open up the .mcz and materialize its contents.  That's
> why, for example, the diffing operation, which doesn't use Magma at
> all, times out.

I measured diff creation on my machine using various VMMaker versions:
- creating a diff of 2 versions takes less than 2 seconds
- from this 2 seconds the actuall diff creation takes about 150ms, which
can easily be halved with some basic optimizations
- more than a second is spent by DataStream reading and mangling strings
for no real reason. We should be able to significantly reduce that as
well.
- the remaining few hundred milliseconds is spent on parsing the ancestry,
which can only be solved by using a new non-recursive ancestry format
which doesn't require you to parse everything to read a single entry.

But 2 seconds is not enough to get a 504 response, so there must be some
other reason for those timeouts.

Levente

>
> When a new version of VMMaker is saved, once again, it must be opened
> up by the server and then, all of its MCDefinitions are enumerated and
> indexed into huge Dictionary's stored by Magma.  This takes a long
> time and even though it occurs in a background local Smalltalk
> Process, it can affect local image performance.
>
> Another cause are the many thousands of files which the server must
> constantly refresh its directory cache, again and again, as it grows
> ever larger.
>
> Yet another issue is the dual persistence -- currently since we use
> both File-based AND Magma, we have to save a "data.obj" file, and so
> we're forced to keep the entire model into memory instead of taking
> advantage of Magma's ability to work efficiently with subgraphs of
> models.  But this is the only way we can have the MC history and
> origin functions.
>
> As I run the exact same code repository on my laptops to manage my own
> code as runs source.squeak.org, after Squeak 5.2, I plan to refresh
> and test my own codebase, bring it up to 64-bit, latest Magma, and
> make some optimizations.  I'll test it by running my own development
> with it for a while and then apply the same upgrades to
> source.squeak.org.
>
> - Chris
> On Tue, Oct 2, 2018 at 12:08 PM Levente Uzonyi <[hidden email]> wrote:
>>
>>
>> My impression is that these slowdowns are present since the update to the
>> Magma backend.
>> I haven't seen the code yet, but I presume the code accessing data from
>> Magma is responsible for this, and someone familiar with Magma can easily
>> fix it.
>>
>> Levente