Gemstone and virtualization

This is probably more a question about virtualization than about
gemstone itself, but maybe someone in this list has some relevant
experience...

We have a development VM running GLASS under VMWare Fusion / MacOSX.
It runs fine.  The OS is CentOS. We use an SSD.

A couple of days ago, we started to test if we could move that VM
under Proxmox/KVM on another machine (also using an SSD).  Althought
proxmox can start the VMDK without any problem, the GC performance is
really bad.  In fact it does not terminate after running for hours.
One of the two allocated cores jumps to 100% and that's the end of it.
The GC takes about 35 minutes on the Mac, it did not finish after 10
hours on the PC.

One of the issue I could think of is that the Mac has an Intel
processor, while the Proxmox PC has a couple of AMD ones.

Obviously we have yet to run other tests (including reinstalling GLASS
from scratch). But being able to move the VM around would have helped
some workflow steps here.


Suggestions?
Thanks!
Thierry

I should probably add that this is a very basic GLASS installation, so
special tweaking done on the MAC  (gemstone 2.4).

Thierry

Thierry,

MFC is i/o intensive so the fact that you are pegging a cpu is puzzling. Perhaps the cpu is pegged by the software that is bridging the gap from the SSD to Proxmox to VMWare to Centos to GemStone.

I am no vm expert, for example I don't know what Proxmox/KVM is, but is there a reason you are not using the VMWare virtualization software on the PC?

Adding one more layer to be virtualized doesn't sound like a performant route to follow.

Perhaps Proxmox/KVM doesn't work well with SSDs?

Dale
----- Original Message -----
| From: "Thierry Thelliez" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Friday, August 10, 2012 1:22:43 PM
| Subject: [GS/SS Beta] Gemstone and virtualization
|
| This is probably more a question about virtualization than about
| gemstone itself, but maybe someone in this list has some relevant
| experience...
|
| We have a development VM running GLASS under VMWare Fusion / MacOSX.
| It runs fine.  The OS is CentOS. We use an SSD.
|
| A couple of days ago, we started to test if we could move that VM
| under Proxmox/KVM on another machine (also using an SSD).  Althought
| proxmox can start the VMDK without any problem, the GC performance is
| really bad.  In fact it does not terminate after running for hours.
| One of the two allocated cores jumps to 100% and that's the end of
| it.
| The GC takes about 35 minutes on the Mac, it did not finish after 10
| hours on the PC.
|
| One of the issue I could think of is that the Mac has an Intel
| processor, while the Proxmox PC has a couple of AMD ones.
|
| Obviously we have yet to run other tests (including reinstalling
| GLASS
| from scratch). But being able to move the VM around would have helped
| some workflow steps here.
|
|
| Suggestions?
| Thanks!
| Thierry
|

Thanks Dale,

AFAIK, using Proxmox does not add a layer to the equation compared to
VMWare Fusion.

We are using Proxmox because the switch from VMWare server to VMWare
ESX did not make sense, financially wise, for us a couple of years
ago.  We are running Proxmox2.0 for ~40 production VMs and it works
quite well.

Regarding our port issue, we are not seeing any read traffic (with
iotop) during the MFC.  A few minutes ago, we ran some rsync and zip
tests with big files and that ran fine.

Somehow Gemstone/S does not like something during this move.

Unless you have other suggestions, we are going to try the following:
1- Run the VM on regular drives, although no issues are reported with
SSDs in the Proxmox forums.
2- Reinstall GLASS from scratch.
3- Explore the OpenVZ container option.
4- Put the MacBookPro in production (just joking...)

Thierry

Is there a way to interrupt the MFC and get some insights about what
is blocking?

Thierry

On Aug 10, 2012, at 1:22 PM, Thierry Thelliez wrote:

> One of the two allocated cores jumps to 100% and that's the end of it.

Which process is using most of the CPU (you should be able to identify this with 'top')?

If the process is a GemStone process have you signaled it to dump a stack? (Use $GEMSTONE/bin/pstack if it is available, otherwise 'kill -SIGUSR1').

Have you captured and looked at statmonitor files?

James

Thierry,

I thought there were free VMWare clients for all platforms ... okay the mac costs a small amount of money, but the linux client is free (or was ... as I say, I'm not an expert here) and I would have thought that a windows client (assuming that you mean windows when you say PC:) would have been free as well ...

Dale

----- Original Message -----
| From: "Thierry Thelliez" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Friday, August 10, 2012 2:04:03 PM
| Subject: Re: [GS/SS Beta] Gemstone and virtualization
|
| Thanks Dale,
|
| AFAIK, using Proxmox does not add a layer to the equation compared to
| VMWare Fusion.
|
| We are using Proxmox because the switch from VMWare server to VMWare
| ESX did not make sense, financially wise, for us a couple of years
| ago.  We are running Proxmox2.0 for ~40 production VMs and it works
| quite well.
|
| Regarding our port issue, we are not seeing any read traffic (with
| iotop) during the MFC.  A few minutes ago, we ran some rsync and zip
| tests with big files and that ran fine.
|
| Somehow Gemstone/S does not like something during this move.
|
| Unless you have other suggestions, we are going to try the following:
| 1- Run the VM on regular drives, although no issues are reported with
| SSDs in the Proxmox forums.
| 2- Reinstall GLASS from scratch.
| 3- Explore the OpenVZ container option.
| 4- Put the MacBookPro in production (just joking...)
|
| Thierry
|

James,

> Which process is using most of the CPU (you should be able to identify this with 'top')?
The MFC was started from Pharo, and then tried again directly with
Topaz.  In both cases that's their process that gets to the top of
'top'.

> If the process is a GemStone process have you signaled it to dump a stack? (Use $GEMSTONE/bin/pstack if it is available, otherwise 'kill -SIGUSR1').
The topaz process did not respond to SIGUSR1.

> Have you captured and looked at statmonitor files?
With statmonitor/vsd on before MFC, I am seeing a peak of reads just
after the start for MFC. And then nothing at all on any of the
variables.  The GarbageCollectionState goes to 1. The
objectReadInBytes goes to 1056000 and then flatlines (like all the
other parameters).

Thierry

Dale,

Yes the VMWare clients might be free, however Proxmox offers
interesting features for managing a private cloud (moving VMs live,
centralized management UI, VM level live backup/restore, ...).

BTW, the 'PC' is running proxmox/openvz directly. No windows ;-)

Thierry

On Aug 10, 2012, at 2:57 PM, Thierry Thelliez wrote:

>> If the process is a GemStone process have you signaled it to dump a stack? (Use $GEMSTONE/bin/pstack if it is available, otherwise 'kill -SIGUSR1').
> The topaz process did not respond to SIGUSR1.

Does it respond to the signal at other times? If so, then it seems that you have a case where the process can get stuck in an infinite loop and not responding to signals.

James

It only responds to the big kill -kill...

Here are some statMonitor records.  I looked at them with VSD, but
besides seeing that it quickly stops I am not sure what to make about
it.


Thierry

So `kill -USR1` just doesn't work at any time on any topaz?

Does everything else appear to behave correctly (backup/restore) so it's only the MFC that is misbehaving?

Clearly something in the stack is not behaving as GemStone expects ...  

You haven't mentioned which version of GemStone you are using ...

You might try attaching to the mfc process with gdb before initiating the mfc and then using ctl-c to interrupt the process when it goes hot, but you might not have any better luck.

The fact that gemstone behaves correctly in the same OS (from GemStone's perspective) says to me that this problem is likely related to an issue with Proxmox itself...

Meybe the proxmox folks have a way of debugging this issue from their end...

Dale
----- Original Message -----
| From: "Thierry Thelliez" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Friday, August 10, 2012 3:49:06 PM
| Subject: Re: [GS/SS Beta] Gemstone and virtualization
|
| It only responds to the big kill -kill...
|
| Here are some statMonitor records.  I looked at them with VSD, but
| besides seeing that it quickly stops I am not sure what to make about
| it.
|
|
| Thierry
|

We finally found the problem.  Vmware has a device driver called
'vmware tools'.  Even thought our other applications work fine,
gemstone does not like that we left 'vmware tools' in place when
moving the vmdk to Proxmox. And 'vmware tools ' has to be removed
before migrating the VM. Removing it after the move did not help. I am
not sure what was the problem with gemstone since we could not get any
useful log info, but it works fine now.


Thanks,
Thierry

I'm glad you figured out the issue!

Since GemStone has no clue as to whether or not it is running in a vm or not, it is likely that Proxmox is doing something odd in the presence of the 'vmware tools' module...

Dale

----- Original Message -----
| From: "Thierry Thelliez" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Sunday, August 19, 2012 8:10:06 AM
| Subject: Re: [GS/SS Beta] Gemstone and virtualization
|
| We finally found the problem.  Vmware has a device driver called
| 'vmware tools'.  Even thought our other applications work fine,
| gemstone does not like that we left 'vmware tools' in place when
| moving the vmdk to Proxmox. And 'vmware tools ' has to be removed
| before migrating the VM. Removing it after the move did not help. I
| am
| not sure what was the problem with gemstone since we could not get
| any
| useful log info, but it works fine now.
|
|
| Thanks,
| Thierry
|