CodeImporter. 256 literals limit

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

CodeImporter. 256 literals limit

Andreas Sunardi
I have written a tool (Pharo5) where user gives an input file to it, where the content is a smalltalk code, a DSL. I used a subclass of CodeImporter class to evaluate this input file.

Recently my user used an input file where it hit the 256 literal limit (total of unique string, number, method name, etc), down in OpalEncoderForV3PlusClosures >> genPushLiteral:. The number seems to be hard coded and related to byte code generator, not something I can simply increase. I wasn't aware of this limitation.

Before I overhaul my tool, I thought I should ask. Is there another alternative to evaluate a smalltalk file/script? The file is small, 27k, but the number of unique literals in it is > 256. Is it possible at all, seeing that the limit is related to byte code generator.

Thank you in advance
--
Andreas Sunardi
Reply | Threaded
Open this post in threaded view
|

Re: CodeImporter. 256 literals limit

Marcus Denker-4
Hello,

The new byte code set that we will use in the future does not hat this limitation.
You can, but only in Pharo6 and later, the new byte code backend (it is called “SistaV1”).
(this is a global preference to be enabled in the Settings).

        Marcus

It is not yet used by default, but you can enable it. Either globally, or,

> On 29 Aug 2017, at 04:52, Andreas Sunardi <[hidden email]> wrote:
>
> I have written a tool (Pharo5) where user gives an input file to it, where the content is a smalltalk code, a DSL. I used a subclass of CodeImporter class to evaluate this input file.
>
> Recently my user used an input file where it hit the 256 literal limit (total of unique string, number, method name, etc), down in OpalEncoderForV3PlusClosures >> genPushLiteral:. The number seems to be hard coded and related to byte code generator, not something I can simply increase. I wasn't aware of this limitation.
>
> Before I overhaul my tool, I thought I should ask. Is there another alternative to evaluate a smalltalk file/script? The file is small, 27k, but the number of unique literals in it is > 256. Is it possible at all, seeing that the limit is related to byte code generator.
>
> Thank you in advance
> --
> Andreas Sunardi


Reply | Threaded
Open this post in threaded view
|

Re: CodeImporter. 256 literals limit

Clément Bera-4
In reply to this post by Andreas Sunardi
Hi,

If your tool works in Pharo 6, you can use the other bytecode set which supports up to 32k literals. To do so, go to:
World Menu > Settings > Compiler > Encoder
and pick SistaV1 instead of V3PlusClosures
Try to load your code. The default Pharo 6 VM supports both bytecode sets.

Alternatively you need to split your methods with many literals in multiple methods with less literals, which is usually quite tricky to do right.

On Tue, Aug 29, 2017 at 4:52 AM, Andreas Sunardi <[hidden email]> wrote:
I have written a tool (Pharo5) where user gives an input file to it, where the content is a smalltalk code, a DSL. I used a subclass of CodeImporter class to evaluate this input file.

Recently my user used an input file where it hit the 256 literal limit (total of unique string, number, method name, etc), down in OpalEncoderForV3PlusClosures >> genPushLiteral:. The number seems to be hard coded and related to byte code generator, not something I can simply increase. I wasn't aware of this limitation.

Before I overhaul my tool, I thought I should ask. Is there another alternative to evaluate a smalltalk file/script? The file is small, 27k, but the number of unique literals in it is > 256. Is it possible at all, seeing that the limit is related to byte code generator.

Thank you in advance
--
Andreas Sunardi

Reply | Threaded
Open this post in threaded view
|

Re: CodeImporter. 256 literals limit

Andreas Sunardi
That sounds good. Unfortunately for me, my work environment is using RedHat6/CentOS6 with glibc 2.12. Is there Pharo6 with glibc < 2.15 support. Or is there a way for me to build that myself?

It's quite a departure to change my DSL into defining multiple methods. But that's my own problem.

I'm happy to hear Pharo 6 can support a lot more literals. I tested my code on Pharo 6 (on my Windows box) and it works. I see SistaV1 compiler in Pharo 5 setting, but that causes Pharo to crash.

Thank you guys for your answers. I think I have the information I need to make decision. But if there is a way to get Pharo 6 that works with glibc < 2.15, please let me know.

Thank you
--
Andreas

On Tue, Aug 29, 2017 at 12:34 AM, Clément Bera <[hidden email]> wrote:
Hi,

If your tool works in Pharo 6, you can use the other bytecode set which supports up to 32k literals. To do so, go to:
World Menu > Settings > Compiler > Encoder
and pick SistaV1 instead of V3PlusClosures
Try to load your code. The default Pharo 6 VM supports both bytecode sets.

Alternatively you need to split your methods with many literals in multiple methods with less literals, which is usually quite tricky to do right.

On Tue, Aug 29, 2017 at 4:52 AM, Andreas Sunardi <[hidden email]> wrote:
I have written a tool (Pharo5) where user gives an input file to it, where the content is a smalltalk code, a DSL. I used a subclass of CodeImporter class to evaluate this input file.

Recently my user used an input file where it hit the 256 literal limit (total of unique string, number, method name, etc), down in OpalEncoderForV3PlusClosures >> genPushLiteral:. The number seems to be hard coded and related to byte code generator, not something I can simply increase. I wasn't aware of this limitation.

Before I overhaul my tool, I thought I should ask. Is there another alternative to evaluate a smalltalk file/script? The file is small, 27k, but the number of unique literals in it is > 256. Is it possible at all, seeing that the limit is related to byte code generator.

Thank you in advance
--
Andreas Sunardi


Reply | Threaded
Open this post in threaded view
|

Re: CodeImporter. 256 literals limit

Richard Sargent
Administrator
my work environment is using RedHat6/CentOS6 with glibc 2.12

That's seven years of unpatched security vulnerabilities! Are you sure you really want to stay at such great risk?


On Tue, Aug 29, 2017 at 8:30 AM, Andreas Sunardi <[hidden email]> wrote:
That sounds good. Unfortunately for me, my work environment is using RedHat6/CentOS6 with glibc 2.12. Is there Pharo6 with glibc < 2.15 support. Or is there a way for me to build that myself?

It's quite a departure to change my DSL into defining multiple methods. But that's my own problem.

I'm happy to hear Pharo 6 can support a lot more literals. I tested my code on Pharo 6 (on my Windows box) and it works. I see SistaV1 compiler in Pharo 5 setting, but that causes Pharo to crash.

Thank you guys for your answers. I think I have the information I need to make decision. But if there is a way to get Pharo 6 that works with glibc < 2.15, please let me know.

Thank you
--
Andreas

On Tue, Aug 29, 2017 at 12:34 AM, Clément Bera <[hidden email]> wrote:
Hi,

If your tool works in Pharo 6, you can use the other bytecode set which supports up to 32k literals. To do so, go to:
World Menu > Settings > Compiler > Encoder
and pick SistaV1 instead of V3PlusClosures
Try to load your code. The default Pharo 6 VM supports both bytecode sets.

Alternatively you need to split your methods with many literals in multiple methods with less literals, which is usually quite tricky to do right.

On Tue, Aug 29, 2017 at 4:52 AM, Andreas Sunardi <[hidden email]> wrote:
I have written a tool (Pharo5) where user gives an input file to it, where the content is a smalltalk code, a DSL. I used a subclass of CodeImporter class to evaluate this input file.

Recently my user used an input file where it hit the 256 literal limit (total of unique string, number, method name, etc), down in OpalEncoderForV3PlusClosures >> genPushLiteral:. The number seems to be hard coded and related to byte code generator, not something I can simply increase. I wasn't aware of this limitation.

Before I overhaul my tool, I thought I should ask. Is there another alternative to evaluate a smalltalk file/script? The file is small, 27k, but the number of unique literals in it is > 256. Is it possible at all, seeing that the limit is related to byte code generator.

Thank you in advance
--
Andreas Sunardi



Reply | Threaded
Open this post in threaded view
|

Re: CodeImporter. 256 literals limit

Andreas Sunardi
Point well taken. Unfortunately, I'm not in charge of the infrastructure. I don't know if my experience is typical or not, but it's difficult for IT to find a period to disrupt engineering projects, potentially breaking tools, without drawing fire from engineering :)


On Tue, Aug 29, 2017 at 8:45 AM, Richard Sargent <[hidden email]> wrote:
my work environment is using RedHat6/CentOS6 with glibc 2.12

That's seven years of unpatched security vulnerabilities! Are you sure you really want to stay at such great risk?


On Tue, Aug 29, 2017 at 8:30 AM, Andreas Sunardi <[hidden email]> wrote:
That sounds good. Unfortunately for me, my work environment is using RedHat6/CentOS6 with glibc 2.12. Is there Pharo6 with glibc < 2.15 support. Or is there a way for me to build that myself?

It's quite a departure to change my DSL into defining multiple methods. But that's my own problem.

I'm happy to hear Pharo 6 can support a lot more literals. I tested my code on Pharo 6 (on my Windows box) and it works. I see SistaV1 compiler in Pharo 5 setting, but that causes Pharo to crash.

Thank you guys for your answers. I think I have the information I need to make decision. But if there is a way to get Pharo 6 that works with glibc < 2.15, please let me know.

Thank you
--
Andreas

On Tue, Aug 29, 2017 at 12:34 AM, Clément Bera <[hidden email]> wrote:
Hi,

If your tool works in Pharo 6, you can use the other bytecode set which supports up to 32k literals. To do so, go to:
World Menu > Settings > Compiler > Encoder
and pick SistaV1 instead of V3PlusClosures
Try to load your code. The default Pharo 6 VM supports both bytecode sets.

Alternatively you need to split your methods with many literals in multiple methods with less literals, which is usually quite tricky to do right.

On Tue, Aug 29, 2017 at 4:52 AM, Andreas Sunardi <[hidden email]> wrote:
I have written a tool (Pharo5) where user gives an input file to it, where the content is a smalltalk code, a DSL. I used a subclass of CodeImporter class to evaluate this input file.

Recently my user used an input file where it hit the 256 literal limit (total of unique string, number, method name, etc), down in OpalEncoderForV3PlusClosures >> genPushLiteral:. The number seems to be hard coded and related to byte code generator, not something I can simply increase. I wasn't aware of this limitation.

Before I overhaul my tool, I thought I should ask. Is there another alternative to evaluate a smalltalk file/script? The file is small, 27k, but the number of unique literals in it is > 256. Is it possible at all, seeing that the limit is related to byte code generator.

Thank you in advance
--
Andreas Sunardi




Reply | Threaded
Open this post in threaded view
|

Re: CodeImporter. 256 literals limit

EstebanLM
you can try the new (still experimental) packaging for Pharo: 

CentOS 6.8:

# Add the repo

# Install 32bit packages (with X11 dependency for *-ui or not)

$ yum install pharo6-32-ui.i686 or pharo6-32.i386

# Install 64bit packages

$ yum install pharo6-64-ui.x86_64 pharo6-64.x86_64

that will install correct vm for your distribution.

cheers,
Esteban



On 29 Aug 2017, at 18:58, Andreas Sunardi <[hidden email]> wrote:

Point well taken. Unfortunately, I'm not in charge of the infrastructure. I don't know if my experience is typical or not, but it's difficult for IT to find a period to disrupt engineering projects, potentially breaking tools, without drawing fire from engineering :)


On Tue, Aug 29, 2017 at 8:45 AM, Richard Sargent <[hidden email]> wrote:
my work environment is using RedHat6/CentOS6 with glibc 2.12

That's seven years of unpatched security vulnerabilities! Are you sure you really want to stay at such great risk?


On Tue, Aug 29, 2017 at 8:30 AM, Andreas Sunardi <[hidden email]> wrote:
That sounds good. Unfortunately for me, my work environment is using RedHat6/CentOS6 with glibc 2.12. Is there Pharo6 with glibc < 2.15 support. Or is there a way for me to build that myself?

It's quite a departure to change my DSL into defining multiple methods. But that's my own problem.

I'm happy to hear Pharo 6 can support a lot more literals. I tested my code on Pharo 6 (on my Windows box) and it works. I see SistaV1 compiler in Pharo 5 setting, but that causes Pharo to crash.

Thank you guys for your answers. I think I have the information I need to make decision. But if there is a way to get Pharo 6 that works with glibc < 2.15, please let me know.

Thank you
--
Andreas

On Tue, Aug 29, 2017 at 12:34 AM, Clément Bera <[hidden email]> wrote:
Hi,

If your tool works in Pharo 6, you can use the other bytecode set which supports up to 32k literals. To do so, go to:
World Menu > Settings > Compiler > Encoder
and pick SistaV1 instead of V3PlusClosures
Try to load your code. The default Pharo 6 VM supports both bytecode sets.

Alternatively you need to split your methods with many literals in multiple methods with less literals, which is usually quite tricky to do right.

On Tue, Aug 29, 2017 at 4:52 AM, Andreas Sunardi <[hidden email]> wrote:
I have written a tool (Pharo5) where user gives an input file to it, where the content is a smalltalk code, a DSL. I used a subclass of CodeImporter class to evaluate this input file.

Recently my user used an input file where it hit the 256 literal limit (total of unique string, number, method name, etc), down in OpalEncoderForV3PlusClosures >> genPushLiteral:. The number seems to be hard coded and related to byte code generator, not something I can simply increase. I wasn't aware of this limitation.

Before I overhaul my tool, I thought I should ask. Is there another alternative to evaluate a smalltalk file/script? The file is small, 27k, but the number of unique literals in it is > 256. Is it possible at all, seeing that the limit is related to byte code generator.

Thank you in advance
--
Andreas Sunardi





Reply | Threaded
Open this post in threaded view
|

Re: CodeImporter. 256 literals limit

Andreas Sunardi
Esteban, I'm happy to report this package works for me, in my RH/CentOS 6 environment. I have to do a bit more since I have to install it in a network path (can't install in each server).

If there's a download option instead of an rpm, such as that in Pharo 5 download website, that would be great. But I don't want to complain about it. The response and support from this community has been very impressive and I'm very grateful for it. Thank you very much!

--
Andreas



On Wed, Aug 30, 2017 at 2:02 AM, Esteban Lorenzano <[hidden email]> wrote:
you can try the new (still experimental) packaging for Pharo: 

CentOS 6.8:

# Add the repo

# Install 32bit packages (with X11 dependency for *-ui or not)

$ yum install pharo6-32-ui.i686 or pharo6-32.i386

# Install 64bit packages

$ yum install pharo6-64-ui.x86_64 pharo6-64.x86_64

that will install correct vm for your distribution.

cheers,
Esteban



On 29 Aug 2017, at 18:58, Andreas Sunardi <[hidden email]> wrote:

Point well taken. Unfortunately, I'm not in charge of the infrastructure. I don't know if my experience is typical or not, but it's difficult for IT to find a period to disrupt engineering projects, potentially breaking tools, without drawing fire from engineering :)


On Tue, Aug 29, 2017 at 8:45 AM, Richard Sargent <[hidden email]> wrote:
my work environment is using RedHat6/CentOS6 with glibc 2.12

That's seven years of unpatched security vulnerabilities! Are you sure you really want to stay at such great risk?


On Tue, Aug 29, 2017 at 8:30 AM, Andreas Sunardi <[hidden email]> wrote:
That sounds good. Unfortunately for me, my work environment is using RedHat6/CentOS6 with glibc 2.12. Is there Pharo6 with glibc < 2.15 support. Or is there a way for me to build that myself?

It's quite a departure to change my DSL into defining multiple methods. But that's my own problem.

I'm happy to hear Pharo 6 can support a lot more literals. I tested my code on Pharo 6 (on my Windows box) and it works. I see SistaV1 compiler in Pharo 5 setting, but that causes Pharo to crash.

Thank you guys for your answers. I think I have the information I need to make decision. But if there is a way to get Pharo 6 that works with glibc < 2.15, please let me know.

Thank you
--
Andreas

On Tue, Aug 29, 2017 at 12:34 AM, Clément Bera <[hidden email]> wrote:
Hi,

If your tool works in Pharo 6, you can use the other bytecode set which supports up to 32k literals. To do so, go to:
World Menu > Settings > Compiler > Encoder
and pick SistaV1 instead of V3PlusClosures
Try to load your code. The default Pharo 6 VM supports both bytecode sets.

Alternatively you need to split your methods with many literals in multiple methods with less literals, which is usually quite tricky to do right.

On Tue, Aug 29, 2017 at 4:52 AM, Andreas Sunardi <[hidden email]> wrote:
I have written a tool (Pharo5) where user gives an input file to it, where the content is a smalltalk code, a DSL. I used a subclass of CodeImporter class to evaluate this input file.

Recently my user used an input file where it hit the 256 literal limit (total of unique string, number, method name, etc), down in OpalEncoderForV3PlusClosures >> genPushLiteral:. The number seems to be hard coded and related to byte code generator, not something I can simply increase. I wasn't aware of this limitation.

Before I overhaul my tool, I thought I should ask. Is there another alternative to evaluate a smalltalk file/script? The file is small, 27k, but the number of unique literals in it is > 256. Is it possible at all, seeing that the limit is related to byte code generator.

Thank you in advance
--
Andreas Sunardi