Re: [squeak-dev] Pool for sysconf constants?

Am 17.10.2020 00:24:11 schrieb Eliot Miranda <[hidden email]>:

Hi David,


> On Oct 16, 2020, at 2:47 PM, David T. Lewis wrote:
>
> Hi Eliot,
>
> Replying back on vm-dev.
>
> Indeed I see the problem now, thanks.
>
> The closest that I can come to a useful suggestion is to write a
> little C program to print out the index values of interest for a
> given platform, then read that into the image to build the pool.
> The index values are supposed to be opaque but I doubt that they
> would change much in practice.

This is what an ExternalPool does *automatically*. One creates a pool
containing a set of named variables, adds metadata to the pool, and
then when the pool needs to initialize or change, it generates the C
program automatically, runs it, reaps the values, and updates class
side code to initialize the constants for specific platforms. When it
is run on a different platform it generates the program for that
platform, etc. The management problem therefore is how to manage the
executable programs because these are needed if one wants to deploy on
a specific platform.

I came up with the idea many years ago at ParcPlace when the problem
first surfaced (that different platforms, though they may use the same
names for symbolic constants, abstract types, etc, but don't use the
same values, concrete types, etc). Monty write the implementation.
Marcel has shepherded it into source.squeak.org/FFI. Marcel has
written an excellent and comprehensive class comment for ExternalPool.
I urge that everyone interested in deployment using FFIs read it ASAP.
That goes for me too, because things have moved on with the
implementation. There's no need to manage external C programs;
there's a need to merge the definition code that is automatically
generated when run on different platforms.

I should push thoguh and sdo a sysconf pool to learn how the current
system really works :-)

>
> I am looking only at a Linux system as I write this, and the only
> way I can find to discover the available parameter names is to look
> in /usr/include/bits/confname.h to see what is defined. I can't
> spot any other way to get at the information, and I can't find any
> runtime utility that provides a listing.
>
> But a little utility program that could be used to update the
> pool might be good enough. Attaching an example C program, I'm
> not sure if it helps.
>
> Dave
>
>
>> On Tue, Oct 13, 2020 at 01:49:00PM -0700, Eliot Miranda wrote:
>> Hi David,
>>
>>> On Mon, Oct 12, 2020 at 8:28 PM David T. Lewis wrote:
>>>
>>> On Sun, Oct 11, 2020 at 11:15:09AM -0700, Eliot Miranda wrote:
>>>> Hi All,
>>>>
>>>> I want to implement a number of processors query on MacOS & other
>>>> unices to complement the info provided by GetSystemInfo on Windows. The
>>>> natural way to do this on unices is to use sysconf (3), and of course
>>> this
>>>> points to the issue for which FFI pools are needed, the constant names
>>> for
>>>> sysconf such as _SC_NPROCESSORS_ONLN are defined, but their values are
>>>> implementation dependent.
>>>>
>>>> But before I go amd implement an FFI pool for this I thought I'd ask
>>>> a) anyone already done this? Is it published anywhere?
>>>> b) how are we going to organize such pools so that people don't have to
>>>> reinvent the wheel?
>>>> c) should there be a pool to cover sysconf or to cover unistd.h? (the
>>>> point here is that while the pool might start off small, it could grow
>>> and
>>>> grow and maybe unistd.h is too much surface to cover)
>>>>
>>>> thoughts, suggestions?
>>>
>>> Hi Eliot,
>>>
>>> At first glance, it looks to me like the hard part of the problem is to
>>> know what sysconf names are available to be queried on any given platform
>>> at runtime. If you know that, or if you are only interested in a limited
>>> number of well-known parameters (which seems likely), then it might be
>>> simplest to just implement the sysconf(3) call either as an FFI call or
>>> a call through the OSProcess plugin.
>>>
>>
>> Well, I *am* implementing it as an FFI call in Terf. The issue is deriving
>> the right integer values for the particular symbolic names, because these
>> vary across implementations. And that's what FFIPools are designed to
>> manage.
>> Right now I don't have the FFIPool so I've hard-wired the constant:
>>
>> !CMacOSXPlatform methodsFor: 'accessing' stamp: 'eem 10/11/2020 17:53'
>> prior: 47990341!
>> numberOfCPUs
>> "Warning, warning, Will Robertson!!!! The sysconf method bakes in a
>> decidedly implementation-specific name for the support library *and* bakes
>> in a value for a symbolic constant. Both of these are implementation
>> dependent and subject to change. That said..."
>> ^self sysconf: 58
>>
>> "CPlatform current numberOfCPUs"! !
>> !CMacOSXPlatform methodsFor: 'private' stamp: 'eem 10/11/2020 17:51'!
>> sysconf: systemInfo
>> "Answer a value from sysconf(3)."
>>
>> '/usr/lib/system/libsystem_c.dylib'>
>> ^self externalCallFailed
>>
>> " | scnprocsonline |
>> scnprocsonline := 58.
>> self current sysconf: scnprocsonline"! !
>>
>> There's much to be unhappy about in this code: libSystem.dylib is the
>> abstract name, but one can't use it because the search facilities can't
>> chain through the libraries that libSystem.B.dylib (ibSystem.dylib is a
>> symbolic link) refers to. So this could break in some future MacOS,
>> whereas if libSystem.dylib did work, or if there was a symbolic name we
>> could use that meant "the C library goddamnit", then it could be relied
>> upon. 58 is meaningful only on MacOS, and presumably only since a
>> particular version. That's why we need an FFIPool to manage the
>> cross-platform variations. The type is longlong, which means we're not
>> using a syntax that matches the declaration, and this will work only on
>> 64-bits. We can fix the FFI to interpret C names accoding to the
>> playtform, but we have a serious backwards-compatibility problem in that
>> all of the FFI definitions up til now have been written under this
>> assumption.
>>
>> But I suspect that I am not answering the right question here. If I am
>>> off base, can you give an example of some of the parameters that you
>>> would want to be able to query?
>>>
>>
>> First off Terf needs _SC_NUM_PROCESSORS_ONLN, which is notionally the
>> number of processors online, but now answers the number of cores, which
>> shows you how fast these ideas go stale (_SC_NUM_PROCESSORS_CONF is the
>> number of processors configured ;-) ). Other ones that could be meaningful
>> for applications include _SC_ARG_MAX (The maximum bytes of argument to
>> execve(2)) _SC_CHILD_MAX (The maximum number of simultaneous processes per
>> user id), _SC_OPEN_MAX (The maximum number of open files per user id),
>> _SC_STREAM_MAX (The minimum maximum number of streams that a process may
>> have open at any one time), _SC_PHYS_PAGES (The number of pages of physical
>> memory).
>>
>> There are many others, quite a few I can't imagine ever being compelling.
>>
>> Now yes, one can imagine implementing a cross-platform abstraction for
>> these but it's a lot of effort for little gain. It;s easier just to suck
>> it up and implement the FFI call.
>>
>> But I'm imagining these things will get used by the community and I don't
>> want people to have to reinvent the wheel. This kind of stuff doesn't
>> belong in trunk, but it does belong somewhere where we can share and
>> coevolve the facilities.
>>
>>
>>> Thanks,
>>> Dave
>>>
>>>
>>>
>>
>> --
>> _,,,^..^,,,_
>> best, Eliot
>
>>
>
>