NB FileSystem Primitives Speedup

For those complaining lately about the new FS being a bit slow ;)
Igor and I hacked together a test version for accessing files using NativeBoost.

From our stupid example (listing /usr/bin with ~1k Files) we were incredible 40 times! faster
than the file primitives in the current image (5ms instead of 200ms).

With the current FS infrastructure such a migration can be done absolutely painless by
simply replacing one back-end a time.

best
cami

yeah that's great!
...but we need a speed up of FS even without that. There are some simple protocols that needs to be enhanced, like the one that checks all entries in a root directory if I execute: '/tmp' asFileReference isDirectory :P

On Jun 1, 2012, at 9:17 PM, Camillo Bruni wrote:

fstat is your friend :)
i don't know if primitives providing access to that very useful function.

On 1 June 2012 21:21, Esteban Lorenzano <[hidden email]> wrote:


--
Best regards,
Igor Stasenko.

On Fri, Jun 1, 2012 at 12:17 PM, Camillo Bruni <[hidden email]> wrote:
Can you share this work?  I've been interested in this particular use
for NB for a while now, but haven't gotten around to trying it (or,
unfortunately, NB itself) yet.  This would probably push me into a
user role - and is definitely something I'd like to see.

-Chris

On 1 June 2012 22:47, Chris Cunningham <[hidden email]> wrote: hehe.. there's almost nothing to show about..
we're just making calls to two C functions:
opendir()
and
readdir()
from standard C library.

a very basic stuff, which you would do if your programming language is C :)

anyways, see attached.

If you on mac, it shall work.. try something like:

(UnixDir on: '/usr/bin') entries


> -Chris

--
Best regards,
Igor Stasenko.

ah.. sorry it will not work without these patches.
These are just couple minor refactorings for accepting '$' as valid
part of function name.
Because we discovered that we had to use 'readdir$INODE64' function
instead of 'readdir'
actually we could use older one, but 64 is a bit better :)

(i will do a new snapshot of Nativeboost-Core soon)


--
Best regards,
Igor Stasenko.

sure you are welcome.
we would also love to get OSProcess plugin rewritten.

Stef

On Jun 1, 2012, at 10:47 PM, Chris Cunningham wrote:

On Fri, Jun 01, 2012 at 11:01:29PM +0200, Igor Stasenko wrote:
That will definitely make a huge performance improvement. Here is something
similar (probably bit rotted but maybe still of interest):

http://wiki.squeak.org/squeak/2274

On 2 June 2012 15:23, David T. Lewis <[hidden email]> wrote:
it is a pity that this stuff was never integrated into VMs..
or actually never added to file plugin replacing it's naive implementation.



--
Best regards,
Igor Stasenko.

>> That will definitely make a huge performance improvement. Here is something
>> similar (probably bit rotted but maybe still of interest):
>>
>> http://wiki.squeak.org/squeak/2274
>>
>
> it is a pity that this stuff was never integrated into VMs..
> or actually never added to file plugin replacing it's naive implementation.

I do not understand the process of VM enh.
David what prevent it to get used?
May be we should add it the pharo version and suddenly it will kick the system.
But I'm rather puzzled and sad.

Stef

On Sat, Jun 02, 2012 at 04:11:16PM +0200, St?phane Ducasse wrote:
That is easy to explain ... I was lazy and did not bother to maintain it ;)

As for process, it is simple. Just build the plugin (though it probably
will need maintenance for bit rot), and add it to your VM builds. On
the image side, add the support to use the plugin if present, else fall
back on old methods.

I like small plugins because they are modular and reliable. They can
be added and removed without a lot of coordination with VM builders,
platform maintainers, etc. On the image side, you can easily detect
if the plugin is present, so it is not necessary to worry so much
about what version of VM or FilePlugin is running.

So the answer is ... Just Do It :)

<OT>
IMHO more people need to learn how to write small plugins. It is important
for modularity and reliability, and for trying new VM experiments without
waiting for a "VM guru" to help. Maybe you can also provide a way for
people to add their experimental plugins to the Jenkins builds too, so
that they can be made quickly available to anyone with an interest.
</OT>

Dave

>
> That is easy to explain ... I was lazy and did not bother to maintain it ;)

I do not buy such easy excuse. This is not your fault. It means that nobody cares.

> As for process, it is simple. Just build the plugin (though it probably
> will need maintenance for bit rot), and add it to your VM builds. On
> the image side, add the support to use the plugin if present, else fall
> back on old methods.

So like that bugs will just be more difficult to track?
I like the idea everybody doing its own little hack in his corner. Sounds like software engineering
but not with the right definition to me.

> I like small plugins because they are modular and reliable. They can
> be added and removed without a lot of coordination with VM builders,
> platform maintainers, etc. On the image side, you can easily detect
> if the plugin is present, so it is not necessary to worry so much
> about what version of VM or FilePlugin is running.

This was the same shitty story as having a slow socketStream in the image and that everybody had to load
the FastSocketStream. Hopefully an idiot like me or marcus renamed FastSocketStream into SocketStream and did not say it
and suddenly we got a decent socket stream. Else we would all have again a bad one.

> So the answer is ... Just Do It :)
Don't you think that this is what we are trying to do since ages: improving the infrastructure
but not by piling junk on junk.

David we are talking about File operations not funky experimental stuff.
To me it just shows that there is no concerted momentum around VMs. Now people will
shoot at me but I do not care just immensely sad. When I imagine that I'm spending my time
trying to make sure that companies can built software around Smalltalk. Sometimes I think that
I'm in the wrong community. Probably Pythoners and ruby people are more concerned.

> <OT>
> IMHO more people need to learn how to write small plugins. It is important
> for modularity and reliability, and for trying new VM experiments without
> waiting for a "VM guru" to help. Maybe you can also provide a way for
> people to add their experimental plugins to the Jenkins builds too, so
> that they can be made quickly available to anyone with an interest.
> </OT>

You mean with an svn structure where guys like igor do not have commit rights?
Probably but this is difficult and not really welcoming.

Stef

>
> Dave
>
>

On 2 June 2012 17:01, David T. Lewis <[hidden email]> wrote:
yes, i would wellcome that. And if you know, for instance we got this lately:
https://ci.lille.inria.fr/pharo/view/Cog/job/CogDroid/

but sometimes i feel like crying it the desert.

1. There's not many people who can be attracted in hacking VM(s)
2. And even among those who can, not many feel excited about idea of improving
and reimplementing it

The principle "don't fix it if it not broken" works pretty well in VM domain ,
because of factor of 10 in having fun & time spent when hacking
smalltalk code vs hacking
VM code.

--
Best regards,
Igor Stasenko.

Stéphane Ducasse wrote

To me it just shows that there is no concerted momentum around VMs... Probably Pythoners and ruby people are more concerned.

Yes. Probably they are concerned because they are so much slower than us ;)
http://shootout.alioth.debian.org/u32/performance.php?test=meteor

Don't get discouraged. The VM is frightening and mysterious to many, but with the help of Igor, Mariano, and others around to write blogs, answer questions, improve the build process, and lead the charge to improve, it seems that working on the VM is becoming more and more accessible/common... Even last year at ESUG, with 1 year using Smalltalk, I found building and working on the VM frustrating and confusing. But now, esp. with Jenkins/git/etc, it's a snap, for example, to build debug VMs, step through execution, and even submit VM patches.

Let's keep our great energy and progress multiplying...

Sean

On 2 June 2012 17:26, Stéphane Ducasse <[hidden email]> wrote:

> Don't you think that this is what we are trying to do since ages: improving the infrastructure
> but not by piling junk on junk.

hehe stef.. I can say this happens everywhere. Piles of junk on top of
each other..

Like figuring out, why output of readdir() function having strange results..
To figure it out, you have to read this code, and understand what's going on:
 this is part of standard C library on mac (/usr/include/dirent.h on
your machine, you can check it)


__BEGIN_DECLS
#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
int alphasort(const void *, const void *) __DARWIN_INODE64(alphasort);
#endif /* not POSIX */
int closedir(DIR *) __DARWIN_ALIAS(closedir);
#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
int getdirentries(int, char *, int, long *)
#if __DARWIN_64_BIT_INO_T
/*
 * getdirentries() doesn't work when 64-bit inodes is in effect, so we
 * generate a link error.
 */
                                                __asm("_getdirentries_is_not_available_when_64_bit_inodes_are_in_effect")
#else /* !__DARWIN_64_BIT_INO_T */
                                                __OSX_AVAILABLE_BUT_DEPRECATED(__MAC_10_0,__MAC_10_6,
__IPHONE_2_0,__IPHONE_2_0)
#endif /* __DARWIN_64_BIT_INO_T */
;
#endif /* not POSIX */
DIR *opendir(const char *) __DARWIN_ALIAS_I(opendir);
#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
DIR *__opendir2(const char *, int) __DARWIN_ALIAS_I(__opendir2);
#endif /* not POSIX */
struct dirent *readdir(DIR *) __DARWIN_INODE64(readdir);
int readdir_r(DIR *, struct dirent *, struct dirent **)
__DARWIN_INODE64(readdir_r);
void rewinddir(DIR *) __DARWIN_ALIAS_I(rewinddir);
#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
int scandir(const char *, struct dirent ***,
    int (*)(struct dirent *), int (*)(const void *, const void *))
__DARWIN_INODE64(scandir);
#ifdef __BLOCKS__
int scandir_b(const char *, struct dirent ***,
    int (^)(struct dirent *), int (^)(const void *, const void *))
__DARWIN_INODE64(scandir_b) __OSX_AVAILABLE_STARTING(__MAC_10_6,
__IPHONE_3_2);
#endif /* __BLOCKS__ */
#endif /* not POSIX */
void seekdir(DIR *, long) __DARWIN_ALIAS_I(seekdir);
long telldir(DIR *) __DARWIN_ALIAS_I(telldir);
__END_DECLS

#endif /* !KERNEL */





--
Best regards,
Igor Stasenko.

You know, Sean, what i like about C that every time i entering C
domain i need to
solve yet another hard to crack puzzle which takes unpredictably big
amount of time.
And you can never be sure that there is no hidden surprises behind next turn..
something which conceptually trivial to implement and even can fit on
one blank of paper
in C code, could take days before you can make it working.

Because C programming practices strictly following WYSIWYDG principle
(What You See Is What You DONT Get).

So, every time you feel like you lack of confusing moments during your day,
just go to /usr/include directory, open any header and try to read it ... :)

Or if you prefer C++, best is to read STL library code, or Boost  code.

On 2 June 2012 17:51, Sean P. DeNigris <[hidden email]> wrote:


--
Best regards,
Igor Stasenko.

2012/6/2 Igor Stasenko <[hidden email]>:
Ouch, Igor, you gonna give us a headache.
That's exatly the kind of niceties that makes FFI impracticle without
a pre-processor, remember our discussion?

Nicolas

2012/6/2 Igor Stasenko <[hidden email]>:
I would say, that you MIGHT get, depending on compiler and options...

> So, every time you feel like you lack of confusing moments during your day,
> just go to /usr/include directory, open any header and try to read it ... :)
>
> Or if you prefer C++, best is to read STL library code, or Boost  code.
>

+1, one of my favourite argument against C++
I always wondered which part of such obfuscation was intentional (vs necessary)

C++ adds complexity for sure, some totally useless like distinction
between delete and delete[], since underneath you still have dumb
pointers, and the compiler can't make a distinction, it's just one
more responsibility for the programmer.

Nicolas

On Jun 2, 2012, at 5:52 PM, Igor Stasenko wrote:

> On 2 June 2012 17:26, Stéphane Ducasse <[hidden email]> wrote:
>
>> Don't you think that this is what we are trying to do since ages: improving the infrastructure
>> but not by piling junk on junk.
>
> hehe stef.. I can say this happens everywhere. Piles of junk on top of
> each other..

still my point is valid.
How can we move on at the VM level by step by step improving it
How can we make sure that the code of david from 2006 which speeds up directories will be integrated?
How can we reduce the complexity of the vm?
This is why I'm working on the consortium to get a chance to the community.

Stef

On 2 June 2012 18:42, Nicolas Cellier
<[hidden email]> wrote:
...
>
> Ouch, Igor, you gonna give us a headache.
> That's exatly the kind of niceties that makes FFI impracticle without
> a pre-processor, remember our discussion?
>

That kind of niceties is a poor man's meta programming.
I don't see it as an argument against FFI. For me it is a clear
argument against system design.

Because look.. most of the times you don't care what happens under the hood of
your sport car.. until it stops working, or no longer meets your demands.
From position of consumer , it is clear what you do next - you go to
market and buy a new one,
or give it to engineers who can fix it.
So, either there is someone who can produce better cars (engineers),
or someone who can fix
your car (engineers). But if there nobody left, and only consumers
around, then you are in trouble.

And FFI actually points to that trouble: you cannot communicate with
system without having
a highly sophisticated compiler and preprocessor and also without
knowledge about which bells you should ring and what whistles you
should whistle.. this is road to nowhere.

This is clearly not FFI's fault. It is fault of system design.
For example, look at config.h file, which produced by autoconfig
utility.. can you tell me, what all those flags controlling? Ok, name
me the purpose of half of them.. Can you?
And then tell me, how you can write a "highly portable" C software and
guarantee that all those bells and whistles are correctly arranged?

From position of an engineer it is wrong to be on the side of
consumers (i don't care, it is something under the hood of my car,
which makes it run).
And for sure, nobody cares about insignificant little fly such as Cog
VM, except from few of us.
So, you have to go and open a hood of a car and remove a rotten cats &
rats from there,
this is how you can move to better future. Otherwise at some point,
cars will stop working,
planes will stop flying, computers will stop working, because nobody
will be left who would
understand how to make or fix them.

So, this is a right question, posed by Stephane: how we can make sure
that we will be able to
improve system in future, once its creators are gone, and nobody can
understand it anymore?
Only by simplifying and cleaning all layers of a system and without
relying on good Santa, which will come one day and clean this mess for
us.

--
Best regards,
Igor Stasenko.