Implementing DataFrame>>dtypes feature in Pharo PolyMath Project

Dear Balaji,

>  I am working on implementing the DataFrame>>dtypes feature which
> checks the datatypes of columns in a DataFrame, as part of my GSOC
> project. I have tried to explain my theoretical work done so far on
> this blog post. Please kindly go through it , as I need advice on what
> could be the optimal way to implement this feature. Any kind of input
> and discussions is most welcome.

Your post looks like an overall accurate description of the current
state of everything - with one exception, and that is Pandas. You say
you didn't look at the Pandas code yet, so that's not surprising.

You seem to assume that Pandas stores Python objects as elements of
DataFrames, but that isn't true. Pandas uses NumPy arrays instead. And
NumPy arrays are very different from standard Python objects, because
their internal data layout is by design the same as used in C or
Fortran. For a full description, see

  https://numpy.org/doc/stable/user/basics.rec.html

However, I am not sure you need to understand this in all detail, as I
am pretty sure that you do not want to copy this approach in Pharo.

The one point that does matter for you is where NumPy and Pandas take a
column's dtype from. The answer is that it's defined when a DataFrame is
created, and it cannot be changed afterwards. If a column is "integer",
it will remain "integer" forever. If you try to assign a string to an
element of such a column, you get an error message. When you create a
DataFrame from existing data, e.g. by reading a CSV file, Pandas scans
the data and determines a suitable dtype, much in the same way as V1.0
in Pharo/PolyMath did. But since Pandas doesn't allow any later change,
there is no serious performance issue.

So that's an option you can add to your list: define the dtypes once and
for all when the DataFrame is created. The main drawback is that you
would have to change the API for DataFrame creation to make this work.

Cheers,
  Konrad
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/0000-0003-0330-9428
Twitter: @khinsen
---------------------------------------------------------------------