How quantile works ?

I don't understand how quantile works.
When I do :

#(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4

I expect to have 2 and I have 5/2.
Werner, how it works ?

Regards,
--
Serge Stinckwich
UCBN & UMI UMMISCO 209 (IRD/UPMC)
Every DSL ends up being Smalltalk
http://www.doesnotunderstand.org/

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

No

#(0 1 2 3 4 5) asSortedCollection quantile: 1 / 4

returns 5/4 ...

On Thu, Dec 8, 2016 at 6:15 PM, Didier Besset <[hidden email]> wrote:
> This is correct. 2 would happen if numbers go from 0 to 5.
> Cheers,
> Didier
>
> Le 8 déc. 2016 17:45, "Serge Stinckwich" <[hidden email]> a
> écrit :
>
> I don't understand how quantile works.
> When I do :
>
> #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4
>
> I expect to have 2 and I have 5/2.
> Werner, how it works ?
>
> Regards,
> --
> Serge Stinckwich
> UCBN & UMI UMMISCO 209 (IRD/UPMC)
> Every DSL ends up being Smalltalk
> http://www.doesnotunderstand.org/
>
> --
> You received this message because you are subscribed to the Google Groups
> "SciSmalltalk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [hidden email].
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "SciSmalltalk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [hidden email].
> For more options, visit https://groups.google.com/d/optout.



--
Serge Stinckwich
UCBN & UMI UMMISCO 209 (IRD/UPMC)
Every DSL ends up being Smalltalk
http://www.doesnotunderstand.org/

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

I don't understand how quantile works.
When I do :

#(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4

I expect to have 2 and I have 5/2.
Werner, how it works ?

Regards,
--
Serge Stinckwich
UCBN & UMI UMMISCO 209 (IRD/UPMC)
Every DSL ends up being Smalltalk
http://www.doesnotunderstand.org/

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Hi Serge,
perhaps you can think about that result this way:
1/4 is a half of a half. the median of #(1 2 3 4 5 6 7) is 4. the median  of the first half of this Array (up to the median: #(1 2 3 4) ) is 5/2.
what this default method does, is _linearly_ interpolate between the numbers (hence #(0 1 2 3 4 5) asSortedCollection quantile: 1 / 4 --> 5/4), if you want to have a probability density without the steps in a histogram. for other purposes other quantile definitions are used, hence i implemented the most used methods in the general quantile method.
but for probability densities you get the correct linearly interpolated result this way. and if you dont want to linearly interpolate, you also would not accept 3/2 as the median of #(0 1 2 4).
werner

On Thu, Dec 8, 2016 at 5:44 PM, Serge Stinckwich <[hidden email]> wrote:

I don't understand how quantile works.
When I do :

#(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4

I expect to have 2 and I have 5/2.
Werner, how it works ?

Regards,
--
Serge Stinckwich
UCBN & UMI UMMISCO 209 (IRD/UPMC)
Every DSL ends up being Smalltalk
http://www.doesnotunderstand.org/

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

No

#(0 1 2 3 4 5) asSortedCollection quantile: 1 / 4

returns 5/4 ...

On Thu, Dec 8, 2016 at 6:15 PM, Didier Besset <[hidden email]> wrote:
> This is correct. 2 would happen if numbers go from 0 to 5.
> Cheers,
> Didier
>
> Le 8 déc. 2016 17:45, "Serge Stinckwich" <[hidden email]> a
> écrit :
>
> I don't understand how quantile works.
> When I do :
>
> #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4
>
> I expect to have 2 and I have 5/2.
> Werner, how it works ?
>
> Regards,
> --
> Serge Stinckwich
> UCBN & UMI UMMISCO 209 (IRD/UPMC)
> Every DSL ends up being Smalltalk
> http://www.doesnotunderstand.org/
>
> --
> You received this message because you are subscribed to the Google Groups
> "SciSmalltalk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [hidden email].
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "SciSmalltalk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [hidden email].
> For more options, visit https://groups.google.com/d/optout.



--
Serge Stinckwich
UCBN & UMI UMMISCO 209 (IRD/UPMC)
Every DSL ends up being Smalltalk
http://www.doesnotunderstand.org/

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

>Hmm, for me there are 6 elements in this collection, so 1/5 is suspect.

Hi Nicolas,
yes, for me the number of elements is also 6, but apart from that i have to admit i dont understand your point. lets say i want to calc the median of #(1 5), which would be the the quantile at 1/2:
numbers: #(1 5)
quantile:     0 1
at 1/2 you get 3.
or with 3 numbers you would need 1/2 steps, not 1/3 steps:
numbers: #(1 2   5)
quantile:     0 1/2 1
and the median is 2.

werner

Hi Nicolas,
perhaps my way of argumenting was too silly to deserve a reply <g>. let me try it again. lets say we have 3 elements #(0 1 2). you say one should use 1/3 steps:

ok, 0 happens in 1/3 of the cases. hence we have quantiles 1/3 2/3 3/3. it is my understanding that this is the default method in eg Mathematica. it is implemented as "quantile: whatever method: 'inverseCDF'"

or lets make it symmetrical, then we have quantiles 1/6 3/6 5/6. it is my understanding that this is the standard way in MathLab. it is implemented as 'hydrologist':
q:={0. (1/4). (1/2). (3/4). 1}.
q collect:[:x| #(0 1 2)asSortedCollection quantile:x method: 'hydrologist']. " {0. (1/4). 1. (7/4). 2}"

the standard paper (Rob J. Hyndman and Yanan Fan) about these things cited by everybody and his aunt can be found eg here:
https://www.amherst.edu/media/view/129116/original/Sample%2BQuantiles.pdf
its prefered way is "method:'medianBased'".

and it says that the method i used as default is used very often in statistical packages. looking around the internet i found this to be true, it seems to be the most often used method! i used 'hydrologist' for interquartilerange, since it seems that it is the standard way for this. but i chose 'modeBased' as the general default method not only because of its widespread use, but because i think it has an intuitive appeal, insofar as it is simple & easy to understand and additionally it returns the median in the usual way, and, i mean, everybody knows how to calc the median and understands that quantiles are in a way a generalization of the median and from this point it makes sense to me, i think eg of 1/4-quantile as the median of the median (i see, i'm already falling back into my usual silly stupid-user argumentation mode with unprecise formulations <g>, but i am a simple user).
werner

On Mon, Dec 12, 2016 at 1:43 PM, werner kassens <[hidden email]> wrote:

Hi Nicolas,

what way of calculating quantiles would make more sense as a default method? btw i would prefer it if the quantile function of a symmetric distribution stays symmetric?

werner

On Sat, Dec 10, 2016 at 3:25 PM, werner kassens <[hidden email]> wrote:

>Hmm, for me there are 6 elements in this collection, so 1/5 is suspect.

Hi Nicolas,
yes, for me the number of elements is also 6, but apart from that i have to admit i dont understand your point. lets say i want to calc the median of #(1 5), which would be the the quantile at 1/2:
numbers: #(1 5)
quantile:     0 1
at 1/2 you get 3.
or with 3 numbers you would need 1/2 steps, not 1/3 steps:
numbers: #(1 2   5)
quantile:     0 1/2 1
and the median is 2.

werner

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Hi Werner,

yes, I was seeing the mid-points separating the bins then realized that there were more than one way to estimate the CDF from experimental data...

Wikipedia givess 9 different ways at https://en.wikipedia.org/wiki/Quantile

2016-12-17 15:03 GMT+01:00 werner kassens <[hidden email]>:

Hi Nicolas,
perhaps my way of argumenting was too silly to deserve a reply <g>. let me try it again. lets say we have 3 elements #(0 1 2). you say one should use 1/3 steps:

ok, 0 happens in 1/3 of the cases. hence we have quantiles 1/3 2/3 3/3. it is my understanding that this is the default method in eg Mathematica. it is implemented as "quantile: whatever method: 'inverseCDF'"

or lets make it symmetrical, then we have quantiles 1/6 3/6 5/6. it is my understanding that this is the standard way in MathLab. it is implemented as 'hydrologist':
q:={0. (1/4). (1/2). (3/4). 1}.
q collect:[:x| #(0 1 2)asSortedCollection quantile:x method: 'hydrologist']. " {0. (1/4). 1. (7/4). 2}"

the standard paper (Rob J. Hyndman and Yanan Fan) about these things cited by everybody and his aunt can be found eg here:
https://www.amherst.edu/media/view/129116/original/Sample%2BQuantiles.pdf
its prefered way is "method:'medianBased'".

and it says that the method i used as default is used very often in statistical packages. looking around the internet i found this to be true, it seems to be the most often used method! i used 'hydrologist' for interquartilerange, since it seems that it is the standard way for this. but i chose 'modeBased' as the general default method not only because of its widespread use, but because i think it has an intuitive appeal, insofar as it is simple & easy to understand and additionally it returns the median in the usual way, and, i mean, everybody knows how to calc the median and understands that quantiles are in a way a generalization of the median and from this point it makes sense to me, i think eg of 1/4-quantile as the median of the median (i see, i'm already falling back into my usual silly stupid-user argumentation mode with unprecise formulations <g>, but i am a simple user).
werner

On Mon, Dec 12, 2016 at 1:43 PM, werner kassens <[hidden email]> wrote:

Hi Nicolas,

what way of calculating quantiles would make more sense as a default method? btw i would prefer it if the quantile function of a symmetric distribution stays symmetric?

werner

On Sat, Dec 10, 2016 at 3:25 PM, werner kassens <[hidden email]> wrote:

>Hmm, for me there are 6 elements in this collection, so 1/5 is suspect.

Hi Nicolas,
yes, for me the number of elements is also 6, but apart from that i have to admit i dont understand your point. lets say i want to calc the median of #(1 5), which would be the the quantile at 1/2:
numbers: #(1 5)
quantile:     0 1
at 1/2 you get 3.
or with 3 numbers you would need 1/2 steps, not 1/3 steps:
numbers: #(1 2   5)
quantile:     0 1/2 1
and the median is 2.

werner

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

On Sun, Dec 18, 2016 at 2:49 PM, werner kassens <[hidden email]> wrote:
> Hi,
> since Nicolas mentioned wikipedias quantile page, just a short comment: i
> occasionally base a program on a wikipedia page or often mention them in a
> doku, iow i generally like them and of course i read that page before i
> implemented quantile, but in this case the page is in my opinion not up to
> wikipedias usual standard and i did _not_ use it. especially the correctness
> of this sentence "The following two examples use the Nearest Rank definition
> of quantile with rounding. For an explanation of this definition, see
> percentiles." is eventually debatable, which makes these examples there -
> uhmm well - not too useful.
> for a substitute i used MathWorlds page, on which i based the
> implementation:
> http://mathworld.wolfram.com/Quantile.html

yes I know the page and this why I ask the question about quantile at
the beginning.
If you see the Alpha code:
http://reference.wolfram.com/language/ref/Quantile.html
this really different of what you are doing actually.

--
Serge Stinckwich
UCBN & UMI UMMISCO 209 (IRD/UPMC)
Every DSL ends up being Smalltalk
http://www.doesnotunderstand.org/

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

On Thu, Dec 8, 2016 at 5:44 PM, Serge Stinckwich
<[hidden email]> wrote:
> I don't understand how quantile works.
> When I do :
>
> #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4
>
> I expect to have 2 and I have 5/2.

Ok, I was able to reproduce the same results than Mathematica :
http://reference.wolfram.com/language/ref/Quantile.html
with the inverseCDF parameter:

#(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 method: 'inverseCDF'

Mathematica use inverseCDF by default and you use modeBase instead.
Which one is better to use as default ?

I see there is 8 different methods:
'modeBased', 'inverseCDF', 'closestObservation', 'California' ,
'hydrologist' , 'Weibull', 'medianBased', 'normalDistribution'
Where are the names are coming from ? I could not find any reference about that.

Instead of using String for parameters, I think this is better to use
a symbols to have something like:

#(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 method: #inverseCDF

I will do some refactoring to go in this direction if you don't mind.

Regards,
--
Serge Stinckwich
UCBN & UMI UMMISCO 209 (IRD/UPMC)
Every DSL ends up being Smalltalk
http://www.doesnotunderstand.org/

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Hi again Serge,
>Where are the names are coming from ? I could not find any reference about that.
same time has already passed since i made that package, but i think i mainly used Mathematica notation and certainly did not want to type too long names.

>Mathematica use inverseCDF by default and you use modeBase instead.
>Which one is better to use as default ?
i am certainly the wrong person to ask. you can take a look at the paper i mentioned earlier, most math programs cite it (mathematica, maple, julia, r, national institute of standards & technology etc). the paper tests methods for different quality criteria: inverseCDF is not symmetric (the median criterium in the paper is also not fulfilled). i made modebased the default because its method made sense to me <g> and i like to use it. i <g>like also medianbased, but i think it has a somehow counterintuitive look (it is not used often as a default, i remember only one case).

>I will do some refactoring to go in this direction if you don't mind.
yes of course, please do that.
werner

On Mon, Dec 19, 2016 at 11:27 AM, Serge Stinckwich <[hidden email]> wrote:

On Thu, Dec 8, 2016 at 5:44 PM, Serge Stinckwich
<[hidden email]> wrote:
> I don't understand how quantile works.
> When I do :
>
> #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4
>
> I expect to have 2 and I have 5/2.

Ok, I was able to reproduce the same results than Mathematica :
http://reference.wolfram.com/language/ref/Quantile.html
with the inverseCDF parameter:

#(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 method: 'inverseCDF'

Mathematica use inverseCDF by default and you use modeBase instead.
Which one is better to use as default ?

I see there is 8 different methods:
'modeBased', 'inverseCDF', 'closestObservation', 'California' ,
'hydrologist' , 'Weibull', 'medianBased', 'normalDistribution'
Where are the names are coming from ? I could not find any reference about that.

Instead of using String for parameters, I think this is better to use
a symbols to have something like:

#(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 method: #inverseCDF

I will do some refactoring to go in this direction if you don't mind.

Regards,
--
Serge Stinckwich
UCBN & UMI UMMISCO 209 (IRD/UPMC)
Every DSL ends up being Smalltalk
http://www.doesnotunderstand.org/

--
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.