I don't understand how quantile works.
When I do : #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 I expect to have 2 and I have 5/2. Werner, how it works ? Regards, -- Serge Stinckwich UCBN & UMI UMMISCO 209 (IRD/UPMC) Every DSL ends up being Smalltalk http://www.doesnotunderstand.org/ -- You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
This is correct. 2 would happen if numbers go from 0 to 5. Cheers, Didier Le 8 déc. 2016 17:45, "Serge Stinckwich" <[hidden email]> a écrit : I don't understand how quantile works. You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
No
#(0 1 2 3 4 5) asSortedCollection quantile: 1 / 4 returns 5/4 ... On Thu, Dec 8, 2016 at 6:15 PM, Didier Besset <[hidden email]> wrote: > This is correct. 2 would happen if numbers go from 0 to 5. > Cheers, > Didier > > Le 8 déc. 2016 17:45, "Serge Stinckwich" <[hidden email]> a > écrit : > > I don't understand how quantile works. > When I do : > > #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 > > I expect to have 2 and I have 5/2. > Werner, how it works ? > > Regards, > -- > Serge Stinckwich > UCBN & UMI UMMISCO 209 (IRD/UPMC) > Every DSL ends up being Smalltalk > http://www.doesnotunderstand.org/ > > -- > You received this message because you are subscribed to the Google Groups > "SciSmalltalk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [hidden email]. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "SciSmalltalk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [hidden email]. > For more options, visit https://groups.google.com/d/optout. -- Serge Stinckwich UCBN & UMI UMMISCO 209 (IRD/UPMC) Every DSL ends up being Smalltalk http://www.doesnotunderstand.org/ -- You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
I have to look into this when I am back home.... Cheers, Didier Le 8 déc. 2016 20:04, "Serge Stinckwich" <[hidden email]> a écrit : No You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
This is not your code. Werner I guess.
But you can have a look :-) On Fri, Dec 9, 2016 at 8:33 AM, Didier Besset <[hidden email]> wrote: > I have to look into this when I am back home.... > Cheers, > Didier > > Le 8 déc. 2016 20:04, "Serge Stinckwich" <[hidden email]> a > écrit : > > No > > #(0 1 2 3 4 5) asSortedCollection quantile: 1 / 4 > > returns 5/4 ... > > On Thu, Dec 8, 2016 at 6:15 PM, Didier Besset <[hidden email]> > wrote: >> This is correct. 2 would happen if numbers go from 0 to 5. >> Cheers, >> Didier >> >> Le 8 déc. 2016 17:45, "Serge Stinckwich" <[hidden email]> a >> écrit : >> >> I don't understand how quantile works. >> When I do : >> >> #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 >> >> I expect to have 2 and I have 5/2. >> Werner, how it works ? >> >> Regards, >> -- >> Serge Stinckwich >> UCBN & UMI UMMISCO 209 (IRD/UPMC) >> Every DSL ends up being Smalltalk >> http://www.doesnotunderstand.org/ >> >> -- >> You received this message because you are subscribed to the Google Groups >> "SciSmalltalk" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [hidden email]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "SciSmalltalk" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [hidden email]. >> For more options, visit https://groups.google.com/d/optout. > > > > -- > Serge Stinckwich > UCBN & UMI UMMISCO 209 (IRD/UPMC) > Every DSL ends up being Smalltalk > http://www.doesnotunderstand.org/ > > -- > You received this message because you are subscribed to the Google Groups > "SciSmalltalk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [hidden email]. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "SciSmalltalk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [hidden email]. > For more options, visit https://groups.google.com/d/optout. -- Serge Stinckwich UCBN & UMI UMMISCO 209 (IRD/UPMC) Every DSL ends up being Smalltalk http://www.doesnotunderstand.org/ -- You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
In reply to this post by SergeStinckwich
Hi Serge, perhaps you can think about that result this way: 1/4 is a half of a half. the median of #(1 2 3 4 5 6 7) is 4. the median of the first half of this Array (up to the median: #(1 2 3 4) ) is 5/2. what this default method does, is _linearly_ interpolate between the numbers (hence #(0 1 2 3 4 5) asSortedCollection quantile: 1 / 4 --> 5/4), if you want to have a probability density without the steps in a histogram. for other purposes other quantile definitions are used, hence i implemented the most used methods in the general quantile method. but for probability densities you get the correct linearly interpolated result this way. and if you dont want to linearly interpolate, you also would not accept 3/2 as the median of #(0 1 2 4). werner On Thu, Dec 8, 2016 at 5:44 PM, Serge Stinckwich <[hidden email]> wrote: I don't understand how quantile works. You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
oops, replace probability density with cumulative density wernerOn Fri, Dec 9, 2016 at 2:30 PM, werner kassens <[hidden email]> wrote:
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
In reply to this post by SergeStinckwich
this is interpolated like the median of #(1 2 3 4) as 2.5: numbers: #(0 1 2 3 4 5)quantiles: 0 1/5 2/5 3/5 4/5 1 1/4 lies between 1/5 & 2/5 1/4-1/5=1/20 2/5-1/4=3/20 hence 1/4 lies a quarter above 1 and 3 quarters below 2, iow 5/4. werner On Thu, Dec 8, 2016 at 8:03 PM, Serge Stinckwich <[hidden email]> wrote: No You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
2016-12-10 11:24 GMT+01:00 werner kassens <[hidden email]>:
Hmm, for me there are 6 elements in this collection, so 1/5 is suspect. For example, 1/6th of the elements are below Float fmin, 1/3 below 1.0 successor.
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
>Hmm, for me there are 6 elements in this collection, so 1/5 is suspect. Hi Nicolas, yes, for me the number of elements is also 6, but apart from that i have to admit i dont understand your point. lets say i want to calc the median of #(1 5), which would be the the quantile at 1/2: numbers: #(1 5) quantile: 0 1 at 1/2 you get 3. or with 3 numbers you would need 1/2 steps, not 1/3 steps: numbers: #(1 2 5) quantile: 0 1/2 1 and the median is 2. werner You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
Hi Nicolas, what way of calculating quantiles would make more sense as a default method? btw i would prefer it if the quantile function of a symmetric distribution stays symmetric?On Sat, Dec 10, 2016 at 3:25 PM, werner kassens <[hidden email]> wrote:
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
Hi Nicolas,
-- perhaps my way of argumenting was too silly to deserve a reply <g>. let me try it again. lets say we have 3 elements #(0 1 2). you say one should use 1/3 steps: ok, 0 happens in 1/3 of the cases. hence we have quantiles 1/3 2/3 3/3. it is my understanding that this is the default method in eg Mathematica. it is implemented as "quantile: whatever method: 'inverseCDF'" or lets make it symmetrical, then we have quantiles 1/6 3/6 5/6. it is my understanding that this is the standard way in MathLab. it is implemented as 'hydrologist': q:={0. (1/4). (1/2). (3/4). 1}. q collect:[:x| #(0 1 2)asSortedCollection quantile:x method: 'hydrologist']. " {0. (1/4). 1. (7/4). 2}" the standard paper (Rob J. Hyndman and Yanan Fan) about these things cited by everybody and his aunt can be found eg here: https://www.amherst.edu/media/view/129116/original/Sample%2BQuantiles.pdf its prefered way is "method:'medianBased'". and it says that the method i used as default is used very often in statistical packages. looking around the internet i found this to be true, it seems to be the most often used method! i used 'hydrologist' for interquartilerange, since it seems that it is the standard way for this. but i chose 'modeBased' as the general default method not only because of its widespread use, but because i think it has an intuitive appeal, insofar as it is simple & easy to understand and additionally it returns the median in the usual way, and, i mean, everybody knows how to calc the median and understands that quantiles are in a way a generalization of the median and from this point it makes sense to me, i think eg of 1/4-quantile as the median of the median (i see, i'm already falling back into my usual silly stupid-user argumentation mode with unprecise formulations <g>, but i am a simple user). werner On Mon, Dec 12, 2016 at 1:43 PM, werner kassens <[hidden email]> wrote:
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
Hi Werner, yes, I was seeing the mid-points separating the bins then realized that there were more than one way to estimate the CDF from experimental data...2016-12-17 15:03 GMT+01:00 werner kassens <[hidden email]>:
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
Hi, since Nicolas mentioned wikipedias quantile page, just a short comment: i occasionally base a program on a wikipedia page or often mention them in a doku, iow i generally like them and of course i read that page before i implemented quantile, but in this case the page is in my opinion not up to wikipedias usual standard and i did _not_ use it. especially the correctness of this sentence "The following two examples use the Nearest Rank definition of quantile with rounding. For an explanation of this definition, see percentiles." is eventually debatable, which makes these examples there - uhmm well - not too useful. for a substitute i used MathWorlds page, on which i based the implementation: http://mathworld.wolfram.com/Quantile.html werner On Sat, Dec 17, 2016 at 5:08 PM, Nicolas Cellier <[hidden email]> wrote:
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
On Sun, Dec 18, 2016 at 2:49 PM, werner kassens <[hidden email]> wrote:
> Hi, > since Nicolas mentioned wikipedias quantile page, just a short comment: i > occasionally base a program on a wikipedia page or often mention them in a > doku, iow i generally like them and of course i read that page before i > implemented quantile, but in this case the page is in my opinion not up to > wikipedias usual standard and i did _not_ use it. especially the correctness > of this sentence "The following two examples use the Nearest Rank definition > of quantile with rounding. For an explanation of this definition, see > percentiles." is eventually debatable, which makes these examples there - > uhmm well - not too useful. > for a substitute i used MathWorlds page, on which i based the > implementation: > http://mathworld.wolfram.com/Quantile.html yes I know the page and this why I ask the question about quantile at the beginning. If you see the Alpha code: http://reference.wolfram.com/language/ref/Quantile.html this really different of what you are doing actually. -- Serge Stinckwich UCBN & UMI UMMISCO 209 (IRD/UPMC) Every DSL ends up being Smalltalk http://www.doesnotunderstand.org/ -- You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
In reply to this post by SergeStinckwich
On Thu, Dec 8, 2016 at 5:44 PM, Serge Stinckwich
<[hidden email]> wrote: > I don't understand how quantile works. > When I do : > > #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 > > I expect to have 2 and I have 5/2. Ok, I was able to reproduce the same results than Mathematica : http://reference.wolfram.com/language/ref/Quantile.html with the inverseCDF parameter: #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 method: 'inverseCDF' Mathematica use inverseCDF by default and you use modeBase instead. Which one is better to use as default ? I see there is 8 different methods: 'modeBased', 'inverseCDF', 'closestObservation', 'California' , 'hydrologist' , 'Weibull', 'medianBased', 'normalDistribution' Where are the names are coming from ? I could not find any reference about that. Instead of using String for parameters, I think this is better to use a symbols to have something like: #(1 2 3 4 5 6 7) asSortedCollection quantile: 1 / 4 method: #inverseCDF I will do some refactoring to go in this direction if you don't mind. Regards, -- Serge Stinckwich UCBN & UMI UMMISCO 209 (IRD/UPMC) Every DSL ends up being Smalltalk http://www.doesnotunderstand.org/ -- You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
In reply to this post by SergeStinckwich
Hi Serge, wernerif it is different, i would say my code is buggy. On Mon, Dec 19, 2016 at 8:37 AM, Serge Stinckwich <[hidden email]> wrote: On Sun, Dec 18, 2016 at 2:49 PM, werner kassens <[hidden email]> wrote: You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
In reply to this post by SergeStinckwich
Hi again Serge, >Where are the names are coming from ? I could not find any reference about that. same time has already passed since i made that package, but i think i mainly used Mathematica notation and certainly did not want to type too long names. >Mathematica use inverseCDF by default and you use modeBase instead. >Which one is better to use as default ? i am certainly the wrong person to ask. you can take a look at the paper i mentioned earlier, most math programs cite it (mathematica, maple, julia, r, national institute of standards & technology etc). the paper tests methods for different quality criteria: inverseCDF is not symmetric (the median criterium in the paper is also not fulfilled). i made modebased the default because its method made sense to me <g> and i like to use it. i <g>like also medianbased, but i think it has a somehow counterintuitive look (it is not used often as a default, i remember only one case). >I will do some refactoring to go in this direction if you don't mind. yes of course, please do that. werner On Mon, Dec 19, 2016 at 11:27 AM, Serge Stinckwich <[hidden email]> wrote: On Thu, Dec 8, 2016 at 5:44 PM, Serge Stinckwich You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
...and inverseCDF is not continuous. wernerOn Mon, Dec 19, 2016 at 4:27 PM, werner kassens <[hidden email]> wrote:
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
...and Mathematica is a very capable program that can deal with discontinuities extremely well wernerOn Mon, Dec 19, 2016 at 5:51 PM, werner kassens <[hidden email]> wrote:
You received this message because you are subscribed to the Google Groups "SciSmalltalk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. |
Free forum by Nabble - Scala forum | Edit this page |