the reasons for ranking
Locked
 
|
Hi all,
Let's talk more directly about the reasons for ranking . . . early today I wrote: >> It also gives us more colored dots. And Yoshiki wrote: > And the explanation of why this is a good thing is not described > yet. Kathleen wasn't happy with it and I think your response wasn't > answering her concern. I'll try to be clearer. As I see things, there are three related issues that are getting combined in the same conversation: 1. showing individual rankings (which is what I think she's most concerned about ... google "Punished by Rewards") 2. showing levels of ongoing community involvement (aka, "top ten contributors") 3. sorting things so the good stuff stays on top and the lesser stuff falls to the bottom The colored dots next to people's usernames are really about #2. You can get points by voting, by commenting, by submitting projects, etc. It is *not* merely a measure of your project vote points. It's not an average either. One person who submits ten projects that gets 5 points each will earn 50 points, which is the same as another person who submits two projects that get 25 points each. There's no way to tell which person got 25 points apiece and which got 5 points apiece. In a normal grading situation, the 25 pointer would clearly "win out" over the 5 pointer. The Squeakland showcase measures *sustained effort*, not specific performance. Displaying the colored dots is a motivational thing. When someone sees a comment by "yoshiki with the purple circle", they can see you're an Etoys enthusiast with a history. It's not "Yoshiki who get's straight A's", it's "Yoshiki who's put in a lot of effort over time and has earned that purple circle." Seeing the purple circle is a reward to you for effort, and a signal to others that your opinion might matter more than "puppetAccount31252" with a white circle. The colored circles are a measure of credibility. Which leads to the most important reason for them .... people with higher levels get asked to rank other projects. They get asked because they've earned the right to have a say in what's valuable. Some important things to keep in mind: * no one ever sees your votes * no one ever knows how their project was ranked, only that it places higher or lower than another project * there are no displays of "raw points" anywhere ... only relative levels. Today the top points is 362. In a year, it might be 23,252. No one ever sees this. As for the more general question of why do we even want to rank projects (#3), the answer is quite plain .... because when there's 250,000 projects on the server, it'll be a big mess that's useful to no one unless there's some ordering that's generally useful. The Squeakland ranking system is an attempt to sort with a measure of actual worth, not mere popularity. The way to gauge this worth should be tied directly to sustained merit. Tim _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: the reasons for ranking
|
|
At Tue, 29 Sep 2009 20:24:39 -0400,
Timothy Falconer wrote: > > Hi all, > > Let's talk more directly about the reasons for ranking . . . early > today I wrote: > > >> It also gives us more colored dots. > > And Yoshiki wrote: > > > And the explanation of why this is a good thing is not described > > yet. Kathleen wasn't happy with it and I think your response wasn't > > answering her concern. > > I'll try to be clearer. There were more in the discussion... > As I see things, there are three related issues that are getting > combined in the same conversation: > > 1. showing individual rankings (which is what I think she's most > concerned about ... google "Punished by Rewards") > > 2. showing levels of ongoing community involvement (aka, "top ten > contributors") > > 3. sorting things so the good stuff stays on top and the lesser stuff > falls to the bottom > > The colored dots next to people's usernames are really about #2. You > can get points by voting, by commenting, by submitting projects, etc. > It is *not* merely a measure of your project vote points. It's not an > average either. One person who submits ten projects that gets 5 > points each will earn 50 points, which is the same as another person > who submits two projects that get 25 points each. There's no way to > tell which person got 25 points apiece and which got 5 points apiece. > In a normal grading situation, the 25 pointer would clearly "win out" > over the 5 pointer. The Squeakland showcase measures *sustained > effort*, not specific performance. > > Displaying the colored dots is a motivational thing. When someone > sees a comment by "yoshiki with the purple circle", they can see > you're an Etoys enthusiast with a history. It's not "Yoshiki who > get's straight A's", it's "Yoshiki who's put in a lot of effort over > time and has earned that purple circle." Seeing the purple circle is > a reward to you for effort, and a signal to others that your opinion > might matter more than "puppetAccount31252" with a white circle. The > colored circles are a measure of credibility. But one of the concerns was that one can lose the colored dot visibly to everybody even when you are doing as much as you have been. When others are saying that "that is de-motivating", you just repeat yourself to say "it is a motivational thing". As Kathleen repeated, coercing people onto a linear scale is not the way to motivate different kind of learners. The external "reward" for some effort in learning settings should be other people's honest and insightful feedback and getting more ideas (and internally "learn" something), but not a dot. Your #2 above is not separated from #1, and the 5-6 levels of dots have way more meaning than "top ten contributors". #3 doesn't mean you need to have a visible dots for individual persons, but merely projects should be sorted. > Which leads to the most important reason for them .... people with > higher levels get asked to rank other projects. They get asked > because they've earned the right to have a say in what's valuable. But that doesn't mean that person's dot should be visible to everybody. > Some important things to keep in mind: > > * no one ever sees your votes > * no one ever knows how their project was ranked, only that it places > higher or lower than another project > * there are no displays of "raw points" anywhere ... only relative > levels. Today the top points is 362. In a year, it might be > 23,252. No one ever sees this. Well, I know that. > As for the more general question of why do we even want to rank > projects (#3), the answer is quite plain .... > > because when there's 250,000 projects on the server, it'll be a big > mess that's useful to no one unless there's some ordering that's > generally useful. > > The Squeakland ranking system is an attempt to sort with a measure of > actual worth, not mere popularity. The way to gauge this worth should > be tied directly to sustained merit. This statement is not an answer to why people should be visibly ranked, and they should lose the rank easily. -- Yoshiki _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: the reasons for ranking
|
|
On Sep 29, 2009, at 9:08 PM, Yoshiki Ohshima wrote: > At Tue, 29 Sep 2009 20:24:39 -0400, > Timothy Falconer wrote: >> >> Hi all, >> >> Let's talk more directly about the reasons for ranking . . . early >> today I wrote: >> >>>> It also gives us more colored dots. >> >> And Yoshiki wrote: >> >>> And the explanation of why this is a good thing is not described >>> yet. Kathleen wasn't happy with it and I think your response wasn't >>> answering her concern. >> >> I'll try to be clearer. > > There were more in the discussion... > >> As I see things, there are three related issues that are getting >> combined in the same conversation: >> >> 1. showing individual rankings (which is what I think she's most >> concerned about ... google "Punished by Rewards") >> >> 2. showing levels of ongoing community involvement (aka, "top ten >> contributors") >> >> 3. sorting things so the good stuff stays on top and the lesser stuff >> falls to the bottom >> >> The colored dots next to people's usernames are really about #2. >> You >> can get points by voting, by commenting, by submitting projects, etc. >> It is *not* merely a measure of your project vote points. It's not >> an >> average either. One person who submits ten projects that gets 5 >> points each will earn 50 points, which is the same as another person >> who submits two projects that get 25 points each. There's no way to >> tell which person got 25 points apiece and which got 5 points apiece. >> In a normal grading situation, the 25 pointer would clearly "win out" >> over the 5 pointer. The Squeakland showcase measures *sustained >> effort*, not specific performance. >> >> Displaying the colored dots is a motivational thing. When someone >> sees a comment by "yoshiki with the purple circle", they can see >> you're an Etoys enthusiast with a history. It's not "Yoshiki who >> get's straight A's", it's "Yoshiki who's put in a lot of effort over >> time and has earned that purple circle." Seeing the purple circle >> is >> a reward to you for effort, and a signal to others that your opinion >> might matter more than "puppetAccount31252" with a white circle. >> The >> colored circles are a measure of credibility. > > But one of the concerns was that one can lose the colored dot > visibly to everybody even when you are doing as much as you have been. > When others are saying that "that is de-motivating", you just repeat > yourself to say "it is a motivational thing". As Kathleen repeated, > coercing people onto a linear scale is not the way to motivate > different kind of learners. I think the piece that's missing here is that it will be pretty rare to actually change from one level to another, especially for younger children. The way it is now will not be the way it is with 1000 users. Color changes won't be a day-to-day goal. My guess it might happen once or twice a year for an individual. Again, I doubt many young children will even make it to level 3. > The external "reward" for some effort in learning settings should > be other people's honest and insightful feedback and getting more > ideas (and internally "learn" something), but not a dot. > > Your #2 above is not separated from #1, and the 5-6 levels of dots > have way more meaning than "top ten contributors". #3 doesn't mean > you need to have a visible dots for individual persons, but merely > projects should be sorted. This is the central issue .... #1 is completely different than #2. When someone can get just as many points by adding comments as getting "grades", it's entirely about #2. It's a measure of effort, not excellence. High ranks merely get someone more points more quickly, and rightly so. >> Which leads to the most important reason for them .... people with >> higher levels get asked to rank other projects. They get asked >> because they've earned the right to have a say in what's valuable. > > But that doesn't mean that person's dot should be visible to > everybody. Yes, we could take the colored circles off the website. I think that people are finding them fun, at least the people I'm talking to, and I think there's genuine value in knowing that a commenter is a regular and not a puppet account. I talked about the whole "Punished by Rewards" thing with a few dozen teachers a year ago when Kathleen first brought up her objections to the Waveplace Awards. She honestly was the only that I talked to that voiced any concern over that event. Quite the opposite, most saw it as a transforming moment for their kids. From what we could tell, the kids who didn't place took it in stride and were proud of their friends. Watch the end of the video to see me address this topic directly, particularly why we thought the awards were a good thing. I agree that competition goes too far in this society. I agree that there are different kinds of learners that thrive with different types of motivation. But I humbly disagree with yours and Kathleen's opinion that we should take the colored dots off the website. To me, the benefits outweigh the potential downside. They're useful and fun, with a real purpose. Tim _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: the reasons for ranking
|
|
At Tue, 29 Sep 2009 22:45:08 -0400,
Timothy Falconer wrote: > > I think the piece that's missing here is that it will be pretty rare > to actually change from one level to another, especially for younger > children. Is this based on the logic described at http://tracker.squeakland.org/browse/SQ-361? It would be a bit surprizing to hear because "maxMojo" change alone affects everybody's rank. Since the owner of maxMojo probably will earn more points quicker than others, and the ranks are linear scale of the point, I would expect to see bigger disparity over time. > Yes, we could take the colored circles off the website. I think that > people are finding them fun, at least the people I'm talking to, and I > think there's genuine value in knowing that a commenter is a regular > and not a puppet account. Well, people should actually look at the contents to judge the quality of comments. I've been writing fairly stupid comments. They should be treated so. > I talked about the whole "Punished by Rewards" thing with a few dozen > teachers a year ago when Kathleen first brought up her objections to > the Waveplace Awards. She honestly was the only that I talked to > that voiced any concern over that event. Quite the opposite, most saw > it as a transforming moment for their kids. From what we could tell, > the kids who didn't place took it in stride and were proud of their > friends. Watch the end of the video to see me address this topic > directly, particularly why we thought the awards were a good thing. Of course, the question occured to me that how they knew how it is going to work a year ago... > I agree that competition goes too far in this society. I agree that > there are different kinds of learners that thrive with different types > of motivation. > > But I humbly disagree with yours and Kathleen's opinion that we should > take the colored dots off the website. To me, the benefits outweigh > the potential downside. They're useful and fun, with a real purpose. Okay. -- Yoshiki _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: the reasons for ranking
|
|
At Tue, 29 Sep 2009 20:08:46 -0700,
Yoshiki Ohshima wrote: > > > I agree that competition goes too far in this society. I agree that > > there are different kinds of learners that thrive with different types > > of motivation. > > > > But I humbly disagree with yours and Kathleen's opinion that we should > > take the colored dots off the website. To me, the benefits outweigh > > the potential downside. They're useful and fun, with a real purpose. > > Okay. Oh, but I forgot to mention that one of my suggestion wasn't remove it, but have probably two levels thing and no going back in usual cases. It can even be visible on the site if the rule is clear. So, it is not "removing the dots or not" question but "how many levels" "how it is calculated", and etc. -- Yoshiki _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: the reasons for ranking
|
|
In reply to this post by Yoshiki Ohshima-2
On Sep 29, 2009, at 11:08 PM, Yoshiki Ohshima wrote: > At Tue, 29 Sep 2009 22:45:08 -0400, > Timothy Falconer wrote: >> >> I think the piece that's missing here is that it will be pretty rare >> to actually change from one level to another, especially for younger >> children. > > Is this based on the logic described at > http://tracker.squeakland.org/browse/SQ-361? It would be a bit > surprizing to hear because "maxMojo" change alone affects everybody's > rank. Since the owner of maxMojo probably will earn more points > quicker than others, and the ranks are linear scale of the point, I > would expect to see bigger disparity over time. Some of the details have changed since that list, but yes, the mojoLevel formula is current. Over time, some people will continue to contribute and others will stop. The formula rewards sustained effort, not past performance. Again, this is not a grade ... it's a thank you. >> Yes, we could take the colored circles off the website. I think >> that >> people are finding them fun, at least the people I'm talking to, >> and I >> think there's genuine value in knowing that a commenter is a regular >> and not a puppet account. > > Well, people should actually look at the contents to judge the > quality of comments. I've been writing fairly stupid comments. They > should be treated so. Originally we were going to rank comments, but recently I decided that would be just too much for people to have to do. Comment points are a recognition of effort ... not a grade, but a thank you. When comment scribbling gets turned on (see the tracker), we'll have a way to prevent bogus comments from gaming the system. _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
|
|
In reply to this post by Timothy Falconer-3
Hello all,
I apologize for joining this conversation so late; I am a longtime, though recently quiet, member of the community. I understand that the ranking system is an attempt to enable people to judge the quality of a contribution (or contributor) based on some directly observable measure of reputation. On first sight, one could argue that such a system overcomes the problem that in the face-to-face world reputation is not directly observable. In order to get an honest assessment of a person's reputation, one has to invest a lot of time building trust with the people familiar with that person. Even then, a trustworthy assessment requires direct observation of the person's actions. A ranking system would appear to reduce that effort by half; knowing the person's reputation among his/her peers, one only needs to assess the work. The problem with that reasoning is that electronic ranking systems are highly susceptible to manipulation. Building reputation becomes the goal of the activity for many people and they use all sorts of seemingly harmless social and technological means to inflate their numbers. Our lab has been studying this phenomenon through both observational studies of online communities and laboratory experiments. The two papers below report on the phenomenon as it presents itself in Digg, the news aggregation site. The first paper demonstrates that a tit-for-tat game of reciprocity inflates the reputation of contributors and their contributions without reflecting anything substantive about their contributions. The second paper really brings out the negative consequences of this phenomenon. The paper reports on an experiment where people judged how interesting they found the contribution. The ranking values of the articles were set by the investigator; sometime the rank of the article was set high, at other times low. Experimental subjects rated higher-ranked contributions as more interesting than lower ranked contributions. The same article was rated as highly interesting when its rank was set high and uninteresting when ranked low. Duncan Watts (of small-world networks fame) observed the same phenomenon with music rating. What this means in the present discussion is that people will likely ignore low ranking contributions. Worse still, when they do actually look at those contributions they are likely to see what the ranking value led them to expect rather than the qualities of the contribution itself. Unless we can find scientific research that demonstrates any benefits to ranking, I think we should be wary of using such systems. All best, John Sadlon, E., Sakamoto, Y., Dever, H. J., & Nickerson, J. V. (2008). The karma of Digg: Reciprocity in online social networks. In R. Gopal and R. Ramesh (Eds.), Proceedings of the 18th Annual Workshop on Information Technologies and Systems. http://cog.mgnt.stevens-tech.edu/~yasu/papers/reciprocity.pdf Sakamoto, Y., Ma, J., & Nickerson, J. V. (2009). 2377 people like this article: The influence of others' decisions on yours. In N. Taatgen, H. van Rijn, L. Schomaker, and J. Nerbonne (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society. http://cog.mgnt.stevens-tech.edu/~yasu/papers/cogscidigg1.pdf Salganik, M. J., Dodds, P. S., and Watts, D. J. (2006). Experimental study of inequality and unpredictability in an artificial cultural market. Science, 311(5762):854-856. http://dx.doi.org/10.1126/science.1121066[/url] -------------------- m2f -------------------- (from forum) http://squeakland.org/forums/viewtopic.php?p=11554#11554 -------------------- m2f -------------------- _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
|
|
Hey, John,
Nice to see you are still around and interested -- hope you are doing well. Thanks for the comments. As for me and Viewpoints -- there is one reason only for ranking and that is filtering in an attempt to bring the best educational examples and those illustrative of etoys' strengths to the attention of the community. So, I, for one, am hoping that those "voting" are not voting for any particular child, author or person, but for the example itself -- I hope those voting/ranking are asking "is this a fine exemplar to help teach a concept, principle, or powerful idea?". The more the example answers back with a "yes" the higher the rank. Also, the more complete an example the higher the rank. How disclosing of what the Etoy is meant to do is in the example? Does it have an explanation of what it is? If it is a game does it have instruction on how to play and the goal of the game? I hope that teachers who encourage their students to upload projects will *only* allow sharing if their student provides an "About" flap, or some intro/explanation of the Etoy and some instruction on where to start, the aim of the project, etc. The most beautiful simulation of something will fall completely flat with someone if they have no real idea of what they are looking at or what the script they are playing with is meant to do. Etoys are intended to teach and help us learn; learning cannot occur without context. No author should assume their project will be self disclosing; it will not. Our users have a variety of levels of expertise in both Etoys and subject matter areas; providing more context can only help. If there is no "ranking" in place and everything is posted the offerings soon get overwhelming and most difficult to navigate. We found this years ago when we had an active "super-swiki". Projects that were no more than a single sketch with no script at all were posted and only "muddied the waters" for those wading in an attempt to find something with some meaningful content. I hope the community will bear this in mind when "ranking" what gets uploaded and understand that not everyone can be represented in a featured showcase. -- Kim On Sep 30, 2009, at 7:58 AM, voiklis wrote: > Hello all, > > I apologize for joining this conversation so late; I am a longtime, > though recently quiet, member of the community. > > I understand that the ranking system is an attempt to enable people > to judge the quality of a contribution (or contributor) based on > some directly observable measure of reputation. On first sight, one > could argue that such a system overcomes the problem that in the > face-to-face world reputation is not directly observable. In order > to get an honest assessment of a person's reputation, one has to > invest a lot of time building trust with the people familiar with > that person. Even then, a trustworthy assessment requires direct > observation of the person's actions. > > A ranking system would appear to reduce that effort by half; knowing > the person's reputation among his/her peers, one only needs to > assess the work. > > The problem with that reasoning is that electronic ranking systems > are highly susceptible to manipulation. Building reputation becomes > the goal of the activity for many people and they use all sorts of > seemingly harmless social and technological means to inflate their > numbers. Our lab has been studying this phenomenon through both > observational studies of online communities and laboratory > experiments. The two papers below report on the phenomenon as it > presents itself in Digg, the news aggregation site. > > The first paper demonstrates that a tit-for-tat game of reciprocity > inflates the reputation of contributors and their contributions > without reflecting anything substantive about their contributions. > The second paper really brings out the negative consequences of this > phenomenon. The paper reports on an experiment where people judged > how interesting they found the contribution. The ranking values of > the articles were set by the investigator; sometime the rank of the > article was set high, at other times low. Experimental subjects > rated higher-ranked contributions as more interesting than lower > ranked contributions. The same article was rated as highly > interesting when its rank was set high and uninteresting when ranked > low. Duncan Watts (of small-world networks fame) observed the same > phenomenon with music rating. > > What this means in the present discussion is that people will likely > ignore low ranking contributions. Worse still, when they do actually > look at those contributions they are likely to see what the ranking > value led them to expect rather than the qualities of the > contribution itself. > > Unless we can find scientific research that demonstrates any > benefits to ranking, I think we should be wary of using such systems. > > All best, > > John > > Sadlon, E., Sakamoto, Y., Dever, H. J., & Nickerson, J. V. (2008). > The karma of Digg: Reciprocity in online social networks. In R. > Gopal and R. Ramesh (Eds.), Proceedings of the 18th Annual Workshop > on Information Technologies and Systems. http://cog.mgnt.stevens-tech.edu/~yasu/papers/reciprocity.pdf > > Sakamoto, Y., Ma, J., & Nickerson, J. V. (2009). 2377 people like > this article: The influence of others' decisions on yours. In N. > Taatgen, H. van Rijn, L. Schomaker, and J. Nerbonne (Eds.), > Proceedings of the 31st Annual Conference of the Cognitive Science > Society.http://cog.mgnt.stevens-tech.edu/~yasu/papers/cogscidigg1.pdf > > Salganik, M. J., Dodds, P. S., and Watts, D. J. (2006). Experimental > study of inequality and unpredictability in an artificial cultural > market. Science, 311(5762):854-856. http://dx.doi.org/10.1126/science.1121066 > [/url] > > > > > -------------------- m2f -------------------- > > (from forum) > http://squeakland.org/forums/viewtopic.php?p=11554#11554 > > -------------------- m2f -------------------- > _______________________________________________ > squeakland mailing list > [hidden email] > http://lists.squeakland.org/mailman/listinfo/squeakland Viewpoints Research is a 501(c)(3) nonprofit organization dedicated to improving "powerful ideas education" for the world's children and advancing the state of systems research and personal computing. Please visit us online at www.vpri.org _______________________________________________ Squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: Re: the reasons for ranking
|
|
In reply to this post by teefal
<base href="x-msg://415/">
On Sep 30, 2009, at 10:58 AM, voiklis wrote:
Thank you for your insightful response. A few things that may make the Squeakland ranking system different from those you cite . . 1) viewers don't rank, so it's not a popularity contest 2) only people with established credibility rank projects ... you've got to earn your say 3) ranking is limited to ten people per project 4) rankers are chosen at random, with bias to higher credibility ... so friends are less likely to rank each other 5) number or value of votes isn't shown anywhere, it's merely the position in the list 6) there are many ways to view projects (by group, by tag, by subject, by region) so you're much more likely to see lower ranked projects than in a straight sort Anyway, these are just off the top of my head. The benefits to the ranking system lie chiefly in: 1) letting people see which accounts have long-term credibility, which I've found useful on other sites 2) distributing the ranking effort while maintaining quality by favoring those with credibility 3) showing the best projects first, in a uniform & fair way, which is what it's all about It's really more of a trust system than a ranking system. It's just that ranking is the chief way to get trust. Tim _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: Re: the reasons for ranking
|
|
In reply to this post by Kim Rose-2
Am 30.09.2009 um 17:20 schrieb Kim Rose: > Hey, John, > > Nice to see you are still around and interested -- hope you are > doing well. Thanks for the comments. > > As for me and Viewpoints -- there is one reason only for ranking and > that is filtering in an attempt to bring the best educational > examples and those illustrative of etoys' strengths to the attention > of the community. That is exactly what the featured showcase is for. However, we also want to encourage everyone to share their work, even if it's not perfect. We need a place for that, too. > So, I, for one, am hoping that those "voting" are not voting for > any particular child, author or person, but for the example itself > -- I hope those voting/ranking are asking "is this a fine exemplar > to help teach a concept, principle, or powerful idea?". The more > the example answers back with a "yes" the higher the rank. Also, > the more complete an example the higher the rank. How disclosing of > what the Etoy is meant to do is in the example? Does it have an > explanation of what it is? If it is a game does it have instruction > on how to play and the goal of the game? > > I hope that teachers who encourage their students to upload projects > will *only* allow sharing if their student provides an "About" > flap, or some intro/explanation of the Etoy and some instruction on > where to start, the aim of the project, etc. The most beautiful > simulation of something will fall completely flat with someone if > they have no real idea of what they are looking at or what the > script they are playing with is meant to do. Etoys are intended to > teach and help us learn; learning cannot occur without context. No > author should assume their project will be self disclosing; it will > not. Our users have a variety of levels of expertise in both Etoys > and subject matter areas; providing more context can only help. All this is absolutely true. But we have to decide if we only want to have these special projects on our website or if we provide a place where children can share their work, even if the projects are not perfect. In my opinion we should have both, but not at the same page. > > If there is no "ranking" in place and everything is posted the > offerings soon get overwhelming and most difficult to navigate. We > found this years ago when we had an active "super-swiki". Projects > that were no more than a single sketch with no script at all were > posted and only "muddied the waters" for those wading in an attempt > to find something with some meaningful content. Thats true. To find meaningful content of good quality one should look at the featured showcase where you can only find the ranked projects. Rita > > I hope the community will bear this in mind when "ranking" what gets > uploaded and understand that not everyone can be represented in a > featured showcase. > -- Kim > > > On Sep 30, 2009, at 7:58 AM, voiklis wrote: > >> Hello all, >> >> I apologize for joining this conversation so late; I am a longtime, >> though recently quiet, member of the community. >> >> I understand that the ranking system is an attempt to enable people >> to judge the quality of a contribution (or contributor) based on >> some directly observable measure of reputation. On first sight, one >> could argue that such a system overcomes the problem that in the >> face-to-face world reputation is not directly observable. In order >> to get an honest assessment of a person's reputation, one has to >> invest a lot of time building trust with the people familiar with >> that person. Even then, a trustworthy assessment requires direct >> observation of the person's actions. >> >> A ranking system would appear to reduce that effort by half; >> knowing the person's reputation among his/her peers, one only needs >> to assess the work. >> >> The problem with that reasoning is that electronic ranking systems >> are highly susceptible to manipulation. Building reputation becomes >> the goal of the activity for many people and they use all sorts of >> seemingly harmless social and technological means to inflate their >> numbers. Our lab has been studying this phenomenon through both >> observational studies of online communities and laboratory >> experiments. The two papers below report on the phenomenon as it >> presents itself in Digg, the news aggregation site. >> >> The first paper demonstrates that a tit-for-tat game of reciprocity >> inflates the reputation of contributors and their contributions >> without reflecting anything substantive about their contributions. >> The second paper really brings out the negative consequences of >> this phenomenon. The paper reports on an experiment where people >> judged how interesting they found the contribution. The ranking >> values of the articles were set by the investigator; sometime the >> rank of the article was set high, at other times low. Experimental >> subjects rated higher-ranked contributions as more interesting than >> lower ranked contributions. The same article was rated as highly >> interesting when its rank was set high and uninteresting when >> ranked low. Duncan Watts (of small-world networks fame) observed >> the same phenomenon with music rating. >> >> What this means in the present discussion is that people will >> likely ignore low ranking contributions. Worse still, when they do >> actually look at those contributions they are likely to see what >> the ranking value led them to expect rather than the qualities of >> the contribution itself. >> >> Unless we can find scientific research that demonstrates any >> benefits to ranking, I think we should be wary of using such systems. >> >> All best, >> >> John >> >> Sadlon, E., Sakamoto, Y., Dever, H. J., & Nickerson, J. V. (2008). >> The karma of Digg: Reciprocity in online social networks. In R. >> Gopal and R. Ramesh (Eds.), Proceedings of the 18th Annual Workshop >> on Information Technologies and Systems. http://cog.mgnt.stevens-tech.edu/~yasu/papers/reciprocity.pdf >> >> Sakamoto, Y., Ma, J., & Nickerson, J. V. (2009). 2377 people like >> this article: The influence of others' decisions on yours. In N. >> Taatgen, H. van Rijn, L. Schomaker, and J. Nerbonne (Eds.), >> Proceedings of the 31st Annual Conference of the Cognitive Science >> Society.http://cog.mgnt.stevens-tech.edu/~yasu/papers/cogscidigg1.pdf >> >> Salganik, M. J., Dodds, P. S., and Watts, D. J. (2006). >> Experimental study of inequality and unpredictability in an >> artificial cultural market. Science, 311(5762):854-856. http://dx.doi.org/10.1126/science.1121066 >> [/url] >> >> >> >> >> -------------------- m2f -------------------- >> >> (from forum) >> http://squeakland.org/forums/viewtopic.php?p=11554#11554 >> >> -------------------- m2f -------------------- >> _______________________________________________ >> squeakland mailing list >> [hidden email] >> http://lists.squeakland.org/mailman/listinfo/squeakland > > > Viewpoints Research is a 501(c)(3) nonprofit organization dedicated > to improving "powerful ideas education" for the world's children and > advancing the state of systems research and personal computing. > Please visit us online at www.vpri.org > > > > > > > _______________________________________________ > Squeakland mailing list > [hidden email] > http://lists.squeakland.org/mailman/listinfo/squeakland Rita Freudenberg [hidden email] _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: the reasons for ranking
|
|
In reply to this post by Timothy Falconer-3
At Wed, 30 Sep 2009 08:44:47 -0400,
Timothy Falconer wrote: > > > On Sep 29, 2009, at 11:08 PM, Yoshiki Ohshima wrote: > > > At Tue, 29 Sep 2009 22:45:08 -0400, > > Timothy Falconer wrote: > >> > >> I think the piece that's missing here is that it will be pretty rare > >> to actually change from one level to another, especially for younger > >> children. > > > > Is this based on the logic described at > > http://tracker.squeakland.org/browse/SQ-361? It would be a bit > > surprizing to hear because "maxMojo" change alone affects everybody's > > rank. Since the owner of maxMojo probably will earn more points > > quicker than others, and the ranks are linear scale of the point, I > > would expect to see bigger disparity over time. > > Some of the details have changed since that list, but yes, the > mojoLevel formula is current. Ok, so the single person's behavior alone can change the whole system, yes? > Over time, some people will continue to contribute and others will > stop. The formula rewards sustained effort, not past performance. Hmm, are you saying that the top person will eventually stop so it is not going to be a problem? > Again, this is not a grade ... it's a thank you. You usually don't take "thank you" away after giving it, right? > >> Yes, we could take the colored circles off the website. I think > >> that > >> people are finding them fun, at least the people I'm talking to, > >> and I > >> think there's genuine value in knowing that a commenter is a regular > >> and not a puppet account. > > > > Well, people should actually look at the contents to judge the > > quality of comments. I've been writing fairly stupid comments. They > > should be treated so. > > Originally we were going to rank comments, but recently I decided that > would be just too much for people to have to do. > > Comment points are a recognition of effort ... not a grade, but a > thank you. > > When comment scribbling gets turned on (see the tracker), we'll have a > way to prevent bogus comments from gaming the system. ... I wasn't talking about comment points. I wasn't talking about gaming the system either... What you said above was that people should feel good when somebody with higher level comments on their projects. And I said that people should read the actual comment and feel good when it is good. -- Yoshiki _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
|
|
In reply to this post by Kim Rose-2
Hi Kim,
I _am_ well and still very much interested, even if my work has led in a very different direction. I fully understand the community's need for some kind of assessment system. I seem to remember that something of the sort was a topic of conversation at the very first Squeakfest. I am glad to see that Tim has modeled the ranking/trust system like some kind of peer review. Nevertheless, I am still wary of whether it will work or work well; peer review is problematic even in the sciences. Some kind of "expert" assessment or assessment tool (a rubric or checklist that asks does it work, does it provide instructions, etc?) might be better suited to your needs. Projects could then be listed in order of how many or which rubric items have been checked. I don't know who if anyone among the so-called experts has the time to devote to assessments, but I also don't know who if anyone among the student contributors will devote that time to peer review. While a chunk of my life has been devoted to solving problems with technologies, I have grown both wary and weary of technological solutions. I hope this solution will be thoroughly tested and not just to see if it functions but to see if it actually solves a human problem without introducing new problems. Best of luck, John On Wed, Sep 30, 2009 at 11:20 AM, Kim Rose <[hidden email]> wrote: > Hey, John, > > Nice to see you are still around and interested -- hope you are doing well. >  Thanks for the comments. > > As for me and Viewpoints -- there is one reason only for ranking and that is > filtering in an attempt to bring the best educational examples and those > illustrative of etoys' strengths to the attention of the community.  So, I, >  for one, am hoping that those "voting" are not voting for any particular > child, author or person, but for the example itself -- I hope those > voting/ranking are asking "is this a fine exemplar to help teach a concept, > principle, or powerful idea?".  The more the example answers back with a > "yes" the higher the rank.  Also, the more complete an example the higher > the rank.  How disclosing of what the Etoy is meant to do is in the example? >  Does it have an explanation of what it is?  If it is a game does it have > instruction on how to play and the goal of the game? > > I hope that teachers who encourage their students to upload projects will > *only* allow sharing if their student provides  an "About" flap, or some > intro/explanation of the Etoy and some instruction on where to start, the > aim of the project, etc.  The most beautiful simulation of something will > fall completely flat with someone if they have no real idea of what they are > looking at or what the script they are playing with is meant to do. Etoys > are intended to teach and help us learn; learning cannot occur without > context.  No author should assume their project will be self disclosing; it > will not.  Our users have a variety of levels of expertise in both Etoys and > subject matter areas; providing more context can only help. > > If there is no "ranking" in place and everything is posted the offerings > soon get overwhelming and most difficult to navigate.  We found this years > ago when we had an active "super-swiki".  Projects that were no more than a > single sketch with no script at all were posted and only "muddied the > waters" for those wading in an attempt to find something with some > meaningful content. > > I hope the community will bear this in mind when "ranking" what gets > uploaded and understand that not everyone can be represented in a featured > showcase. >  -- Kim > > > On Sep 30, 2009, at 7:58 AM, voiklis wrote: > >> Hello all, >> >> I apologize for joining this conversation so late; I am a longtime, though >> recently quiet, member of the community. >> >> I understand that the ranking system is an attempt to enable people to >> judge the quality of a contribution (or contributor) based on some directly >> observable measure of reputation. On first sight, one could argue that such >> a system overcomes the problem that in the face-to-face world reputation is >> not directly observable. In order to get an honest assessment of a person's >> reputation, one has to invest a lot of time building trust with the people >> familiar with that person. Even then, a trustworthy assessment requires >> direct observation of the person's actions. >> >> A ranking system would appear to reduce that effort by half; knowing the >> person's reputation among his/her peers, one only needs to assess the work. >> >> The problem with that reasoning is that electronic ranking systems are >> highly susceptible to manipulation. Building reputation becomes the goal of >> the activity for many people and they use all sorts of seemingly harmless >> social and technological means to inflate their numbers. Our lab has been >> studying this phenomenon through both observational studies of online >> communities and laboratory experiments. The two papers below report on the >> phenomenon as it presents itself in Digg, the news aggregation site. >> >> The first paper demonstrates that a tit-for-tat game of reciprocity >> inflates the reputation of contributors and their contributions without >> reflecting anything substantive about their contributions. The second paper >> really brings out the negative consequences of this phenomenon. The paper >> reports on an experiment where people judged how interesting they found the >> contribution. The ranking values of the articles were set by the >> investigator; sometime the rank of the article was set high, at other times >> low. Experimental subjects rated higher-ranked contributions as more >> interesting than lower ranked contributions. The same article was rated as >> highly interesting when its rank was set high and uninteresting when ranked >> low. Duncan Watts (of small-world networks fame) observed the same >> phenomenon with music rating. >> >> What this means in the present discussion is that people will likely >> ignore low ranking contributions. Worse still, when they do actually look at >> those contributions they are likely to see what the ranking value led them >> to expect rather than the qualities of the contribution itself. >> >> Unless we can find scientific research that demonstrates any benefits to >> ranking, I think we should be wary of using such systems. >> >> All best, >> >> John >> >> Sadlon, E., Sakamoto, Y., Dever, H. J., & Nickerson, J. V. (2008). The >> karma of Digg: Reciprocity in online social networks. In R. Gopal and R. >> Ramesh (Eds.), Proceedings of the 18th Annual Workshop on Information >> Technologies and Systems. >> http://cog.mgnt.stevens-tech.edu/~yasu/papers/reciprocity.pdf >> >> Sakamoto, Y., Ma, J., & Nickerson, J. V. (2009). 2377 people like this >> article: The influence of others' decisions on yours. In N. Taatgen, H. van >> Rijn, L. Schomaker, and J. Nerbonne (Eds.), Proceedings of the 31st Annual >> Conference of the Cognitive Science >> Society.http://cog.mgnt.stevens-tech.edu/~yasu/papers/cogscidigg1.pdf >> >> Salganik, M. J., Dodds, P. S., and Watts, D. J. (2006). Experimental study >> of inequality and unpredictability in an artificial cultural market. >> Science, 311(5762):854-856. http://dx.doi.org/10.1126/science.1121066[/url] >> >> >> >> >> -------------------- m2f -------------------- >> >> (from forum) >> http://squeakland.org/forums/viewtopic.php?p=11554#11554 >> >> -------------------- m2f -------------------- >> _______________________________________________ >> squeakland mailing list >> [hidden email] >> http://lists.squeakland.org/mailman/listinfo/squeakland > > > Viewpoints Research is a 501(c)(3) nonprofit organization dedicated to > improving "powerful ideas education" for the world's children and advancing > the state of systems research and personal computing. Please visit us online > at www.vpri.org > > > > > > > _______________________________________________ > Squeakland mailing list > [hidden email] > http://lists.squeakland.org/mailman/listinfo/squeakland > squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
|
|
In reply to this post by Timothy Falconer-3
To paraphrase a fellow named Alan I heard give a talk a long time ago:
"If I can do it in one click, I'll do it If it takes me two click's I might do it If it takes me three or more click's forget about it" Perhaps radio buttons instead of arrows would be a better interface design. In regards to the method of ranking/tags et al, not sure what works best, but the goals should be as stated before: 1) Encouraging kids to post their projects 2) Encouraging participation of the community in providing constructive criticism and encouragement 3)
One additional idea would be to allow people to have their own "page" like it is done at Scratch (ex: http://scratch.mit.edu/users/sanda427 ) where in one spot you can see all of a person's projects (preferably ordered by date, so one could look and see how they have improved and perhaps what they learned) along with a list of friends (to encourage a sense of community) and Galleries. Galleries could be useful for example for teachers who post projects or better yet lesson plans they found useful. Stephen Thomas -------------------- m2f -------------------- (from forum) http://squeakland.org/forums/viewtopic.php?p=11736#11736 -------------------- m2f -------------------- _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: Re: the reasons for ranking
|
|
<base href="x-msg://14/">Thanks for your comments and your use of the showcase. Be sure to have a look at the pending showcase tasks:
I'll reference existing tickets below. On Oct 5, 2009, at 11:57 PM, mrsteve wrote:
These are great goals. We're also hoping to encourage many adults (teachers, etc) to post projects, which speaks to #3.
http://tracker.squeakland.org/browse/SQ-366 (groups, aka galleries)
We appreciate your thoughts, and ask for a little more patience as we add these features. Feel free to add any new requests to the tracker directly. Take care, Tim _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
|
|
In reply to this post by Timothy Falconer-3
Tim,
Thanks for the rapid response.
Please don't take my comments the wrong way, I am greatly impressed with the project and all that has been accomplished so far and I understand the need to prioritize and take the 1000 mile journey one step at a time. One thing it took me a long time to learn is patience (I still have some more work to do in that area). I am working on helping with educational examples and ask your patience as I finalize them and flesh out the proper framework I want to use before posting. Thanks, Stephen Thomas -------------------- m2f -------------------- (from forum) http://squeakland.org/forums/viewtopic.php?p=11742#11742 -------------------- m2f -------------------- _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
Re: Re: the reasons for ranking
|
|
<base href="x-msg://19/">Stephen,
I particularly appreciate your spanish comments for the Nicaragua and Mi Istoria Waveplace stories. I haven't yet announce that the storybooks are on the showcase, and it's very nice that the children will have their native language to read. Also, Randy, if you're reading, thank you for your comments as well. This will mean a lot to these children. Tim On Oct 6, 2009, at 3:55 PM, mrsteve wrote:
-- Timothy Falconer Squeakland Foundation 610-797-3100 -- "Intelligence is what you use when you don't know what to do." ... piaget _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
|
|
In reply to this post by Timothy Falconer-3
i still find the ranking accurate..
got no objection about it... promise! -------------------- m2f -------------------- (from forum) http://squeakland.org/forums/viewtopic.php?p=11792#11792 -------------------- m2f -------------------- _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
|
|
I agree, my relatives can't wait to have a copy of the stories that they can read in their own language. -------------------- m2f -------------------- (from forum) http://squeakland.org/forums/viewtopic.php?p=11809#11809 -------------------- m2f -------------------- _______________________________________________ squeakland mailing list [hidden email] http://lists.squeakland.org/mailman/listinfo/squeakland |
«
Return to SqueakLand
|
1 view|%1 views
| Free forum by Nabble | Edit this page |