[vwnc] Fwd: Regex11

> Today I've written Vassili Bykov because of a litte bug in Regex11 which  
> could be intersting for you too. Any opinions?
>
>
> ------- Weitergeleitete Nachricht -------
> An: "Vassili Bykov" <[hidden email]>
> Kopie:
> Betreff: Regex11
> Datum: Mon, 15 Sep 2008 09:33:13 +0200
>
> Dear Vassili Bykov,
>
> as far as I can see you've written the Regex11 package and I think I've
> found a bug. The documentation states:
>
> \w any word constituent character (same as [a-zA-Z0-9_])
>
> but the actual implementation fails with underscores:
>
> '_' matchesRegex: '\w' "evaluates to false"
> '_' matchesRegex: '[a-zA-Z0-9_]' "evaluates to true"
>
> Should the bug be fixed because we rely on the definition of \w or is it
> the better way to adjust the documentation in order not to break existing
> applications? How can I help?
>
> Greetings,

> Different regex libraries differ in whether \w includes underscore. The  
> current implementation of Regex11 does not have a bug per se: the code  
> clearly says "char isAlphaNumeric"; rather, the documentation is  
> incorrect.
>
>
> However, the move in regex libraries in general seems to be towards  
> including underscore:
>
>
> http://jakarta.apache.org/regexp/changes.html (added in 1.3, 2003)
>
> http://nlp.stanford.edu/nlp/javadoc/gnu-regexp-docs/changes.html (added  
> in 1.1, 2001)
>
> http://savannah.gnu.org/bugs/?19637#discussion (Gnu documentation "\w =  
> [[:alnum]]" corrected 2007)
>
>
> How much do people feel adding underscore to \w in Regex11 would be a  
> breaking change, and how much a bug fix?
>
>
> Steve
>
>
>
> From: [hidden email] [mailto:[hidden email]] On  
> Behalf Of Sudhakar Krishnamachari
> Sent: 16 September 2008 13:48
> To: Steffen Märcker
> Cc: vwnc mailing list; [hidden email]
> Subject: Re: [vwnc] Fwd: Regex11
>
>
> What Steffen states is obviously right..
>
>
>> From Smacc Scanner.html:
>
>
> \w
>
> Matches any letter, number or underscore, [A-Za-z0-9_].
>
>
> '_'  or the underscore should be matched...
>
>
> regards
>
> skrish
>
>
> On Tue, Sep 16, 2008 at 12:22 PM, Steffen Märcker <[hidden email]> wrote:
>
> Hi,
>
>
>> Isn't the documentations stating what the code does?
>
> exactly, the documentation states '_' should match '\w', but the code  
> does not. Just evaluate the given code examples:
>
>
>>>      '_' matchesRegex: '\w'                  "evaluates to false"
>>>      '_' matchesRegex: '[a-zA-Z0-9_]'        "evaluates to true"
>
> As far as I understand both have to give true... Or have I missed  
> something?
>
> Regards,
> Steffen
> _______________________________________________________________________
> Jetzt neu! Schützen Sie Ihren PC mit McAfee und WEB.DE. 30 Tage
> kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220
>
>
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
>

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Vassili Bykov
> Sent: 17 September 2008 00:16
> To: Steffen Märcker
> Cc: vwnc mailing list
> Subject: Re: [vwnc] Fwd: Regex11
>
> I think \w should include underscore to match what seems to be the
> expected behavior out there. Going a step further, perhaps a Character
> predicate named something like #isIdentifierConstituent to include $_
> in addition to pure alphanumerics would be useful.
>
> Cheers,
>
> --Vassili
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

> Cesar Rabak wrote:
>> Well, thinking about it I _believe_ #isWordConstituent is as ambiguous
>
>> as it gets since we do not use numbers for plain (English or natural
>> language) words as well, right?
>
> Good point about _ and [0-9] not being in English words! I blame the
> regex designers who named or defined \w poorly :-)
>
> Vassili Bykov wrote:
>> #isWordConstituent was my first thought too, but "word" may suggest
>> other interpretations. For example, a locale-specific test for
>> characters acceptable in words of the current locale's language.
>
> And #isIdentifierConstituent still tends to imply ASCII. Fair enough,
> although it's not true in VW, which accepts non-ASCII characters in
> identifiers. #isAlphaNumeric and #isAlphabetic only accept ASCII
> characters. Oddly, #isVowel and #isUppercase allow non-ASCII characters.
> Maybe we should have #isAlphaNumeric_ :-)
>
> Steve
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

> Cesar Rabak [mailto:[hidden email]] wrote:
>> [:alnum:] any alphanumeric, i.e., a word constituent, character
>
> Thanks, I missed that one since I was only checking the docs for \w, _,  
> and underscore. I've corrected it and a couple of other mistakes in that  
> list, and published as 1.3.1, also making the issues of locale explicit:
>
> \w any word constituent character (same as [a-zA-Z0-9_])
> \W any character but a word constituent
> \d a digit (same as [0-9])
> \D anything but a digit
> \s a whitespace character (same as [:space:] below)
> \S anything but a whitespace character
>
> [:alnum:] any alphanumeric character (same as [a-zA-Z0-9])
> [:alpha:] any alphabetic character (same as [a-zA-Z])
> [:cntrl:] any control character. (any character with code < 32)
> [:digit:] any decimal digit (same as [0-9])
> [:graph:] any graphical character. (any character with code >= 32).
> [:lower:] any lowercase character (including non-ASCII lowercase  
> characters)
> [:print:] any printable character. In this version, this is the same as  
> [:graph:]
> [:punct:] any punctuation character:  . , ! ? ; : ' - ( ) ` and double  
> quotes
> [:space:] any whitespace character (space, tab, CR, LF, null, form feed,  
> Ctrl-Z, 16r2000-16r200B, 16r3000)
> [:upper:] any uppercase character (including non-ASCII uppercase  
> characters)
> [:xdigit:] any hexadecimal character (same as [a-fA-F0-9]).
>
> Note that many of these are only as consistent or inconsistent on issues
> of locale as the underlying Smalltalk implementation. Values shown here
> are for VisualWorks 7.6.
>
> Steve