making an avatar track a Web-cam

> The youtube video below is stunning. Be sure to go through at least 1:30.
>
> How can we either get or reproduce the tracking data from the video feed
> to animate avatars in Croquet?
>
> (I mean, like, drop everything, quit your job, and make this work in
> Croquet by like next week.)
>
> -H
>
> Florent THIERY wrote on [hidden email]:
>> ...
>> We can do the same analogy with 3D avatar animation VS video stream.
>> If we managed to track a user's body movement in realtime using a
>> camera/webcam (as many as the avatar's degrees of liberty allow it),
>> we could animate avatars without transferring the video itself,
>> resulting in much lighter data flow.... That's why i'm pretty sure
>> that Video_To_3D automatic reconstruction/modelization will give
>> online 3D a huge boost (apart from games). Check out this awesome
>> logitech video:
>
> http://www.youtube.com/watch?v=r7Gn2TyEyHw
>
> [Florent also has pointed out http://mxrtoolkit.sourceforge.net/ and
> http://www-evasion.imag.fr/Membres/Lionel.Reveret/animated-speakers/index.html]
>

> Nice video,
>
> Not as complete nor the same intended, but perhaps a start:  
> http://www.cosmocows.com/fun/ paragraph: 'webcam fun with squeak'
>
> It doesn't interpretate/construct face-gestures. But with a webcam and
> your real hands you can handle morph-objects on your screen. Esspecially
> bouncing them gives, after a while, an odd feeling; as if you feel real
> impact from a virtual morph-object on your hands (see WebCamBounceMorph
> ). We also made them react to real collored plastic balls we bought in a
> toyshop around the corner. Morph-objects (re)acted upon color, movements
> and shape of these balls. The shape of these balls was changed by
> squeezing them. Three shapes are distinguished: not squeezed, squeezed
> along x axis, squeezed along y axis. These balls could also act as a
> mouse (grabbing, clicking, moving mouse pointer). You could also
> 'strike' morph-objects to other desktops (this particular networking
> software piece is not included in download, don't know where it is). We
> were thinking about an (anti-rsi) game, but didn't fulfill it. This
> webcam stuff was mostly (if not all) built by Wiebe Baron in a few days.
>
> Greetings,
> Mathieu van Echtelt
>
>
> On 4/3/07, *Howard Stearns* <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     The youtube video below is stunning. Be sure to go through at least
>     1:30.
>
>     How can we either get or reproduce the tracking data from the video
>     feed to
>     animate avatars in Croquet?
>
>     (I mean, like, drop everything, quit your job, and make this work in
>     Croquet by
>     like next week.)
>
>     -H
>
>     Florent THIERY wrote on [hidden email]
>     <mailto:[hidden email]>:
>      > ...
>      > We can do the same analogy with 3D avatar animation VS video stream.
>      > If we managed to track a user's body movement in realtime using a
>      > camera/webcam (as many as the avatar's degrees of liberty allow it),
>      > we could animate avatars without transferring the video itself,
>      > resulting in much lighter data flow.... That's why i'm pretty sure
>      > that Video_To_3D automatic reconstruction/modelization will give
>      > online 3D a huge boost (apart from games). Check out this awesome
>      > logitech video:
>
>     http://www.youtube.com/watch?v=r7Gn2TyEyHw
>
>     [Florent also has pointed out http://mxrtoolkit.sourceforge.net/ and
>     http://www-evasion.imag.fr/Membres/Lionel.Reveret/animated-speakers/index.html]
>
>
>
>
> --
> AG5 Product manager

>
> Check out the "Silhouette" technology by Integrated Virtual networks
> (IVN) -- http://www.ivn3.com/technology0.php
>
> As I have been the Chief Operations Officer of IVN for some time, I am
> very familiar with the Silhouette technology, and even contributed to
> it to some extent. Silhouette, a patented invention by L.A. patent
> lawyer and IVN-founder William Benman,... is still rough around the
> edges, including literally. Speaking from personal experience, I can
> testify that it does work though.
>
> Simply put, you point a generic webcam at the place where you normally
> sit. Background can be totally arbitrary, as long as there is no or
> very little movement in it. No blue or green background needed as with
> classical chromakeying. After hitting a key to set this background,
> you come into view and the software extracts -makes a 'silhouette' of-
> your (moving) video image in real time from each frame. Your moving
> Silhouetted video image is then transposed unto a (otherwise
> transparent) polygon inside a (multi-user) 3D virtual world (such as
> CyberTown, which William Benman bought from Blaxxun USA quite a few
> years ago) and your voice's audio VOIP-stream is 3D spatialized with
> its source roughly at the location of your head. Instead of a user's
> regular avatar there now is now a real-time moving video avatar. For
> the rest, one uses such a Silhouette video-av in just the same way as
> one uses regular avs.
>
> Some notes:
>
> - Heavily protected by patents made by the inventor who is himself a
> very astute patent lawyer. So, don't even think about it. Some of the
> relevant patents are referenced online -- see USPTO Patent # 5784546
> and USPTO Patent # 5966130. Google on
> http://www.google.com/search?hl=en&q=Integrated+Virtual+Networks+patent&btnG=Search 
>
>
> - So far, upper body only. So it -naturally- only displays (extracted
> and transposed) video of the parts of your body which your webcam
> sees. (Part of my involvement was providing a seamlessly aligning
> "rest-of-the-body" solution... which was not quite implemented yet the
> last time I used Silhouette myself.)
>
> - The Silhouetted real-time video sprite itself is 2D -- flat video.
> When, inside of a 3D virtual world, you go and look around the back of
> a Silhouette-based avatar, you see the same video as on the front, but
> mirrored. (Here too, one of the original goals of my involvement was
> to look into and possibly develop ways of not only providing the rest
> of the body below the shoulders, but also in full 3D. One obvious way
> which was explored was mapping the real-time Silhouetted video stream
> unto a one-mesh-fits-all polygon mesh of a generic head & shoulders.
> That works relatively well for men, as long as they don't have long or
> big hair. For women with long or big hair, it does not work, unless
> you provide them with a set of template 3D meshes to drape their
> Silhouetted video on which roughly suit the 3D space occupied by their
> heirdo of that particular moment. That still results -for those cases-
> in an often unsatisfactory hit-or-miss solution. The variety of 3D
> volumes occupied by female heirdos out there is just too big a space
> to do justice with a small set of 3D mesh templates.
>
> If there is interest in this from the Croquet community, I can offer
> to be the interface to IVN and William Benman to negoatiate a friendly
> licensing agreement.)
>
> Philippe Van Nedervelde
> E-SPACES
> SkypeID:  philippevannedervelde
> http://www.e-spaces.com/lbw/portfolio/portfolio_time.htm
>
>
>

> On Tue, 2007-04-03 at 18:58 -0400, David P. Reed wrote:
>  
>> Well, to each his own.   If you got a valid patent, good luck on
>> figuring out a business model beyond trolling.
>>
>> The world can probably find an alternative for 17 years.
>>    
> That's the RedHat/Linux approach. Stay cleanly away from patents that
> aren't open-sourced. My openGL problems would go away over night if they
> would release their driver code. We might consider using ogg Vorbis
> instead of mp3 as there are plenty of patent disputes over this file
> format. Suppose Croquet becomes wildly popular, and the mp3 guy's lawyer
> comes around for the rent overdue since the beginning. It happens.
> Ogg-Vorbis actually seems to be a better music format anyway. Then there
> is ogg-theora for video. All free and unencumbered. Ric
>  

> Ric Moore wrote:
>> On Tue, 2007-04-03 at 18:58 -0400, David P. Reed wrote:
>>
>>> Well, to each his own.   If you got a valid patent, good luck on  
>>> figuring out a business model beyond trolling.
>>>
>>> The world can probably find an alternative for 17 years.
>>>
>> That's the RedHat/Linux approach. Stay cleanly away from patents that
>> aren't open-sourced. My openGL problems would go away over night  
>> if they
>> would release their driver code. We might consider using ogg Vorbis
>> instead of mp3 as there are plenty of patent disputes over this file
>> format. Suppose Croquet becomes wildly popular, and the mp3 guy's  
>> lawyer
>> comes around for the rent overdue since the beginning. It happens.
>> Ogg-Vorbis actually seems to be a better music format anyway. Then  
>> there
>> is ogg-theora for video. All free and unencumbered. Ric
>>
> +1 on ogg for audio.

> David & Les,
>
> I'm not a patent lawyer either.  (I also am not the inventor of
> Silhouette, nor do I own shares in IVN or presently have a business
> interest in Silhouette).
>
> Imnasho, Silhouette definitely is an interesting approach to doing avs
> which are much more expressive than the relatively static usual avs.
> It works too. Kinda.
>
> I do not know whether or not Benman's patents re his Silhouette video
> avatar technology hold water and would prevail in court. Given that
> Benman is a patent lawyer by trade, the odds are good that he did his
> homework properly though. I for one would not like to go to court over
> something invented as well as patented by a patent lawyer.
>
> Although... I occasionally pointed out products to Benman which I
> thought came close to Silhouette and asked him if they infringed his
> patent or not. In the case of the EyeToy, for instance, he felt it did
> not. So far, I cannot recall a single product I pointed out to him
> which he felt infringed on his patent.
>
> As to finding alternatives, that may not be necessary. I believe
> Benman may be persuaded to very friendly licensing terms. IVN indeed
> has found it relatively challenging to date to market the Silhouette
> license and to get it being used by any significant number of users.
> So a distribution platform like OpenCroquet with its ambitions of
> -given half a chance- even going so far as replacing the 2D web as
> humanity's default interface to online information and services... is
> definitely attractive.
>
> Should that approach not yield satisfying results, I believe
> -preferably subject to verification (of the full Benman patents -- I
> am not sure the full text of all of them are online) by a patent
> lawyer- Benman's patents may not cover the case where one drapes
> otherwise unprocessed video of a person over a 3D mesh. I mean, where
> that video is the full unprocessed frame. In other words, if you use
> straight video without extracting the video of just the person from
> the background. I can think of a few ways to use that approach and
> essentially end up with a similar result... and perhaps yielding a
> result that is even better than Silhouette.
>
> By the way -full disclosure- I presently do not work for IVN (only
> worked for IVN on a consultancy basis too at that) plus I own no
> shares in IVN or any business associated with it or William Benman. I
> do maintain a friendly relationship with him.
>
> Philippe Van Nedervelde
> E-SPACES
> SkypeID:  philippevannedervelde
> http://www.e-spaces.com/lbw/portfolio/portfolio_time.htm
>
>
> David P. Reed wrote:
>> Well, to each his own.   If you got a valid patent, good luck on
>> figuring out a business model beyond trolling.
>>
>> The world can probably find an alternative for 17 years.
>>
> Les wrote:
> Hi, Phillippe,
> I read the two patent listings that came up on Google, and as that runs
> it is a trivial extension of prior art.  The means described have been
> available from a number of image manipulation packages for many years.
>
> I cannot argue patent applicability, as I am not an attorney, but
> simply a programmer with over 3 decades of experience.
>
> However what I have proposed is much more complex, and may or may not
> be easily produced.  It is much closer to puppet animation technology,
> and matches with work done in Blender and several other 3D packages only
> adding the facial tracking capability.  
>
> Regards,
> Les H
>  

> I am not totally opposed to patents, and have several, mostly assigned
> to prior employers, as well as having some patent applications in
> process.   Patents obtained by patent lawyers, however, are especially
> worth being wary of, because it is unfortunately possible to construct
> a patent that is less than an invention and more a gaming of the
> patent system.   Lawyers can certainly invent, but if the invention is
> not "useful" except as a way of collecting fees from more complete
> inventions that solve the problems of making something truly useful
> like a productized form, it is a bit odorous, at least to me.  
> Reduction to practice has become somewhat vestigial in the patent
> system, whereas once upon a time one had to show a working model and
> discuss how it would really be useful.
>
> There is standard advice to engineers given by many patent lawyers -
> never do a patent search.  The reason appears to be because if it can
> be shown that you had ever seen a patent that is litigated against
> you, you will be liable for additional (treble?) damages - knowing
> infringement is worse than accidental infringement, knowing omission
> of prior art in a filing is worse than accidental omission.
>
> Since open source code is typically licensed for free (unless one uses
> a multiple license business model), and in the case of GPL has been
> held to be an implicit royalty-free license for all patents embodied
> in the GPL code to be used in any legitimate use of the code, the
> engineer should probably take even greater pains to avoid *knowing*
> infringement.   Thus, I try to avoid reading about patents, and
> suggest that others try to avoid that as well.   There is no duty to
> read the patent literature, and lots of risk, apparently.
>
> Philippe Van Nedervelde wrote:
>> David & Les,
>>
>> I'm not a patent lawyer either.  (I also am not the inventor of
>> Silhouette, nor do I own shares in IVN or presently have a business
>> interest in Silhouette).
>>
>> Imnasho, Silhouette definitely is an interesting approach to doing
>> avs which are much more expressive than the relatively static usual
>> avs. It works too. Kinda.
>>
>> I do not know whether or not Benman's patents re his Silhouette video
>> avatar technology hold water and would prevail in court. Given that
>> Benman is a patent lawyer by trade, the odds are good that he did his
>> homework properly though. I for one would not like to go to court
>> over something invented as well as patented by a patent lawyer.
>>
>> Although... I occasionally pointed out products to Benman which I
>> thought came close to Silhouette and asked him if they infringed his
>> patent or not. In the case of the EyeToy, for instance, he felt it
>> did not. So far, I cannot recall a single product I pointed out to
>> him which he felt infringed on his patent.
>>
>> As to finding alternatives, that may not be necessary. I believe
>> Benman may be persuaded to very friendly licensing terms. IVN indeed
>> has found it relatively challenging to date to market the Silhouette
>> license and to get it being used by any significant number of users.
>> So a distribution platform like OpenCroquet with its ambitions of
>> -given half a chance- even going so far as replacing the 2D web as
>> humanity's default interface to online information and services... is
>> definitely attractive.
>>
>> Should that approach not yield satisfying results, I believe
>> -preferably subject to verification (of the full Benman patents -- I
>> am not sure the full text of all of them are online) by a patent
>> lawyer- Benman's patents may not cover the case where one drapes
>> otherwise unprocessed video of a person over a 3D mesh. I mean, where
>> that video is the full unprocessed frame. In other words, if you use
>> straight video without extracting the video of just the person from
>> the background. I can think of a few ways to use that approach and
>> essentially end up with a similar result... and perhaps yielding a
>> result that is even better than Silhouette.
>>
>> By the way -full disclosure- I presently do not work for IVN (only
>> worked for IVN on a consultancy basis too at that) plus I own no
>> shares in IVN or any business associated with it or William Benman. I
>> do maintain a friendly relationship with him.
>>
>> Philippe Van Nedervelde
>> E-SPACES
>> SkypeID:  philippevannedervelde
>> http://www.e-spaces.com/lbw/portfolio/portfolio_time.htm
>>
>>
>> David P. Reed wrote:
>>> Well, to each his own.   If you got a valid patent, good luck on
>>> figuring out a business model beyond trolling.
>>>
>>> The world can probably find an alternative for 17 years.
>>>
>> Les wrote:
>> Hi, Phillippe,
>>     I read the two patent listings that came up on Google, and as
>> that runs
>> it is a trivial extension of prior art.  The means described have been
>> available from a number of image manipulation packages for many years.
>>
>>     I cannot argue patent applicability, as I am not an attorney, but
>> simply a programmer with over 3 decades of experience.
>>     However what I have proposed is much more complex, and may or may
>> not
>> be easily produced.  It is much closer to puppet animation technology,
>> and matches with work done in Blender and several other 3D packages only
>> adding the facial tracking capability.
>> Regards,
>> Les H
>>  

> I have been following that rule as well. It is a real pain. What do
> you think about the PTO posting pending patents for open comment? In
> theory, this would give us experts a chance to weigh in on a patent
> before it is actually granted. However, if you follow the corporate
> "don't look" model, this puts you in just as much jeapardy. Does HP or
> other big hi-tech companies have a policy about this?
>
> You never write anymore...
>
> David
>
>
> David P. Reed wrote:
>> I am not totally opposed to patents, and have several, mostly
>> assigned to prior employers, as well as having some patent
>> applications in process.   Patents obtained by patent lawyers,
>> however, are especially worth being wary of, because it is
>> unfortunately possible to construct a patent that is less than an
>> invention and more a gaming of the patent system.   Lawyers can
>> certainly invent, but if the invention is not "useful" except as a
>> way of collecting fees from more complete inventions that solve the
>> problems of making something truly useful like a productized form, it
>> is a bit odorous, at least to me.   Reduction to practice has become
>> somewhat vestigial in the patent system, whereas once upon a time one
>> had to show a working model and discuss how it would really be useful.
>>
>> There is standard advice to engineers given by many patent lawyers -
>> never do a patent search.  The reason appears to be because if it can
>> be shown that you had ever seen a patent that is litigated against
>> you, you will be liable for additional (treble?) damages - knowing
>> infringement is worse than accidental infringement, knowing omission
>> of prior art in a filing is worse than accidental omission.
>>
>> Since open source code is typically licensed for free (unless one
>> uses a multiple license business model), and in the case of GPL has
>> been held to be an implicit royalty-free license for all patents
>> embodied in the GPL code to be used in any legitimate use of the
>> code, the engineer should probably take even greater pains to avoid
>> *knowing* infringement.   Thus, I try to avoid reading about patents,
>> and suggest that others try to avoid that as well.   There is no duty
>> to read the patent literature, and lots of risk, apparently.
>>
>> Philippe Van Nedervelde wrote:
>>> David & Les,
>>>
>>> I'm not a patent lawyer either.  (I also am not the inventor of
>>> Silhouette, nor do I own shares in IVN or presently have a business
>>> interest in Silhouette).
>>>
>>> Imnasho, Silhouette definitely is an interesting approach to doing
>>> avs which are much more expressive than the relatively static usual
>>> avs. It works too. Kinda.
>>>
>>> I do not know whether or not Benman's patents re his Silhouette
>>> video avatar technology hold water and would prevail in court. Given
>>> that Benman is a patent lawyer by trade, the odds are good that he
>>> did his homework properly though. I for one would not like to go to
>>> court over something invented as well as patented by a patent lawyer.
>>>
>>> Although... I occasionally pointed out products to Benman which I
>>> thought came close to Silhouette and asked him if they infringed his
>>> patent or not. In the case of the EyeToy, for instance, he felt it
>>> did not. So far, I cannot recall a single product I pointed out to
>>> him which he felt infringed on his patent.
>>>
>>> As to finding alternatives, that may not be necessary. I believe
>>> Benman may be persuaded to very friendly licensing terms. IVN indeed
>>> has found it relatively challenging to date to market the Silhouette
>>> license and to get it being used by any significant number of users.
>>> So a distribution platform like OpenCroquet with its ambitions of
>>> -given half a chance- even going so far as replacing the 2D web as
>>> humanity's default interface to online information and services...
>>> is definitely attractive.
>>>
>>> Should that approach not yield satisfying results, I believe
>>> -preferably subject to verification (of the full Benman patents -- I
>>> am not sure the full text of all of them are online) by a patent
>>> lawyer- Benman's patents may not cover the case where one drapes
>>> otherwise unprocessed video of a person over a 3D mesh. I mean,
>>> where that video is the full unprocessed frame. In other words, if
>>> you use straight video without extracting the video of just the
>>> person from the background. I can think of a few ways to use that
>>> approach and essentially end up with a similar result... and perhaps
>>> yielding a result that is even better than Silhouette.
>>>
>>> By the way -full disclosure- I presently do not work for IVN (only
>>> worked for IVN on a consultancy basis too at that) plus I own no
>>> shares in IVN or any business associated with it or William Benman.
>>> I do maintain a friendly relationship with him.
>>>
>>> Philippe Van Nedervelde
>>> E-SPACES
>>> SkypeID:  philippevannedervelde
>>> http://www.e-spaces.com/lbw/portfolio/portfolio_time.htm
>>>
>>>
>>> David P. Reed wrote:
>>>> Well, to each his own.   If you got a valid patent, good luck on
>>>> figuring out a business model beyond trolling.
>>>>
>>>> The world can probably find an alternative for 17 years.
>>>>
>>> Les wrote:
>>> Hi, Phillippe,
>>>     I read the two patent listings that came up on Google, and as
>>> that runs
>>> it is a trivial extension of prior art.  The means described have been
>>> available from a number of image manipulation packages for many years.
>>>
>>>     I cannot argue patent applicability, as I am not an attorney, but
>>> simply a programmer with over 3 decades of experience.
>>>     However what I have proposed is much more complex, and may or
>>> may not
>>> be easily produced.  It is much closer to puppet animation technology,
>>> and matches with work done in Blender and several other 3D packages
>>> only
>>> adding the facial tracking capability. Regards,
>>> Les H
>>>  
>
>