Thursday, December 2, 2010

Unity 3D FaceAPI Integration Tutorials

First of all Im really sorry for taking a long time for releasing this awaited tutorials with all working codes.

Since I dont have web space of my own, I rely on free file hostings, I have uploaded the required files with the mirrors::
Demo:
http://www.youtube.com/watch?v=yN_YEnDs2uk


Tutorials PDF file: (Only the below two files/links are enough, other links are mirror links)
http://www.4shared.com/office/u4ZV8-ME/Unity_3D_FaceAPI_Integration.html
OR
http://www.megaupload.com/?d=G88L0PY1
OR
http://www.esnips.com/doc/2e10244f-ee05-473b-8c82-dbf1fc1aa244/Unity_3D_FaceAPI_Integration

Full Project (Unity Package File):
http://www.4shared.com/zip/f0-PHD1h/faceAPI.html
OR
http://www.megaupload.com/?d=YKX2B382
OR
http://www.esnips.com/doc/11acf06f-232a-4941-abe5-12a80aa713b0/faceAPI

Download the PDF file, that contains links of files to be downloaded...



IMPORTANT UPDATE:
In new Unity versions, there is an error "Null reference", this bug is fixed. Download this file
http://www.4shared.com/file/9E6oGLWg/script.html
OR
 http://www.mediafire.com/?ovz29bodd013gbk
OR
and replace the original script.cs file from /scripts in the unity package...

Thank god, found this script from my video(10:37), dont forget to check the video. Check the link down.

Dont forget to view this post:
http://mypersonalsoft.blogspot.com/2011/05/unity3d-faceapi-integration-to-achieve.html


Enjoy....

Please don't forget to post your comments here. This is my first tutorial, and your comment will really help me in improving myself...


And Unity 3D handVU Integration Tutorials will take some more time...sorry again friends!!

41 comments:

  1. Great work and nice to share! Thank you very much. Now, let's study it :D

    ReplyDelete
  2. Thank you for share the project whith us :)

    but i having some problems. My face is detected by the streamer but when i build and run the project, the cilinder dont move.I have install Face API and 6dofstreamer.
    How i fix this?

    Thx:) sorry for the bad enghish.

    ReplyDelete
  3. @Dennis: Hello Dennis, I have checked everything again, its perfectly working. To troubleshoot this, I have made a simple C# code that can stream the faceAPI streamers data. Its just a 5KB file, download this

    http://www.megaupload.com/?d=9DZ4E56Z

    The source code of that file is very simple
    -------
    using System;
    using System.Net;
    using System.Net.Sockets;
    using System.Text;
    namespace ConsoleApplication1
    {
    class Program
    {
    static void Main(string[] args)
    {
    UdpClient udp = new UdpClient(29129);
    IPEndPoint ipe = new IPEndPoint(IPAddress.Any, 29129);
    while (true)
    {
    byte[] rcv = udp.Receive(ref ipe);
    Console.WriteLine(Encoding.ASCII.GetString(rcv));
    }
    }
    }
    }
    -------

    Make sure that faceAPI Non commercial edition is installed, and 6dofstreamer (socket.exe file) is running, run the newely downloaded file ConsoleApplication1.exe file.

    If you see some weird float data streamed continuously in ConsoleApplication1.exe, then everything is working fine. If not, some problem exists.

    UDP port 29129 is used for communication

    Please check it and let me know whether it works or not...

    ReplyDelete
  4. One general comment on how everything is working.

    1. 6dofstreamer streams your face data by using faceAPI Non commercial edition.
    2. It broadcasts the data in UDP port 29129
    3. From Unity 3D, we use socket programming, we create a UDP client which listens in UDP port 29129 and we get the face data which comprises of about six important components three face positions, three rotations (corresponding to three axes)
    4. We split the data from 6dofstreamer we convert the data to our new scale (I used hit and trial method for this, but if you carefully use some perfect mathematical mapping, you will get very smooth transition) to match our game screen
    5. And we use the Unity's Update() function and continuously alter the position of any game object
    6. So, now when we change our face orientation, the streamer streams new values and our game object is altered accordingly

    This is the complete logic behind this...

    So, I sum up the installation procedure
    (All the links given in the PDF file in original post)

    1. Install faceAPI Non commercial version (file named something like setup_FaceTrackingAPI_NC...exe)
    2. Download 6dofstreamer from google projects (faceapistreamer...exe)
    3. Run socket.exe file from extracted 6dofstreamer
    4. Then make sure that Unity 3D is installed and direct run the unitypackage file (faceAPI.unitypackage file) (link from original post)
    5. Directly Build and Run the project (Windows build)

    Now when the faceAPI recognizes your face, try to move your face back and forth , turn your head right and left. You can see that the camera in our project moves according....

    I hope this helps...

    ReplyDelete
  5. Hi Sriram. I downloaded ConsoleApplication1.exe
    and run,like you said and i cant see any float in ConsoleAplication windows, its just black.

    My unity version is 2.6. this is a problem?

    Dennis.

    ReplyDelete
  6. Huh! Check your firewall...actually the socket.exe runs on UDP socket 29129 and our application will stream form that socket. If by some means our application is not able to reach to that socket, there can be problems. So, the problem is not with Unity, the problem is WITH YOUR PORT. Check whether the port is blocked by some means. And you should get streamed data in ConsoleApplication1.exe, if you get it, then everything is prefect.

    Please check your port settings....and try again...*cheers*

    And everything is working 100% in my friends system. I have checked it... Follow the instructions...

    ReplyDelete
  7. Hi Sriram,

    You seem have done quite some stuff in Unity3d.

    I am new to Unity.

    I am looking for a simple UDP receive and send sort of program with which I can control joints of a character in Unity. Specifically, in my application, a matlab program sends UDP stuff over to Unity that control character joints.

    I am looking for a simple example of UDP receive that can control say a small ball in Unity.

    To save your time,
    I saw this code @
    http://forum.unity3d.com/threads/15900-simple-udp-implementation-%28send-read-via-mono-c-%29
    I downloaded two codes, added them as assets to the project.. and then what...I am kind of stuck...

    If you can give me your own example, thats fine too. Any help is appreciated...Many thanks!

    ReplyDelete
  8. Hey, that is very easy. I have posted a new topic on it. Please check

    http://mypersonalsoft.blogspot.com/2010/12/unity-3d-integration-with-any.html

    I have posted the complete working code there. If any problems you can post there or ask me!

    Please check the new post link

    ReplyDelete
  9. hi Sriram....can you please help me in this VR..to obtain 3d view in the desktop...ive downloaded all the required software...please im trying from a month to do....

    ReplyDelete
  10. Sriram, thank you for the great work.
    But, it's not very convenien to doanload pdf's.
    I have suggestion - you can post pdf's into blogspot using scribd. (Just publish pdf onto scribd.com and get "embed code" for your post, then pdf will be inside your posts). - for example, so is my blog - www.uralvision-en.blogspot.com

    ReplyDelete
  11. Hye Sriram.. a great work you have there.. but i hv some problem with my unity.. it says the error was:

    SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted.

    System.Net.Sockets.Socket.Bind (System.Net.EndPoint local_end)
    System.Net.Sockets.UdpClient.InitSocket (System.Net.EndPoint localEP)
    System.Net.Sockets.UdpClient..ctor (Int32 port)
    script..ctor ()

    everything works fine except this.. can you help me solve this matter..

    ReplyDelete
  12. Hi can I make it works with hand tracking, not with head ?
    Maybe you can create another tutorial with some kind of implementation the Hand tracking to unity3d ? ;>
    Cos i've tried to do it with openCV but i have lot's of trouble with that..
    Thanks in advance ;)

    ReplyDelete
  13. Hey i had already done it, even i have put a demo of if, plz check this:
    http://www.youtube.com/watch?v=-GxykrIB3yM

    And tutorials, i will start it from today, in 1 or 2 days i will post the tutorials, with 100% working code for free!! :)

    ReplyDelete
  14. Hi Sririam,
    thank you for the great work. But i have a problem :/ it dosen't work, i have this message in the console of unity "NullReferenceException: Object reference not set to an instance of an object
    script.Update () (at Assets/Scripts/script.cs:15)
    "
    please help me. thank you sooo much

    ReplyDelete
  15. Hey man,

    Really sorry to say this but I'm getting the same error as the user above.... =(

    (****"NullReferenceException: Object reference not set to an instance of an object
    script.Update () (at Assets/Scripts/script.cs:15)
    "****)

    It would be awesome if you can help resolve this issue As soon as possible, as I've tried all the solutions you've mentioned to no success!

    Thanks!

    ReplyDelete
  16. P.S. Can't wait for the hand tracking tutorials, I keep checking back everyday!! Keep up the great work!

    =)

    ReplyDelete
  17. well...which unity version you guys are using by the way, i guess i used 2.6.1

    i will do one thing, today i dont have the files for unity3d faceapi project, tomorrow, i will check it with unity3d 3.0 and then inform you guys, also i have the complete build of the working project, i will upload it tomorrow...thanks for the support!

    And unity3d handvu, by this weekend, i will definitely release the code!!

    Thanks all!

    ReplyDelete
  18. Thanks so much man, you're the best.... And yes I'm using Unity 3 as I only jumped on the unity bandwagon two months ago and Unity 3 was the latest one out! I Look forward to you updates over the weekend!!

    Regards

    Jazeem

    ReplyDelete
  19. Hello!
    When I double-click the socket.exe file, it show me a console window and close. Is that alright?

    ReplyDelete
  20. hi Sriram .A.S i see your blog and i think this is very interesting
    I followed your tutorial and i donwload unity3D v3.3 faceapi non commercial v3.2.6, i work in windows 7 64 bits but when i try tu build and run the project i get the next error

    NullReferenceException: Object reference not set to an instance of an object
    script.Update () (at Assets/Scripts/script.cs:15)

    I donwloade de consolapplication1.exe but when i try to run i see an error window and it close

    I hope you can help me

    Victor

    ReplyDelete
  21. thanks for reply good luck with exams =)

    Victor

    ReplyDelete
  22. Hi,
    I was in touch with you previously for UDP based control of a Unity Character from MATLAB/ LabView.

    Having a small issue here.

    In my csharp code, here is a pseudocode, from which i built an executable.
    ======================================================
    Declarations
    :
    :
    Public class client: MonoBehvaiour
    {
    public Game Object

    Void Start()
    {
    UDP port declaration etc...

    GameObject.Find(); //etc
    }//Start ends

    Void Update()
    {

    if ( (Input.GetKey("escape")) )
    Application.Quit();
    else
    {
    Read from UDP packets

    Update character Arm position etc...
    }

    }//update ends


    }//Public Class ends
    ======================================================

    When ever the MATLAB or LabView application ends or hangs abruptly due to some known reasons,
    I want to be able to quit Unity Executable neatly without killing it from task manager.
    Right now it seem to hang waiting for UDP packets. As it is hung it wont take the Esc key to quit.

    Thanks for the help.

    ReplyDelete
  23. Hey all, im back to my home and have good Internet connection. I resolved that bug "Null reference", download the new script.cs file and replace the original script.cs file from /scripts in the unitypackage!!

    here is the new file:
    http://www.megaupload.com/?d=BOOK6WCD

    I have made a video tutorial on how to make Unity3d faceAPI stuff to work. I guess that will help all of you. I have completed the video tutorial, now compressing it, just in few minutes i will upload the video to youtube and give the link...

    thanks to all for your patience...from now i have good Internet, lets work on some cool stuffs :)

    ReplyDelete
  24. @UnityStudent
    You can do it like this, use a try, catch in Unity c# script, when you receive data from MATLAB using Receive() function, and if MATLAB closes abruptly, it will generate an exception, check the manual for what type of exception it will cause, you can use this exception in Unity and can also close Unity safely!!

    Hope this works...!!

    ReplyDelete
  25. @Dog
    Hey make sure that faceAPI non commercial setup is installed. Just in few minuted, i will give the link of youtube video with full video tutorials of Unity3d faceAPI...it will answer all the questions

    ReplyDelete
  26. Guys the video tutorial is uploaded and live now...

    http://mypersonalsoft.blogspot.com/2011/05/unity3d-faceapi-integration-to-achieve.html

    http://www.youtube.com/watch?v=4SBAEgMa4Ng

    Please do check it...

    ReplyDelete
  27. is it possible to deploy it online, to make it work in realtime like a web cam inititated augmented reality thing.

    ReplyDelete
  28. yeah ofcourse. in this example, the server socket.exe file runs in my local machine and i used "localhost" in C#, if the server is running somewhere else, enter the IP of that machine and port, and it will work across network!

    ReplyDelete
  29. the latest file u uploaded in megaupload.com cannot be found.
    maybe u could upload some other server for me to download?

    I am also facing problem that when i activated the script, i tried playing it, it cause my whole unity program not responding. then i have to close it and re open it again..
    Do u know y?

    ReplyDelete
  30. Ahh!! OMG! Megaupload is shutdown because of some piracy law. Much of the files i uploaded in that only... I will reupload all. Thanks for reporting.

    And regarding crashing, please check if the response is coming proper in that port, check the 3rd comment in same post, i posted a simple code to check the port output from faceapi, first check if response is proper, then we will debug further. Thankyou.

    ReplyDelete
  31. Can you post this files in another place? like mediafire?

    ReplyDelete
  32. hello Sriram. i really liked your post.its very useful! :)) but im afraid i couldnt download your modified script.cs because megaupload is closed now. can i get it from another mirror? thanks

    warm regards,

    ReplyDelete
  33. Can someone please re-up the corrected script.cs file??? Maybe even post the source here in this blog? Can't wait to us it:)

    ReplyDelete
  34. Hello ..
    I don't have permission to access the Main script link .. So please what's wrong ??

    ReplyDelete
  35. Hi all. I re-uploaded new script.cs file. Please check the link given at the last of the post, it contains the video of how to assemble all. From that i identified script.cs and uploaded it back :)

    ReplyDelete
  36. Hi sriram...

    my requirement matches this article...

    now megaupload is siezed....
    so from where i have to download your code...

    please help me...

    Thanks in advance...

    ReplyDelete
  37. Hi Sriram , thanks for tutorials . my socket.exe throughs the exception : " An exception occured . exception Nr"
    do you know how to solve that ?
    thanks

    ReplyDelete
  38. in the pdf, this no longer exist .
    "
    The main script (as Unity Package)
    http://megaupload.com/?d=YKX2B382
    "

    have any other links? .
    Thanks, this is awesome..

    ReplyDelete
  39. Hi

    Is there a way to stream the video into Unity as display as a texture on a plane?

    thanks

    Paul

    ReplyDelete
  40. Hello Sir,

    Would you please let us know how to get the socket address.

    ReplyDelete