Thursday, December 2, 2010

Unity 3D FaceAPI Integration Tutorials

First of all Im really sorry for taking a long time for releasing this awaited tutorials with all working codes.

Since I dont have web space of my own, I rely on free file hostings, I have uploaded the required files with the mirrors::

Tutorials PDF file: (Only the below two files/links are enough, other links are mirror links)

Full Project (Unity Package File):

Download the PDF file, that contains links of files to be downloaded...

In new Unity versions, there is an error "Null reference", this bug is fixed. Download this file
and replace the original script.cs file from /scripts in the unity package...

Thank god, found this script from my video(10:37), dont forget to check the video. Check the link down.

Dont forget to view this post:


Please don't forget to post your comments here. This is my first tutorial, and your comment will really help me in improving myself...

And Unity 3D handVU Integration Tutorials will take some more time...sorry again friends!!


  1. Great work and nice to share! Thank you very much. Now, let's study it :D

  2. Thank you for share the project whith us :)

    but i having some problems. My face is detected by the streamer but when i build and run the project, the cilinder dont move.I have install Face API and 6dofstreamer.
    How i fix this?

    Thx:) sorry for the bad enghish.

  3. @Dennis: Hello Dennis, I have checked everything again, its perfectly working. To troubleshoot this, I have made a simple C# code that can stream the faceAPI streamers data. Its just a 5KB file, download this

    The source code of that file is very simple
    using System;
    using System.Net;
    using System.Net.Sockets;
    using System.Text;
    namespace ConsoleApplication1
    class Program
    static void Main(string[] args)
    UdpClient udp = new UdpClient(29129);
    IPEndPoint ipe = new IPEndPoint(IPAddress.Any, 29129);
    while (true)
    byte[] rcv = udp.Receive(ref ipe);

    Make sure that faceAPI Non commercial edition is installed, and 6dofstreamer (socket.exe file) is running, run the newely downloaded file ConsoleApplication1.exe file.

    If you see some weird float data streamed continuously in ConsoleApplication1.exe, then everything is working fine. If not, some problem exists.

    UDP port 29129 is used for communication

    Please check it and let me know whether it works or not...

  4. One general comment on how everything is working.

    1. 6dofstreamer streams your face data by using faceAPI Non commercial edition.
    2. It broadcasts the data in UDP port 29129
    3. From Unity 3D, we use socket programming, we create a UDP client which listens in UDP port 29129 and we get the face data which comprises of about six important components three face positions, three rotations (corresponding to three axes)
    4. We split the data from 6dofstreamer we convert the data to our new scale (I used hit and trial method for this, but if you carefully use some perfect mathematical mapping, you will get very smooth transition) to match our game screen
    5. And we use the Unity's Update() function and continuously alter the position of any game object
    6. So, now when we change our face orientation, the streamer streams new values and our game object is altered accordingly

    This is the complete logic behind this...

    So, I sum up the installation procedure
    (All the links given in the PDF file in original post)

    1. Install faceAPI Non commercial version (file named something like setup_FaceTrackingAPI_NC...exe)
    2. Download 6dofstreamer from google projects (faceapistreamer...exe)
    3. Run socket.exe file from extracted 6dofstreamer
    4. Then make sure that Unity 3D is installed and direct run the unitypackage file (faceAPI.unitypackage file) (link from original post)
    5. Directly Build and Run the project (Windows build)

    Now when the faceAPI recognizes your face, try to move your face back and forth , turn your head right and left. You can see that the camera in our project moves according....

    I hope this helps...

  5. Hi Sriram. I downloaded ConsoleApplication1.exe
    and run,like you said and i cant see any float in ConsoleAplication windows, its just black.

    My unity version is 2.6. this is a problem?


  6. Huh! Check your firewall...actually the socket.exe runs on UDP socket 29129 and our application will stream form that socket. If by some means our application is not able to reach to that socket, there can be problems. So, the problem is not with Unity, the problem is WITH YOUR PORT. Check whether the port is blocked by some means. And you should get streamed data in ConsoleApplication1.exe, if you get it, then everything is prefect.

    Please check your port settings....and try again...*cheers*

    And everything is working 100% in my friends system. I have checked it... Follow the instructions...

  7. Hi Sriram,

    You seem have done quite some stuff in Unity3d.

    I am new to Unity.

    I am looking for a simple UDP receive and send sort of program with which I can control joints of a character in Unity. Specifically, in my application, a matlab program sends UDP stuff over to Unity that control character joints.

    I am looking for a simple example of UDP receive that can control say a small ball in Unity.

    To save your time,
    I saw this code @
    I downloaded two codes, added them as assets to the project.. and then what...I am kind of stuck...

    If you can give me your own example, thats fine too. Any help is appreciated...Many thanks!

  8. Hey, that is very easy. I have posted a new topic on it. Please check

    I have posted the complete working code there. If any problems you can post there or ask me!

    Please check the new post link

  9. hi Sriram....can you please help me in this obtain 3d view in the desktop...ive downloaded all the required software...please im trying from a month to do....

  10. Sriram, thank you for the great work.
    But, it's not very convenien to doanload pdf's.
    I have suggestion - you can post pdf's into blogspot using scribd. (Just publish pdf onto and get "embed code" for your post, then pdf will be inside your posts). - for example, so is my blog -

  11. Hye Sriram.. a great work you have there.. but i hv some problem with my unity.. it says the error was:

    SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted.

    System.Net.Sockets.Socket.Bind (System.Net.EndPoint local_end)
    System.Net.Sockets.UdpClient.InitSocket (System.Net.EndPoint localEP)
    System.Net.Sockets.UdpClient..ctor (Int32 port)
    script..ctor ()

    everything works fine except this.. can you help me solve this matter..

  12. Hi can I make it works with hand tracking, not with head ?
    Maybe you can create another tutorial with some kind of implementation the Hand tracking to unity3d ? ;>
    Cos i've tried to do it with openCV but i have lot's of trouble with that..
    Thanks in advance ;)

  13. Hey i had already done it, even i have put a demo of if, plz check this:

    And tutorials, i will start it from today, in 1 or 2 days i will post the tutorials, with 100% working code for free!! :)

  14. Hi Sririam,
    thank you for the great work. But i have a problem :/ it dosen't work, i have this message in the console of unity "NullReferenceException: Object reference not set to an instance of an object
    script.Update () (at Assets/Scripts/script.cs:15)
    please help me. thank you sooo much

  15. Hey man,

    Really sorry to say this but I'm getting the same error as the user above.... =(

    (****"NullReferenceException: Object reference not set to an instance of an object
    script.Update () (at Assets/Scripts/script.cs:15)

    It would be awesome if you can help resolve this issue As soon as possible, as I've tried all the solutions you've mentioned to no success!


  16. P.S. Can't wait for the hand tracking tutorials, I keep checking back everyday!! Keep up the great work!


  17. well...which unity version you guys are using by the way, i guess i used 2.6.1

    i will do one thing, today i dont have the files for unity3d faceapi project, tomorrow, i will check it with unity3d 3.0 and then inform you guys, also i have the complete build of the working project, i will upload it tomorrow...thanks for the support!

    And unity3d handvu, by this weekend, i will definitely release the code!!

    Thanks all!

  18. Thanks so much man, you're the best.... And yes I'm using Unity 3 as I only jumped on the unity bandwagon two months ago and Unity 3 was the latest one out! I Look forward to you updates over the weekend!!



  19. Hello!
    When I double-click the socket.exe file, it show me a console window and close. Is that alright?

  20. hi Sriram .A.S i see your blog and i think this is very interesting
    I followed your tutorial and i donwload unity3D v3.3 faceapi non commercial v3.2.6, i work in windows 7 64 bits but when i try tu build and run the project i get the next error

    NullReferenceException: Object reference not set to an instance of an object
    script.Update () (at Assets/Scripts/script.cs:15)

    I donwloade de consolapplication1.exe but when i try to run i see an error window and it close

    I hope you can help me


  21. thanks for reply good luck with exams =)


  22. Hi,
    I was in touch with you previously for UDP based control of a Unity Character from MATLAB/ LabView.

    Having a small issue here.

    In my csharp code, here is a pseudocode, from which i built an executable.
    Public class client: MonoBehvaiour
    public Game Object

    Void Start()
    UDP port declaration etc...

    GameObject.Find(); //etc
    }//Start ends

    Void Update()

    if ( (Input.GetKey("escape")) )
    Read from UDP packets

    Update character Arm position etc...

    }//update ends

    }//Public Class ends

    When ever the MATLAB or LabView application ends or hangs abruptly due to some known reasons,
    I want to be able to quit Unity Executable neatly without killing it from task manager.
    Right now it seem to hang waiting for UDP packets. As it is hung it wont take the Esc key to quit.

    Thanks for the help.

  23. Hey all, im back to my home and have good Internet connection. I resolved that bug "Null reference", download the new script.cs file and replace the original script.cs file from /scripts in the unitypackage!!

    here is the new file:

    I have made a video tutorial on how to make Unity3d faceAPI stuff to work. I guess that will help all of you. I have completed the video tutorial, now compressing it, just in few minutes i will upload the video to youtube and give the link...

    thanks to all for your patience...from now i have good Internet, lets work on some cool stuffs :)

  24. @UnityStudent
    You can do it like this, use a try, catch in Unity c# script, when you receive data from MATLAB using Receive() function, and if MATLAB closes abruptly, it will generate an exception, check the manual for what type of exception it will cause, you can use this exception in Unity and can also close Unity safely!!

    Hope this works...!!

  25. @Dog
    Hey make sure that faceAPI non commercial setup is installed. Just in few minuted, i will give the link of youtube video with full video tutorials of Unity3d will answer all the questions

  26. Guys the video tutorial is uploaded and live now...

    Please do check it...

  27. is it possible to deploy it online, to make it work in realtime like a web cam inititated augmented reality thing.

  28. yeah ofcourse. in this example, the server socket.exe file runs in my local machine and i used "localhost" in C#, if the server is running somewhere else, enter the IP of that machine and port, and it will work across network!

  29. the latest file u uploaded in cannot be found.
    maybe u could upload some other server for me to download?

    I am also facing problem that when i activated the script, i tried playing it, it cause my whole unity program not responding. then i have to close it and re open it again..
    Do u know y?

  30. Ahh!! OMG! Megaupload is shutdown because of some piracy law. Much of the files i uploaded in that only... I will reupload all. Thanks for reporting.

    And regarding crashing, please check if the response is coming proper in that port, check the 3rd comment in same post, i posted a simple code to check the port output from faceapi, first check if response is proper, then we will debug further. Thankyou.

  31. Can you post this files in another place? like mediafire?

  32. hello Sriram. i really liked your post.its very useful! :)) but im afraid i couldnt download your modified script.cs because megaupload is closed now. can i get it from another mirror? thanks

    warm regards,

  33. Can someone please re-up the corrected script.cs file??? Maybe even post the source here in this blog? Can't wait to us it:)

  34. Hello ..
    I don't have permission to access the Main script link .. So please what's wrong ??

  35. Hi all. I re-uploaded new script.cs file. Please check the link given at the last of the post, it contains the video of how to assemble all. From that i identified script.cs and uploaded it back :)

  36. Hi sriram...

    my requirement matches this article...

    now megaupload is siezed....
    so from where i have to download your code...

    please help me...

    Thanks in advance...

  37. Hi Sriram , thanks for tutorials . my socket.exe throughs the exception : " An exception occured . exception Nr"
    do you know how to solve that ?

  38. in the pdf, this no longer exist .
    The main script (as Unity Package)

    have any other links? .
    Thanks, this is awesome..

  39. Hi

    Is there a way to stream the video into Unity as display as a texture on a plane?



  40. Hello Sir,

    Would you please let us know how to get the socket address.