Wednesday, December 15, 2010

Unity 3D Integration with any application/code/tool [TUTORIAL]

So, have you got a very good algorithm for Real Time eye detection with webcam in MATLAB, and want to integrate it with Unity 3D like, the realtime coordinates of your eye alter a game object or are you having a cool hand detection algorithm in any language and want to integrate it with Unity 3D? You have come to the right place.


Any application/external code can be integrated with Unity 3D. The concept is very simple, we are going to make them (unity and the external app/code) speak the same language (i.e.) Socket Programming. If you are new to Socket Programming, I recommend you to read some tutorial. Anyways, here I have done a simple Unity 3D UDP Server/Client as example. This code can be used to integrate Unity 3D with other codes/applications/tools.

Tutorials::
Download the project files first, check the end of this post for download link

Unity 3D UDP Server
1. Open Unity 3D and create a new project
2. Right Click in "Project" pane and do: Create-->C Sharp Script
3. Now you get a new script called NewBehaviourScript in your project pane
4. Download the attachment with this post and copy the code of "server.cs" file from the downloaded file into the NewBehaviourScript
5. Make sure that you rename NewBehaviourScript to server
6. Attach this script(server) to any gameobject, for simplicity, Main Camera
7. Go to Edit-->Project Settings-->Player and check "Run In Background" (Unity 3D applications by default, freeze when the focus is removed from them, so to prevent it, we use this)
8. Build and Run this project

Unity 3D UDP Client

1. Open Unity 3D and create a new project
2. Right Click in "Project" pane and do: Create-->C Sharp Script
3. Now you get a new script called NewBehaviourScript in your project pane
4. Download the attachment with this post and copy the code of "client.cs" file from the downloaded file into the NewBehaviourScript
5. Make sure that you rename NewBehaviourScript to client
6. Attach this script(client) to any gameobject, for simplicity, Main Camera
7. Go to Edit-->Project Settings-->Player and check "Run In Background" (Unity 3D applications by default, freeze when the focus is removed from them, so to prevent it, we use this)


For testing, we will use both client and server from Unity 3D
Make sure that server application is running, Run the client application from within Unity 3D itself, because the code that I have given uses print() function and you can see it within Unity 3D only. Otherwise, you have to check the logs.

You can now observe that inside Unity 3D, client application prints integers, which is sent by none other than the independent and separate server application.

Now, use your imagination to build some cool Unity 3D applications like you can send your real time eye coordinates or say eye blink etc., from webcam through any application/language/tool and use UDP client of this project and use this data to control any gameobject inside Unity 3D
These are few example projects based on this idea:
http://www.youtube.com/watch?v=yN_YEnDs2uk
http://www.youtube.com/watch?v=-GxykrIB3yM

Project Files
Download Link 1:
http://www.megaupload.com/?d=IIV1H4WB

44 comments:

  1. This might be a little dumb Question for you.

    I was not able to run two unity projects simultaneously. Unity keeps crashing.

    Instead I imported two codes server and client into assets of one project. and ran the project and nothing happens!

    ReplyDelete
    Replies
    1. Hy unityStudent,
      Alan is my name and i try to make a communication between Matlab and Unity3d.
      i wanted to ask you if you can send me you code. because you to already do it.
      please i need it to try to finish my Work.

      I will be really happy,if you can do it.

      thank for your reply. Alan

      Delete
  2. But I used this in MATLAB

    >> u=udp('127.0.0.1',1000);
    >> fopen(u)
    >> fwrite(u, '1.14')

    and could see the number on Unity Client Side.

    This is a great start. I really appreciate your help

    ReplyDelete
  3. You cant run two projects in Unity. Thats why I told you. First do the server, and BUILD its exe and run it (externally as windows application). Make sure its running. Then you open Unity and make the client and now you can run Client from inside Unity itself! Then you can see in the client output bottom screen (print area), integers put by server to client!
    Its perfectly working in mine. I recommend you to try it with Unity's client and server first.

    Try this method, and it will work and plz post your comments. Any type of questions are invited!!

    ReplyDelete
  4. Working with UnityServer is not my priority anyway. In my program as I said Matlab talks to Unity. So we are good there...

    some problems I am facing right now:

    1. Really weird is the numbers already sent are not accepted.
    Say I send 7.14 and I do 9.14 and later I send 7.14, Client did not seem to update the print.
    But I do 8.14 it updates the print.

    2. Seems to be really slow. Can i really use it in real time?!!!!

    3. Client seems to hang pretty bad and Unity crashes to close the program (probably because server/sender is not on all the time).

    ReplyDelete
  5. I guess I solved the problems 2 and 3. It was just the order of the handshake. Now works pretty well. But Problem 1 in my previous post still remains mysterious.

    I use this code to print numbers in Unity

    u = udp('127.0.0.1', 1000);
    fopen(u);
    fwrite(u, '1');
    fwrite(u, '1');
    pause()

    for i=1:1000
    p=num2str(i);
    fwrite(u, p);
    pause(0.1)
    i
    end

    This works fine. Keeps the pace and prints numbers in real time.

    If I add another for loop such as this to the code, it wont print the numbers any more..it is stuck at 1000.

    for i=1000:-1:1
    p=num2str(i);
    fwrite(u, p);
    pause(0.1)
    i
    end

    ReplyDelete
  6. I learnt a lot in short time about UDP control in Unity. Now I am able to send a string of numbers from LabView and MATLAB and print them in real time.

    Would you mind posting some simple code, may be a box in the scene and how do i go about controlling it (translating and rotating) with this UDP packet.
    I guess I can drag a box. BUt how do I control it with the received numbers.

    Sorry for too many questions. You seem to have the right answers for me. There is a whole unity forum which could not give me an answer. Multiple users posted their codes, but none wrote how to embed them in project and use them.

    Thanks again for your help.

    ReplyDelete
  7. You can ask any number of questions here, dont worry even if they are silly, i will try to answer the questions as much as possible, if i know the answer :)

    And regarding a box control in Unity, I have done a simple script that can help you. Download this file

    http://www.megaupload.com/?d=FYPJM5JH

    Steps:
    1. Create a new Unity project
    2. Export this package

    Make sure that from server you dont send much bigger data and better delay each value by atleast a second (pause() in MATLAB) to check the live movement. Try from 1 to 5...

    The code contains its explanation also. You will find that i just did two alterations to the old code (originally from this main post)

    1. Removed print() from client and added a new line (check the package)
    2. Attached the script this time to the cude, whereas, last time it was with main camera!

    Any more doubts, codes you are free to ask...!!

    ReplyDelete
  8. Actually I got it to working using the following code. I just saw your post. I will use yours too and learn more and post some feedback.

    void Update ()
    {
    //Start receiving
    byte[] k = udp.Receive(ref ipe);
    string g = Encoding.ASCII.GetString(k);
    //print(Encoding.ASCII.GetString(k));

    //Split the string of several numbers
    System.String[] nums = g.Split(',');
    //print(nums[3]);

    //Convert to double
    double m = double.Parse(nums[3]);
    //print(double.Parse(nums[3]) );
    //print(m);
    //print(System.Convert.ToInt32(m));

    //Rotate the box. Code is still attached to Main Camera.
    //Controls the speed of rotation by 3rd number
    go1 = GameObject.Find("MyBox");
    go1.transform.Rotate(0, System.Convert.ToInt32(m), 0);

    }

    ReplyDelete
  9. In your script and my script, I guess we were restricted to using integers (Convert.ToInt32 / Int32.Parse ) to control the position or velocity of rotation/translation of a gameObject.

    I was wondering this seems like a major limitation, as the real signals are not integers.

    Is this Unity Limitation? Is there a work around for it?

    Thanks,

    ReplyDelete
  10. Hey no problem, i was taking integer as an example. Instead of Int32.parse(), use float.Parse(). You will get float then!!

    Hope this helps!

    ReplyDelete
  11. What is a good way to rotate joints?

    I used

    gameObject1.transform.eulerAngles = new Vector3(0, 180, float.Parse(nums[4]));
    gameObject1.transform.rotation = new Quaternion(0, 180, float.Parse(nums[4]),1);

    nums[4] is a number in the string of numbers that is UDPed into unity from LabView.

    The problem I am having with both eulerAngles as well as rotation is that when ever I control the rotation along Z axis or angles around Z axis, the other angles seem to vary simultaneously.

    Ideally what I want is the angles should be set to 0 for X, 180 for Y and real time changing number for Z. Can you help me with this?

    Also, whats fourth input in Quaternion?

    Many thanks!

    ReplyDelete
  12. Hey,

    I am planning to use multiple cameras in Unity3d.
    Do you have any tutorial that talks about that?
    I would like to do: within the main scene, I want to provide inset views from two other cameras that front view and side view, for example.

    ReplyDelete
  13. Sorry buddy, i have not tried multiple cameras much, but still, check the unity manual, it has a cool code for multiple cameras...if you have got some cool stuffs, you can share it with us also!! we will also be happy and benefited.

    ReplyDelete
  14. I am trying to insert some simple code in my c# program to detect collision between hand of the virtual arm and a sphere in the space. I have inserted spherical colliders on both of them. Is Trigger checked. and am looking for code...
    tried this does not do anything, does not complain as wrong...

    if (Collision.Equals(Targ, go7) )
    {
    go7.transform.position = new Vector3(-3, 3, 1);
    print("Collision");
    }

    Any help appreciated

    ReplyDelete
  15. I am a beginner in 3D unity. today, I want to create a 3D simulation using a unity, by implementing the algorithm Artificial Intelligent-Multiobjectif GA that I've created in Matlab. Can help me about the tutorial? how step-by step? whether the application also uses this UDP application? maybe you can help, since I was a layman and a new learning

    ReplyDelete
  16. I am a beginner in 3D unity. today, I want to create a 3D simulation using a unity, by implementing the algorithm Artificial Intelligent-Multiobjectif GA that I've created in Matlab. Can help me about the tutorial? how step-by step? whether the application also uses this UDP application? maybe you can help, since I was a layman and a new learning

    ReplyDelete
  17. Hi Sriram
    My Name is Laurence - Durban, South Africa.
    PLEASE HELP. I have scoured the web for a C# answer that works in UNITY 3D. No luck.
    I am using a VB 6 form to send a value to unity using your 'client.cs' script to recieve it. This works 100%. I am able to rotate the cube. Thanks for sharing your knowledge.

    What I need to do is send say 3 or more values from the form, for example: "1,23,180" then using your client.cs script to split that into 3 values using the ',' as a deliminator. Then to put those values into seperate variables to use to rotate the cube, up , right etc.
    If you can help, it will be much appreciated.
    As they said in Star Wars, "You are our only hope Obiwan..."
    Many thanks -Laurence

    ReplyDelete
  18. Hi friend, sure we can do that. From VB, when button click event is performed, just concatenate all three values and send as one. I guess there wont be difficulty in that part. Now from CSharp, do like this, let us assume "text" is string type variable that contains the data in CSharp that you received from VB. Now you can get individual components by the delimiter by this type of statement:

    --------------------------------
    string text=RECEIVED_DATA;
    string[] parts = text.Split(',');
    --------------------------------

    now you can access individual elements by parts[0], and so on. Hope this helps. :)

    ReplyDelete
  19. @UnityStudent: Hello friend, please try with a mesh collider. Seriously i have not tried it, but sure, if you elaborate what you need, i can also try, may be i can help you in finding the solution :)

    @dwi rolliawati: this can be done easily with the codes available in the same page. Try downloading the codes in the same page and see, if you have any problems, i can write the complete code for you, for the integration part. Please email me your elaborated requirement to my email:
    sriramdasty7@gmail.com

    ReplyDelete
  20. Hello there Sriram!

    This is maybe a little bit off-topic, but is this method can be used to integrate speech recognition to unity? I am currently working on a project using CMU Sphinx to integrate into Unity with no success.

    This is the CMU Sphinx project website:
    http://cmusphinx.sourceforge.net/

    And I'm really looking forward to any suggestion in making this particular system works in Unity. Thanks before.

    ReplyDelete
  21. Hey sorry, didnt see this post. Im now in my home for vacation, i dont have my laptop and JVM with me. Anyways, i will see that one and will post you what could be done!!

    ReplyDelete
    Replies
    1. plz upload the file again coz megaupload is DIE >.< thx a lot

      Delete
  22. Hi
    The Link Not Work please Re upload

    ReplyDelete
  23. The Link Not Work please Re upload +1
    Thank you.

    ReplyDelete
  24. Hi,
    I would also like to be able to download the files for this project - can you give us a new link please?
    thanks in advance

    ReplyDelete
  25. Please can you relink your files?? Thanks =)

    ReplyDelete
  26. Megaupload is no longer. Would you mind posting the files elsewhere, or even better doing a new UDP client/server demo for unity4. TIA!

    ReplyDelete
  27. Can you reupload please? :(

    Thanks!

    ReplyDelete
  28. Megaupload don't work anymore... :(
    Please Reupload!!!

    ReplyDelete
  29. Link is DEADBEEF. Could you please upload?

    ReplyDelete
  30. Could you please upload the code again
    Thanks

    ReplyDelete
  31. Could you please upload the code

    ReplyDelete
  32. Hi Sriram, This is Ram here. I am new to Unity and not familiar with communication between applications. Also, not so good in coding. I am looking for a sample that helps in sending data (like position of an object in x, y, z coordinates) from external C++ application to Unity application where I have a car placed on the terrain. Could you please support me with a sample.

    ReplyDelete
  33. Any chance to re-upload the source files?

    ReplyDelete
  34. Aaaargh... This is what I need but the link is dead :(

    ReplyDelete
  35. Hi Sriram, this projects seems great! Is it possible for you to reactive the sample code and files you provided?

    ReplyDelete
  36. Hey Sriram, I was wondering if what you suggested can be done without internet . Like sharing simple data between an APK and a unity3d game from within internal storage of a android device . Can that be done ?

    ReplyDelete
  37. Hi Sriram, this projects seems great! Is it possible for you to reactive the sample code and files you provided?

    thank to you.

    ReplyDelete
  38. Please reupload, thanks!

    ReplyDelete
  39. The upload link doesn't work!

    ReplyDelete
  40. I think those who did have the fortune of downloading the sample code, could also share the uploaded code perhaps?

    ReplyDelete