Playing 360 video in HoloLens Unity 3D app

HoloSheep

The "developer highlights" for HoloTour (about half way down the page) mention the use of off the shelf 360 cameras to capture video content to playback inside our own apps that create similar 360 experiences to HoloTour.

Has anyone seen or worked with any samples or tutorials that would help with getting started in the job of taking Theta S 360 output and displaying that in a 360 experience inside a Unity HoloLens app?

Jimbohalo10

Here is a quote from a user
"Photos are in .jpg format 1920x1080 I think and they appear as rectangular but unless you have a 360 degree viewer like the Ricoh Theta one, they appear distorted especially at the edges. To view the photos they have to be exported by wifi from the Theta to a linked device like an iPad. "

I can see this in Amazon UK, purchase with software, must be on Amazon US, but the comments at bottom of page, as above are interesting.

Ricoh-Theta-S-Camera-Black

Here is link to the User Manual https://theta360.com/uk/support/manual/s/content/pc/pc_04.html

HoloSheep

Thanks for the link @Jimbohalo10,

I have Theta S, but was looking for guidance on ways to display that kind of output in a Unity HoloLens app to experience the videos and stills in an immersive 360 experience similar to what has been done in HoloTour.

Jimbohalo10

Update:-
I have just realised. Do you want to migrate the Ricoh Theta S Camera software to Unity HoloLens app? Well converting and compiling code would be easier in Visual Studio C++/C# template, but then you would need to display using DirectX3D and to connect via the USB to the HoloLens device.

You may be able to use the Web Cam extensions. If that's what the Ricoh Theta S Camera looks like when connected to a PC, but would a second camera, other than Kinect, be would this be seen and available in the HoloLens Device.
In another answer about Web Cam, this is not supported in the emulator, or the HoloLens device.

First thing would just be to convert and compile and run the Ricoh Theta S Camera software in Unity/Visual Studio, then think about maybe merging it in the 360 degree VR Player Starter Kit described below.

---

Other Answers
Could to convert to HoloLens by just changing the VR Supported device to HoloLens from the Unity Asset Store 360 degree VR Player Starter Kit
Watch the YouTube video demo of Unity3D 360 VR Starter Kit

In the video you see locations just like Holo Tour, from you own position, you would have to add the Spatial Mapping Gaze functions.

Try 360 Panorama Capture

Cannot load the 360 VR Camera Capture Rig but looks good

In U.K., where I live, so can use HoloLens emulator only.

Fireman

I am looking for the same thing.

In Unity asset store, there is an extention to convert Google street view to a cubemap. Somthing to think about.

Fireman

wintermoot

+1 to Theta viewer

jonflanders

I took a 360 picture - and added it as a texture on a Sphere in Unity. Then added a script that reversed the normals - cribbed it from here - http://wiki.unity3d.com/index.php?title=ReverseNormals

Jimbohalo10

@jonflanders said:
I took a 360 picture - and added it as a texture on a Sphere in Unity. Then added a script that reversed the normals - cribbed it from here - http://wiki.unity3d.com/index.php?title=ReverseNormals

That's a great idea. I just used your idea on Web Cam in Unity

wc = new WebCamTexture ();
sp = new Texture2D (wc.width, wc.height);
GameObject.Find ("/Sphere").GetComponent<Renderer().material.mainTexture = sp;
sp.SetPixels(wc.GetPixels());
sp.Apply();

Just so easy in Unity

Obviously this does not run on HoloLens Emulator, but will deploy to Lumina Windows 10, running latest build, quite a bit slower on phone camera.

Therefore probably works on HoloLens device.
Thanks again

yena

The webcametexture works on unity editor but does not work on hololens device. Atleast it doesn't for me.

Jimbohalo10

@yena said:
The webcametexture works on unity editor but does not work on hololens device. Atleast it doesn't for me.

Yes you are right you need to use locatable-camera-in-unity

To get the HoloLens camera image and convert this to texture same as a webcam

Jimbohalo10

Interested in making 360 degree 3D picture of yourselves (selfies!! ) and wanting something easy to use.
I have Kinect 2 for Windows, basically an Xbox One Kinect with a USB 3 adapter, on same Desktop PC for HoloLens Emulator. Cost under £200 from Amazon UK.

This comes with masses of free software support, with ALL the source code to build you own version. Window 10 Insider build 14xxx onward support as this well and brings in with it Windows Store Apps 3D Scan and 3D Builder.

Set the Kinect 2 at arms length away looking at you level with your face. Start 3D Scan and very slowly turn on your swivel chair. When you stop 3D Builder is called and repair will create a the 3D picture of you after about 60 minutes, depending on your PC.
This picture can be saved imported into Unity.

Finally there is an article gesture-control-in-unity-via-kinect-for-windows/ about using Gestures in Unity. This hardware and software may be an alternative for Non-US based users who cannot get HoloLens device yet

Vdoman

Hi Guys!! Great info. Im a VR 360 videographer , I shoot with a 7 cam rig. It would be awesome to view my videos in my hololens using avail apps...is this even possible or do I need to use unity as the above have stated?

christsaragih

Hy I Have try to play 360 Video on Unity Hololens. But The FPS is very Low. On Hololen i just get 6-8 FPS. Anyone get High FPS using 360 Video?

HoloSheep

Unity announced yesterday during their keynote at Unit that they are working on a new video player for Unity 5.6 that will handle 4K 360 video with multi platform support. Fingers crossed that UWP and Windows Holographic are one of the initial platforms that the new player supports.

christsaragih

@HoloSheep said:
Unity announced yesterday during their keynote at Unit that they are working on a new video player for Unity 5.6 that will handle 4K 360 video with multi platform support. Fingers crossed that UWP and Windows Holographic are one of the initial platforms that the new player supports.

in uity site just unity 5,5

HoloSheep

@christsaragih

Yes, 5.6 is still showing as at the Alpha stage according to the Roadmap.
Note that 5.5 shows a target of November, so we may see a Beta of 5.6 in the next month or so.

christsaragih

@HoloSheep said:
@christsaragih

Yes, 5.6 is still showing as at the Alpha stage according to the Roadmap.
Note that 5.5 shows a target of November, so we may see a Beta of 5.6 in the next month or so.

yup i have see that. Hmm Okay. It's possible for VR Right? Thanks BTW

HoloSheep

@christsaragih
Here is the quote from the Unity blog:

We also heard from our VR Principal Engineer Dioselin Gonzalez who announced our completely new video player. We’ve rebuilt this from scratch with performance in mind, so we can support 360 VR videos and smooth 4K playback. The video player will be coming to Unity 5.6, with a preview coming soon.

The YouTube version of the keynote's video & audio sync seems to be messed up at the moment, but once they fix it you should be able to seek out her short 4 minute presentation currently starting around the 1:43 mark which showed her walking through the steps to create a simple sphere based app that leveraged the player for a 360 VR experience with a few clicks. Also note that the person who announced the new player has the title of VR Principal Engineer, so it is a pretty safe bet that the player will make it possible to make performant 360 VR experiences on supported VR platforms.

HTH.

Ankit09_Sangani

Hello All,
this working fine for me
http://theta360.guide/blog/video,/unity/2016/10/16/video-file-on-unity-sphere.html

Thanks,
Ankit

HanserWork1

@Ankit09_Sangani
Sorry to bump this. Thank you for the link, Ankit09_Sangani. This works for me too, but it's not fine. Using a file from the official Theta website Runs at about 6fps even in release.

Basic movie file properties:
length: 11s
codec: H264 - MPEG-4 AVC (part 10) (avc 1)
Display resolution: 1920x960
Framerate: 29.97

Did you make any modifications?

Ankit09_Sangani

@HanserWork1
yes i have also problem with video Display resolution 1920*960 ..sorry for that

Did you make any modifications ?

HanserWork1

@Ankit09_Sangani
Hello. While the video with the encoding listed above works, the terribly low framerate makes it unusable. I was under the impression yours was playing at a reasonable framerate (i.e., >25fps).

I was wondering if anyone knew how they did it in holotour. I'm sure it's a method similar to how we're doing it, but I've no idea what encoding/resolution video(texture) has.

Ankit09_Sangani

@HanserWork1
Yes correct below 25fps cant load proper.
for that i change all video in 30 fps and also in (.ogv) file format below link. that direct supported Unity3d.

http://video.online-convert.com/convert-to-ogg

thanks