I’ve got the pointcloud renderer compiling locally, but on execution it looks like one of the (many) linked-to libraries was compiled with VS2013 instead of the VS2010 I’m using … I’ll wrestle with it some more next week.

Naveen’s pointed out some suitable test files, and Ross suggests modifying shaders for the “painting” effect; so: thanks guys, that’ll help speed things along.

That leaves:
– getting things to actually run (sorting out dll troubles)
– getting things to show on the oculus
– making sure all the needed sensors are available

Which should probably be done next week.

Oculus released a new thing that works with the free version of Unity. It provides a camera that can be dropped into any scene to make it Rift-friendly.

However:
– old apps (including the official Unity demo) flicker like crazy — it looks like every other frame is black (which makes things too dark — taking the Rift off is like stepping out of a dim cave)
– new apps (ones I make myself) have a weird oscillating “double vision” effect, with the distance between the two images increasing with head movement speed (this makes me a little sick if I focus on it)
– there’s what looks like a screen-tear seam in the right fourth of the right eye (this is probably minor)
– importing assets from Sketchup has yet to be successful (it looks like you have to do something weird for textures, and I haven’t done it right yet)

The second one is the biggest problem; there are rumors of fixes online (forcing DX11, or turning off timewarp), but neither seemed to work.

I’ve also tried Unreal, which claims it’s building a Rift-capable exe, but on execution it doesn’t seem to notice the Rift.

Both are pretty undocumented, as regards Rift integration.

Shelving them for now; maybe they’ll fix themselves while I do other things.

Going to look at painting pointclouds next, I think.

Here’s a some screenshots of OculusWorldDemo, showing a bit of how the post-render, pre-warp shaders interact with FOV and IPD — and the larger system that the functions I’ve been hijacking are meant to be a part of.

(Note: the demo was targeting the DK1)

The defaults:

With “zero IPD” toggled:

Max FOV at 45.4 degrees (can’t go lower than 20, which is similar):

Max FOV at 130 degrees (it goes higher, but you see no change):

These are just the things easily exposed in the demo’s menu; they don’t do exactly what we’d want to test.

“Zero IPD” is described as:

 // ForceZeroIpd does three things:
 // 1) Sets FOV to maximum symmetrical FOV based on both eyes
 // 2) Sets eye ViewAdjust values to 0.0 (effective IPD == 0)
 // 3) Uses only the Left texture for rendering.

So that’s about what we’d expect to see.

Max FOV is used similarly to the clamping function mentioned in earlier posts, and is an FovPort; it looks here that FovPorts may have more viewport to them, than FOV.

It’s still not clear exactly how best to build modified FOVs.  We need more complicated scenes; here are a few things we might use to generate them:

Unity

Unity has a nice editor, and we can expect students to be familiar with it.  I don’t think we get source code.

Unity claims they’ll have Rift support for free users soon.  That was in September.

Unity pro already has support, and costs either $75 / month (with a 12 month contract), or $1,500.  That’s per component; if we want the base and android, that’s $150 / month, or $3,000.

For educational licencing, we could contact them as suggested on the official site:

https://store.unity3d.com/education

Or purchase from the official reseller:

http://www.studica.com/unity

They offer all components in a watermarked version for $150 per year, individual components for a one-time $750, or all components for a one-time $1,999; we’d want the main component, and maybe android or ios.

These are all pre-orders for Unity 5.

Studica claims all of their discounts end on October 31st, 2014.

Unreal Engine

With this we get source; it’s unclear how it compares to Unity.  They also have a visual editor, and some weird pegs-and-wires visual programming system that I’m a little curious to see in the wild, how it shapes the way people think about programming.

Free to students via the Github Student Developer Pack.  They’ve given me access for a year; I think there’s some kind of renewal process after.

Free to schools by filling out the form at the bottom of this page.

Non-educational licences are $20 / month; with both educational and non-, they claim 5% of your profits if you launch a commercial product.

Just Load Something and Draw It

Both of those will sometimes be inflexible; even with the full source code of Unreal, even simple modifications mean a lot of learning their system.  Implicit in the act of research is doing things established engines don’t expect.

For quick tests and simple scenes, we might want a really barebones way to load, manipulate, and render models.  For that, I’m looking at Open Asset Import Library.

I haven’t yet had time to look at these in detail; a future post may have some kind of comparison.

Current plan is to go ahead without the Oculus SDK’s clamping; worst case, we can compare against default Oculus renders.

This means the next step is finding scenes to display — I’m going to take a quick look at Unity and Unreal, while integrating with our in-house code.

Also, there’s a guy (Oliver Kreylos, of UC Davis and Vrui) who made a simulator of sorts for the Rift’s optics.  Interesting for at least two reasons:

1. It might be useful to build something similar ourselves, to ease exploration and explanation.

2. He’s really gung-ho about eye tracking, but concedes (to his blog commenters) that placing the virtual camera in the center of the eyeball (rather than an unknown pupil) is an okay approximation.  It results in the point of focus being properly aligned, and the Rift’s lenses help to minimize off-focus distortion.

In the following pictures, the eyes are focused on the top corner of the diamond.  Green is the actual shape and incoming light; purple is the perceived path of light and perceived shape.

Centered at “rest” pupil (and poorly calibrated?):

Centered in the eye, but no lenses:

Centered, with lenses:

His posts and videos here:

http://doc-ok.org/?p=756

http://doc-ok.org/?p=764

First link talks about the Rift in general (20 mins); the second link talks about centering the virtual camera within the eye (5 mins).

I’ve found a path by which the Oculus SDK generates the field of view (FOV):

CalculateFovFromHmdInfo calls CalculateFovFromEyePosition, then ClampToPhysicalScreenFov.  (It also clamps eye relief to a max of 0.006 meters, which is thus far not reproduced in my code.)

All from OVR_Stereo.cpp/.h.

CalculateFovFromEyePosition calculates the FOV as four tangents from image center — up, down, left and right.  Each is simply the offset from image center (lens radius + the eye offset from lens center) divided by the eye relief (distance from eye to lens surface).  It also does that weird correction for eye rotation mentioned in an earlier post; the max of the corrected and uncorrected tangents are used.

ClampToPhysicalScreenFov estimates the physical screen FOV from a distortion (via GetPhysicalScreenFov).  It returns the min of an input FOV and the estimated physical FOV.

Last week’s images were made using Calculate, but not Clamp. Clamping makes my FOV match the default, but adds odd distortions outside of certain bounds for eye relief (ER) and offsets from image center (which I’m deriving from interpupillary distance (IPD).  I haven’t yet thought much about why, but here’s some quick observations in the direction of when (all values in meters):

Values for ER less than -0.0018 result in a flipped image (the flipping is expected for negative values, so we would expect this to happen as soon as we dip below 0; the surprise is that it waits so long).

Values of ER greater than 0.019 cause vertical stretch, fairly uniform in magnitude between top and bottom, and modest rate of increase.  It seems fairly gradual with the current stimulus.

Those both hold fairly well for all values of IPD.  However, bounds on IPD are sensitive to ER.

At negative values up to -0.0018, IPD doesn’t cause distortions (tested for values >1 and <-0.7).  It’s clearly entered some kind of weird state with negative ERs; something to keep in mind for future debugging / modeling, but we shouldn’t need negative ER directly.

At ER of 0.0001, IPD distorts outside of range 0.0125 to 0.1145.

At ER of 0.01, IPD distorts outside of range 0.034 to 0.094.  (This ER is the Oculus SDK’s default.)

At ER of 0.019, IPD distorts outisde of range 0.0535 to 0.075.

Large IPD cause the image of both eyes to stretch away from the nose, small values towards.  The distortion is drastic and increases fairly quickly with distance from the “safe” range of IPD values.

These ranges might be a little restrictive for our concerns, but should be workable; another worry is that the distortions may imply the clamping method itself is flawed.

We’ll also need to be aware of when things get clamped when designing experiments that care about specific values for IPD and ER.

I’ve attempted to recreate the FOV used by the Rift, by calling CalculateFovFromEyePosition using defaults found in OVR_Stereo.cpp (mostly in OVR_Stereo.cpp::CreateDebugHMDInfo):

 info.ProductName = "Oculus Rift DK2"; 
 info.ResolutionInPixels = Sizei ( 1920, 1080 );
 info.ScreenSizeInMeters = Sizef ( 0.12576f, 0.07074f );
 info.ScreenGapSizeInMeters = 0.0f;
 info.CenterFromTopInMeters = info.ScreenSizeInMeters.h * 0.5f;
 info.LensSeparationInMeters = 0.0635f;
 info.Shutter.Type = HmdShutter_RollingRightToLeft;
 info.Shutter.VsyncToNextVsync = ( 1.0f / 76.0f );
 info.Shutter.VsyncToFirstScanline = 0.0000273f;
 info.Shutter.FirstScanlineToLastScanline = 0.0131033f;
 info.Shutter.PixelSettleTime = 0.0f;
 info.Shutter.PixelPersistence = 0.18f * info.Shutter.VsyncToNextVsync;


 case EyeCup_DKHD2A:
 renderInfo.LensDiameterInMeters = 0.035f;
 renderInfo.LensSurfaceToMidplateInMeters = 0.02357f;
 // Not strictly lens-specific, but still wise to set a reasonable default for relief.
 renderInfo.EyeLeft.ReliefInMeters = 0.010f; 
 renderInfo.EyeRight.ReliefInMeters = 0.010f; 

 renderInfo.EyeLeft.NoseToPupilInMeters = 0.032f; 

const float OVR_DEFAULT_EXTRA_EYE_ROTATION = 30.0f * MATH_FLOAT_DEGREETORADFACTOR;

I expected to reproduce the default FOV, which yields a scene like so:

What I see instead is this:

It needs some work.

That work may be probing the values via debugger, as the default FOV is built; or, I could work through the code in that function and decide how things would go awry as displayed above.  Or arguing with include files some more, so I can probe values via code.  (Or, finding some bug in my code that’s doing something dumb.)

But soon that work will have to become bypassing the Oculus SDK as much as possible; it’d be nice to standardize to their system, but it’d be even nicer to actually get this working.

Brief entry, I’m still kind’ve beat from quals.

(Maybe I’ll flesh this entry out into a description of the Rift’s particular rendering peculiarities sometime — seems roughly three stages: virtual-to-screen, screen-to-lens, and lens-to-eye. But for now:)

Deep in the 0.4.2 Oculus Rift SDK lurk functions for setting display render properties. One of these functions has some illustrative comments.

FovPort CalculateFovFromEyePosition ( float eyeReliefInMeters,
                                      float offsetToRightInMeters,
                                      float offsetDownwardsInMeters,
                                      float lensDiameterInMeters,
                                      float extraEyeRotationInRadians /*= 0.0f*/ )

Returned is an FovPort, which describes a viewport’s field-of-view as the tangents of the angles between the viewing vector and the edges of the field of view — that is, four values for up, down, left, and right. The intent is summed up in another comment:

// 2D view of things:
//       |-|            <--- offsetToRightInMeters (in this case, it is negative)
// |=======C=======|    <--- lens surface (C=center)
//  \    |       _/
//   \   R     _/
//    \  |   _/
//     \ | _/
//      \|/
//       O  <--- center of pupil

 // (technically the lens is round rather than square, so it's not correct to
 // separate vertical and horizontal like this, but it's close enough

Which shows an asymmetric view frustum determined by the eye’s position relative to the len’s. This seems to describe the physical field-of-view through the lens onto the screen; it’s unclear what other rendering properties this might influence (render target size is implied in a comment in a calling function, and it should also be expected to influence the distortion shader), but I’ve confirmed that it affects the projection matrix.

But then it gets a bit weird:

// That's the basic looking-straight-ahead eye position relative to the lens.
// But if you look left, the pupil moves left as the eyeball rotates, which
// means you can see more to the right than this geometry suggests.
// So add in the bounds for the extra movement of the pupil.

// Beyond 30 degrees does not increase FOV because the pupil starts moving backwards more than sideways.

// The rotation of the eye is a bit more complex than a simple circle. The center of rotation
// at 13.5mm from cornea is slightly further back than the actual center of the eye.
// Additionally the rotation contains a small lateral component as the muscles pull the eye

Which is where we see extraEyeRotationInRadians put to use; this may imply an interest in eye tracking.
Also, in the function that calls CalculateFovFromEyePosition:

// Limit the eye-relief to 6 mm for FOV calculations since this just tends to spread off-screen
 // and get clamped anyways on DK1 (but in Unity it continues to spreads and causes
 // unnecessarily large render targets)

So you aren’t allowed closer than 0.006 meters. They cite rendering target size concerns; are there also physical or visual arguments?

How close is even physically plausible? According to a single paper I just looked up at random: “eyelash length rarely exceeds 10 mm”. So at 6, seems there’s a good chance we’re within uncomfortable eyelash-touching territory — that is, closer than wearers are likely to want to be.

More directly relevant to the current work: what are the visual ramifications for lens-induced artifacts, for the rendered scene, and for any interplay between them. It looks like the Rift has aspirations to precisely calibrate its render to eye position, down to the lateral movement of the eye as it rotates. What happens if we de-sync the virtual and physical (or maybe: scene-to-screen and lens-to-eye) aspects of the calibration they’re so meticulously constructing?

More on that next week.

references:

Thibaut, S., De Becker, E., Caisey, L., Baras, D., Karatas, S., Jammayrac, O., … & Bernard, B. A. (2010). Human eyelash characterization. British Journal of Dermatology, 162(2), 304-310.

For a while I was attempting to build visual stimulus in the Rift, but they always seemed off.  Sense of depth was off, it seemed difficult to fuse the two images at more than one small region at a time, and there were lots of candidates for why:

• a miscalculation in the virtual eye position, resulting in the wrong binocular disparity
• a miscalibration of the physical Rift (the lens depth is deliberately adjustable, and there are multiple choices for lens cups; our DK1 also had a bad habit of letting the screen pop out of place up to maybe an inch on the right side)
• lack of multisampling causing a lack of sub-pixel information, which may be of particular importance considering the DK1’s low resolution
• incorrect chromatic abberation correction causing visual acuity to suffer away from image and lens center (which could have been separate, competing problems in the case of miscalibration)
• something wrong in the distortion shader, causing subtle, stereo-cue destroying misalignments

Here’s two images I used to test.  In both images, I tried to center my view on on a single “corner” point of a grid pattern, marked with a red dot; image center is marked with yellow axes.

First, from the official OculusWorldDemo:

So, that’s roughly where we want our eye-center red dots to be, relative to image center.

And from our code:

Which matches what I was experiencing with the undistorted images in the Rift — things were in almost the right places, but differences between the images seem exaggerated in not immediately coherent ways.

The key difference is in the shape of the individual eye’s images.  In the image from my code, the shape of the right eye image is “flipped” relative to the official demo’s; or rather, our code hadn’t flipped the right eye’s coordinate system, relative to the left’s.

This is most apparent with the hard right edge — I’d written this off before as a mismatch in screen resolutions between PC and rift causing the screen to get cut off, but nope, turns out that lurking deep within OculusRoomTiny (the blueprint for our Rift integration), the center offset for the right eye was being inverted before being passed to the shader.  This happened well away from the rest of the rendering code, so it was easy to miss.

I changed our code to match — and the difference was striking.  Full stereo fusion came naturally, and general visual acuity and awareness of the scene was improved.  And so was awareness of a whole host of new flaws in the scene — the lack of multisampling and screen door effect were much more apparent, tracking errors more annoying, and errors in recreations of realistic scenes far more in focus.  It’s interesting how thoroughly the distortion misalignment dominated those other visual artifacts.

Related — the misaligned distortion may have been causing vertical disparity, which I hear is the main problem when using toe-in to create stereo pairs.  Vertical disparity is accused of decreasing visual acuity; perhaps this takes the form of inattention, rather than blurriness, which would explain our suddenly becoming more attentive to other artifacts when the distortion shader was fixed.  Maybe more in a future post.

Two studies, both using head mounted displays and realistic environments, seem to have conflicting findings regarding distance estimation in virtual environments.

Sahm et al (2005) find that distance estimations in the real and virtual worlds, measured via blind walking and blind throwing, exhibit consistent and significant underestimates of about 30%. They use a realistic virtual environment meant to reproduce the physical experiment environment, like so:

In two papers by Interrante et al (2006, 2008), no significant difference is found between distance estimations measured by blind walking in the real or virtual world.  They use a similar realistic environment that reproduces the physical environment:

And in the later study, they also test an enlarged and shrunken version of the virutal environment (moving the wall in or out 10%, while leaving objects like doors the same size):

They find no significant difference real and same-sized virtual world estimations; they see a significant effect in the larger virtual rooms, and a marginally significant effect in the smaller. Both resized rooms cause underestimations. This suggests that some kinds of inaccurate reproduction of (or better: deviation from?) the real world either induce distance underestimation, or prevent whatever state of sync between mental models of the virtual and physical worlds participants otherwise may have entered. Also weird that neither resized case saw distance over-estimation, only under-.

But of key interest for this post: why did one group see distance under-estimation, and the other not? This may be an interesting area of investigation — are there cues present in the one environment, but not in the other? For instance, depth cues: The hallway presents a fairly strong horizon cue, while the room may not. Or, is it a “realism” cue, some technique of lighting or texture handling or any other minute rendering detail?  Is it specific inaccuracies in the modeling of the spaces — both were hand-made (or, in the resized case, edited), so some inaccuracies are likely inevitable.  There may be some artistry in choosing where those inaccuracies are allowed to fall (and perhaps the smaller rooms in the resized case saw a marginal effect because they chose “better” — and perhaps implying a relationship between exocentric and egocentric distance estimation?)  Or is it some lower-level perceptual difference; the Sahm hallway has strong black outlines separating walls from floor, where the Interrante room has a significantly darker floor.  The specifics are unclear, but their respective studies’ results suggest that some difference in the environments may be responsible for the reuse (or more accurate application?) of real-world rules or models.

It may be worth keeping an eye out for how these sorts of environments are constructed elsewhere in the literature.

references:

1. Interrante, V., Ries, B., & Anderson, L. (2006, March). Distance perception in immersive virtual environments, revisited. In Virtual Reality Conference, 2006 (pp. 3-10). IEEE.
2. Interrante, V., Ries, B., Lindquist, J., Kaeding, M., & Anderson, L. (2008). Elucidating factors that can facilitate veridical spatial perception in immersive virtual environments. Presence: Teleoperators and Virtual Environments, 17(2), 176-198.
3. Sahm, C. S., Creem-Regehr, S. H., Thompson, W. B., & Willemsen, P. (2005). Throwing versus walking as indicators of distance perception in similar real and virtual environments. ACM Transactions on Applied Perception (TAP), 2(1), 35-45.