About a year ago I built a bouldering wall in my garage. About the same time I had purchased a quest 2. So naturally, the question was how to merge the two.
It's been a spare time project, more to teach me how to use Unity and as a way to improve my C# skills than anything else. But, the resulting desktop, AR and VR apps are quite interesting in their own way and so I figured it was worth sharing some of what I learnt along the way, in case other developers find it useful.
This first blog gives an overview of the apps, and then I plan to write individual blogs about the following:
- Creating the 3d model. Photogrammetry, mesh simplification and reskinning
- Alignment part 1: Uniform scaling.
- Alignment part 2: Tweaking the 3d model with mesh deformations
[NB: I will add links to this list when the blogs are ready]
Both the desktop and VR apps have other features, such as route creators and VR training games that I don't plan to go into here as they are not particularly interesting from a development point of view... good fun for the climbers though.
Overview of the apps:
The desktop app solved my initial problem
The problem:
At first, I just wanted a way to keep track of the routes in my shiny new home climbing wall. Typically, climbers use chalk markings on the wall, bits of tape or tags on the holds and/or use only handholds of the same colour to identify a route.
I won't go into too much detail, but these things do not work for me, or any home wall for that matter - chalk rubs off, tags everywhere get confusing and resetting routes all the time may work for a commercial bouldering centre, but is time consuming and inefficient use of holds for a home wall.
So, how do I keep track of hundreds of routes that use holds of different colors on a 'spray wall'?
The solution:
So this is where I ended up taking a 3d scan of the wall and getting it into Unity. This is what I'll cover in BLOG NUMBER 1. The pipeline was Agisoft Metashape -> Instant Mesh -> Blender -> Unity. From there it was fairly easy to use some raycasts and a UI library to build a desktop interface allowing the user to place hold markers on the 3d wall and save them. The save format here is JSON. This is not great for polymorphic route types, but this reddit post helped out with that.
This video shows an early version of the app:
The VR app solved the problem that I am reaaaaaally lazy.
The problem:
So, the desktop app was awesome for me. I could track routes on my wall and never run out of new route ideas, meaning less resetting and always having lots of variety of graded routes.
But….
Like, it was totally annoying having to, like, click the route on my laptop, squint at the screen trying to see what holds the marker were on, and then like, having to turn around and look at the actual wall, and then back the screen, back to the wall etc, until you can remember the holds well enough to actually climb the route. Really, I kept having to jump off mid-climb to walk over to my laptop and check what the next hold was.
I probably could have gone back to using tape at this point… but instead of that…
The solution:
By this time, I had already had a mess about with the Augmented Reality version on my android phone (more on that in a moment). So, it wasn't too much of a stretch to import the oculus integration library and see what's what. Compared to my smart phone the 3d tracking on the quest 2 headset was far superior, and it would mean I could put the headset on and literally see which holds where part of the route while I was climbing! No more having to use my own terrible memory...
The VR app came with siginificant challenges around aligning the model and these challenges are detailed in blogs 2&3.
Significant challenges remain such as foot tracking, which I think will be impossible until Oculus give developers access to the camera feed on the headset (IDEAS WELCOME!) and also, a quicker more user friendly way to produce the 3d model.
The VR app has a bunch of other features in progress, such as an asteroid game, environment selections, training tools, but I don't think I will cover these in this blog… I'm not a game developer, so they are buggy as hell and poorly implemented. (game dev with too much free time? Get in touch)
Honorable mention - The AR App
The Problem:
This was my first attempt to solve the problem of having to go back and forth from the computer to the wall to figure out which holds were part of a route.
The Solution:
You can see the solution at around the 1 minute mark of the video of the desktop app above. I ended using ARKit in unity. I used a photo of the entire end section of my wall as the source tracking image.. When the mobile app recognised it was looking at this part of the wall, it used it as a reference to position the 3d model. Most people use QR codes for this, but the QT code is small compared to the whole room and it just lost tracking precision quite quickly on my slightly crappy old phone. But, it kinda worked, I could walk around holding my phone in front of the wall, and it would show me where the holds were for any given route.
The Problem With The Solution: Performance, alignment, drift, you name it… it just didn’t pan out.