Using Tilt Brush To Paint 3D Sketches
By Neil Blevins
Created On: July 2nd 2016

So a good friend of mine let me test out Google's new Tilt Brush software, a 3d painting program on the HTC Vive virtual reality device. Like a 2D paint program, you paint with various brushes. However, you are painting in 3 dimensional space. What's more, you are inside the 3 dimensional space using your vr goggles, rather than seeing the 3d space through a viewport on a 2d screen. So unlike working in 3d in say 3dsmax on your computer, you can walk around your model with the ease of walking around a real life object.

This lesson isn't really a tutorial on how to use Tilt Brush, it's more a discussion of the process I went through, and some discussion of my views on the future of the technology.

Painting In Virtual Reality

Once you have the vr goggles on and pick up the two hand controllers, you are now in a empty room, except for a faint horizon (the default environment for painting, but you can choose others). Your right hand controller is the paint brush, and the left is an artist palette, much like an old school artist palette where you mix your paint. This palette though has many of the features one would expect from a 2d app like photoshop, paint brushes, color picker, eraser, etc.


I painted the space battle scene above using primarily 4 brushes...

I also used the following tools...
Viewing The Painting In Virtual Reality

Once I finished the 3d painting, I took a number of screenshots, where I walked around the battle and used a camera feature to take snapshots just like I would do if I were using a real camera and walking around a practical model. Here's the results:







This was a great way to choose exciting camera angles on my scene, far quicker and more intuitive than doing so in a standard 3d application.

I then recorded myself walking around the 3d painting, seeing it from all sorts of angles (even inside). Because the object feels very real, I noticed myself trying to avoid bumping into the object, even though I could walk right through it without any issues, since it's not real.

Then I save the painting in the Tilt Brush format. Later, if I load the painting back up, it loads by painting it step by step, which is a great way to see the painting progression.





Exporting The Painting

Another feature is the ability to export the 3d painting to an fbx file. While you are using paint brushes to "paint" the painting, what you're actually doing is constructing 3d splines that have width, like ribbons in space. So using the fbx export, you can export the geometry of the painting to a full featured 3d application like 3dsmax or maya. Here's a render of the scene in 3dsmax using a shader to see the polygonal edges. Note it exports tris and not quads.



Click this image to see it larger...



You can shade these objects just like in Tilt Brush, so it wouldn't be that difficult to replicate your Tilt Brush scene inside your 3d app of choice. Or replace the meshes with more detailed meshes, using the Tilt Brush geometry as templates, now that you've worked out the overall composition.

The Future

I see so many applications for this sort of software in the future. Certainly one thing is to have fun painting, but I also think it has many applications to the kind of work many of us computer graphics folk do every day in the entertainment or design field, especially as a collaborative tool. Here's an example.

Say I'm working on an animated film. I am a modeler, and we need a room for a scene we're going to have in our film. I invite the director of photography (the person in charge of the camera) and the director (in charge of the story) to a room. All 3 of us put on vr goggles, and go into the same virtual room. We all have the ability to paint with hand controllers. So I start, I make a 3d sketch of the room at a scale of say 1 foot by 2 foot, as though it was a dollhouse in front of me. I sketch the floor and 4 walls, and some quick furniture like chair, a table and a floor lamp. None of this needs to be super detailed, all I need is to communicate scale, proportion and position of the room and interior objects. I finish my doodle, then the director says "Don't forget, we need a doorway since our main character will be walking into the room. The director carves a doorway into my wall. I say "How high does the door need to be?" I import into the scene a representation of one of our characters from the film, and move the character beside the door. The director says "Since our character is taller than a standard human, we'll probably need the doorway to be a bit bigger than normal. He carves a larger doorway to accommodate the character. The director says "I think we should have him looking out a window once he enters", so I carve a window into the side of the room. "What will be out the window" I ask. The director says "This takes place in a city". I import a city model, and place it outside the room. "We'd better check that we see the city properly though the window?" The Director Of Photography takes the scene and scales it up to be life sized, and looks through his virtual camera attached to his hand. He changes lenses to whatever lens he think would work best in the scene. I move the character towards the window. The Director Of Photography says "That window should work, I'm seeing the buildings outside, and the character is nicely placed in the foreground. Can we move that chair out of the way though? It's sticking in the left of the shot!". I walk over to the chair and move it out of the way. "Great". Now we shrink the scene back down to dollhouse size. "I think we're done" says the director. I export the result of our painting to fbx. I then go back to my desk, import the model into maya, it's rough, but it has everything I need to now start modeling the real chairs, the real room, etc using more traditional 3d software.

Now you can certainly do these interactive collaborative sessions right now using regular 3d software, but its a lot harder, everyone is crowed around a small monitor, only one person has control of the keyboard and mouse so there's lots of hand waving and "can you just move this here, no a little more, no a little more". Having everyone in the same space and giving everyone the ability to modify the scene would allow for far better communication, which would lead to a much better workflow than what we currently have.

Wishlist

Here's a short wishlist of stuff I'd love to see added to the software, much of which would allow the workflow I outlined above
Conclusion

This is only the tip of the iceberg, I'm sure we'll see newer versions of tilt brush, and other 3d painting and sculpting apps appear on the vive and other vr devices in the near future. I'm super excited, hopefully you're excited too!


This site is ©2023 by Neil Blevins, All rights are reserved.
NeilBlevins.com Twitter Mastodon Bluesky Instagram Blogger Facebook LinkedIn ArtStation Kickstarter Gumroad YouTube IMDB