Change RealityKit Materials With SwiftUI

Daily Coding Tip 039

If you’re interested to know how to change materials like you can do in the Bernie AR app, it’s not as difficult as you might expect.

Every ModelEntity in RealityKit has a ModelComponent property. If it exists, as it is an optional and may be nil, this component contains both the mesh and the materials array. This is all we need to modify in order to change materials, although we first need to find all of the ModelEntity instances in the hierarchy.

Create a new Xcode project with the Augmented Reality template, choosing SwiftUI as the user interface.

We’re starting by extending a couple of RealityKit classes. The first is to set a material at an index to a colour. The ModelComponent property called model is optional, so we’re using optional chaining with the ? safe call operator. This means that passing an index is safe, because the chain will simply stop executing if it’s equal to nil. I’ve added an assert because while we shouldn’t ever be in a situation where the index is out of range, this will abort execution of the app (only in debug mode) to let us know.

I added a convenient function to Entity so that it will only add itself to an array if it can be cast to ModelEntity. The array has been passed as an inout property, meaning it is passed by reference, not value. This distinction is important for other reasons in this tutorial too. ModelEntity is a class and is therefore a reference type. If you pass it as a parameter anywhere, you are referencing the original object in memory, instead of making a copy of it.

While ModelEntity is a reference type, ModelComponent is a value type.

Like any structure a ModelComponent is passed as a constant value to a function. Storing an array of all of the ModelComponent instances would not give you access to that component on the original ModelEntity. The same goes for their properties, the MeshResource or the Material array, which are also copied as values when stored elsewhere.

We make use of the same inout pass-by-reference technique on the recursive function getModelEntitiesInDescendants().

Later we’ll be calling this function later like this:

let boxAnchor = try! Experience.loadBox()
var models = [ModelEntity]()
boxAnchor.getModelEntitiesInDescendants(models: &models)

Having the array as an inout parameter allows the function to call itself for all children it can find, always passing the reference to the array wherever it needs to add to it. We’re going to be calling this on the boxAnchor, which is the default anchor in the Xcode template project. This is a horizontal plane anchor that loads the steel box from the Experience.rcproject file. The reason I call it there is because it is at the top of the hierarchy, and therefore its children represent the entire scene hierarchy. If you find yourself using multiple anchors, try using a for loop like this:

var models = [ModelEntity]() 
arView.scene.anchors.forEach {
 $0.getModelEntitiesInDescendants(models: &models)
boxAnchor.getModelEntitiesInDescendants(models: &models)

But I’m getting ahead of myself. You won’t need to do this until much later. First we need to create the UI that is going to display every material of every ModelEntity in the hierarchy. This consists of a MaterialsView that will display a row for every material with a MaterialColourView. Each MaterialColourView is a ColorPicker with a name according to its index. Instances of the Material value type do not have associated names, which is why I’ve had to give them boring names instead.

We’re going to display the materials in ARSettingsView, which is a Form that creates a Section for each ModelEntity. Entities do have names although, as we’ll find out pretty soon, the default name for a ModelEntity is often generated by whatever 3D modelling software was used to create the original file. I’ve used a bottom-aligned Group here instead of giving in to my usual habit of a VStack with a Spacer at the top. This will layout its subviews from the bottom upwards, and we’re giving our Form a height of just 100, so we’ll have plenty of space above it to see our AR scene.

Finally we can bring AR into the app. I’ve dispensed with the usual ARViewContainer in favour of a generic AnyViewRepresentable structure. This would allow me to pass any UIView to it, which means I can convert any UIKit element into SwiftUI with ease. We are given the automatically generated memberwise initialiser for the structure, since the UIView (or ARView in this case) is a reference type stored as a property.

This means that we can create the ARView in ContentView, giving us access to the hierarchy without including any logic in the AnyViewRepresentable methods.

I’m repeating Apple’s use of try! when loading the box from the Experience.rcproject file. It’s not ideal as it would crash if no file was found, but it would also remove the ability of this app to do anything as it would have no model or materials. I’m settings my models array during the init of ContentView, meaning the array is constant. If you expect your hierarchy to change at runtime, you may need to make models a @State property. If you do that, remember to explicitly use the main thread, as it is considered a UI update:

DispatchQueue.main.async {
 boxAnchor.getModelEntitiesInDescendants(models: &models)

Want more Daily Coding Tips in your inbox?

This tip was heavily reliant on Max’s Getting Started with RealityKit: Materials.