Skill v1.0.4

currentAutomated scan100/100

dpearson2699/swift-ios-skills/realitykit

3 files

──Details

PublishedMay 28, 2026 at 06:39 AM

Content Hashsha256:46b167fdd7bf4a73...

Git SHA6b469de45285

Bump Typepatch

Compare with v1.0.3

──Files

Files (1 file, 16.4 KB)

SKILL.md16.4 KBactive

SKILL.md · 502 lines · 16.4 KB

version: "1.0.4" name: realitykit description: "Build iOS augmented reality and 3D experiences with RealityKit and ARKit. Use when adding RealityView content, loading entities or USDZ models, anchoring objects to planes or world positions, distinguishing entity hit tests from ARKit real-world raycasts, handling AR camera availability, world tracking, scene updates, or RealityKit entity gestures and interactions."

RealityKit

Build AR experiences on iOS using RealityKit for rendering and ARKit for world tracking. Covers RealityView, entity management, raycasting, scene understanding, and gesture-based interactions. Targets Swift 6.3 / iOS 26+.

Setup
RealityView Basics
Loading and Creating Entities
Anchoring and Placement
Raycasting
Gestures and Interaction
Scene Understanding
Common Mistakes
Review Checklist
References

Setup

Project Configuration

Add NSCameraUsageDescription to Info.plist
On iOS, RealityViewCameraContent displays an AR camera view by default (iOS 18+, macOS 15+); use .virtual camera mode for explicit non-AR fallback
No entitlement is required for basic AR. If AR is core to the app, add the arkit required-device capability; otherwise gate AR UI with isSupported.

Device Requirements

AR features require devices with an A9 chip or later. Always check ARWorldTrackingConfiguration.isSupported before presenting AR UI.

swift

import ARKit
 
guard ARWorldTrackingConfiguration.isSupported else {
    showUnsupportedDeviceMessage()
    return
}

Key Types

Type	Platform	Role
`RealityView`	iOS 18+, visionOS 1+	SwiftUI view that hosts RealityKit content
`RealityViewCameraContent`	iOS 18+, macOS 15+	Content displayed through an AR camera view on iOS, non-AR on macOS
`Entity`	All	Base class for all scene objects
`ModelEntity`	All	Entity with a visible 3D model
`AnchorEntity`	All	Tethers entities to a real-world anchor

RealityView Basics

RealityView is the SwiftUI entry point for RealityKit. RealityViewCameraContent is the iOS/macOS content type. On iOS, it uses an AR camera view by default and can use content.camera = .virtual for non-AR mode when requested or when AR/camera access is unavailable.

swift

import ARKit
import SwiftUI
import RealityKit
 
struct ARExperienceView: View {
    var body: some View {
        RealityView { (content: RealityViewCameraContent) in
            if !ARWorldTrackingConfiguration.isSupported {
                content.camera = .virtual
            }
 
            let sphere = ModelEntity(
                mesh: .generateSphere(radius: 0.05),
                materials: [SimpleMaterial(
                    color: .blue,
                    isMetallic: true
                )]
            )
            sphere.position = [0, 0, -0.5]  // 50cm in front of camera
            content.add(sphere)
        }
    }
}

Make and Update Pattern

Use the update closure to respond to SwiftUI state changes:

swift

struct PlacementView: View {
    @State private var modelColor: UIColor = .red
 
    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(
                    color: .red,
                    isMetallic: false
                )]
            )
            box.name = "colorBox"
            box.position = [0, 0, -0.5]
            content.add(box)
        } update: { content in
            if let box = content.entities.first(
                where: { $0.name == "colorBox" }
            ) as? ModelEntity {
                box.model?.materials = [SimpleMaterial(
                    color: modelColor,
                    isMetallic: false
                )]
            }
        }
 
        Button("Change Color") {
            modelColor = modelColor == .red ? .green : .red
        }
    }
}

Loading and Creating Entities

Loading from USDZ Files

Load 3D models asynchronously to avoid blocking the main thread:

swift

RealityView { content in
    if let robot = try? await ModelEntity(named: "robot") {
        robot.position = [0, -0.2, -0.8]
        robot.scale = [0.01, 0.01, 0.01]
        content.add(robot)
    }
}

Adding Components

Entities use an ECS (Entity Component System) architecture. Add components to give entities behavior:

swift

let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)
 
// Make it respond to physics
box.components.set(PhysicsBodyComponent(
    massProperties: .default,
    material: .default,
    mode: .dynamic
))
 
// Add collision shape for interaction
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))
 
// Enable input targeting for gestures
box.components.set(InputTargetComponent())

Anchoring and Placement

AnchorEntity

Use AnchorEntity to anchor content to detected surfaces or world positions:

swift

RealityView { content in
    // Anchor to a horizontal surface
    let floorAnchor = AnchorEntity(.plane(
        .horizontal,
        classification: .floor,
        minimumBounds: [0.2, 0.2]
    ))
 
    let model = ModelEntity(
        mesh: .generateBox(size: 0.1),
        materials: [SimpleMaterial(color: .orange, isMetallic: false)]
    )
    floorAnchor.addChild(model)
    content.add(floorAnchor)
}

Anchor Targets

Target	Description
`.plane(.horizontal, ...)`	Horizontal surfaces (floors, tables)
`.plane(.vertical, ...)`	Vertical surfaces (walls)
`.plane(.any, ...)`	Any detected plane
`.world(transform:)`	Fixed world-space position

Raycasting

Keep RealityKit scene queries separate from ARKit real-world raycasts:

RealityViewCameraContent.ray(through:in:to:) returns a camera ray in

RealityKit coordinate spaces. It projects a screen point into the virtual scene; it is not proof of a detected physical surface.

RealityViewCameraContent.hitTest(point:in:query:mask:) hits virtual

entities made hittable by CollisionComponent shapes. Use those shapes for entity picking and targeted gestures, not ARKit plane detection.

Use AnchorEntity(.plane(...)) for simple placement on detected planes.
Use ARKit ARRaycastQuery plus ARSession.raycast(_:) when the task needs

a one-shot intersection with real-world surfaces, then anchor with AnchorEntity(raycastResult:).

swift

let results = session.raycast(query)
if let result = results.first {
    let anchor = AnchorEntity(raycastResult: result)
    anchor.addChild(model)
    content.add(anchor)
}

Do not treat entity hit tests as substitutes for ARKit surface raycasts.

Gestures and Interaction

For gesture-based entity interaction, add CollisionComponent for the hittable shape and InputTargetComponent for input targeting. Use AccessibilityComponent for entity labels/actions. Hand detailed SwiftUI gesture composition and VoiceOver/Switch Control policy to sibling skills.

Drag Gesture on Entities

swift

struct DraggableARView: View {
    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .blue, isMetallic: true)]
            )
            box.position = [0, 0, -0.5]
            box.components.set(CollisionComponent(
                shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
            ))
            box.components.set(InputTargetComponent())
            box.name = "draggable"
            content.add(box)
        }
        .gesture(
            DragGesture()
                .targetedToAnyEntity()
                .onChanged { value in
                    let entity = value.entity
                    guard let parent = entity.parent else { return }
                    entity.position = value.convert(
                        value.location3D,
                        from: .local,
                        to: parent
                    )
                }
        )
    }
}

For selection, CollisionComponent is the mechanism that makes an entity hittable by RealityViewCameraContent.hitTest, SpatialTapGesture, or targetedToAnyEntity(). Pair it with InputTargetComponent; this enables virtual entity picking, not ARKit surface detection.

Scene Understanding

Per-Frame Updates

Subscribe to scene update events for continuous processing:

swift

RealityView { content in
    let entity = ModelEntity(
        mesh: .generateSphere(radius: 0.05),
        materials: [SimpleMaterial(color: .yellow, isMetallic: false)]
    )
    entity.position = [0, 0, -0.5]
    content.add(entity)
 
    _ = content.subscribe(to: SceneEvents.Update.self) { event in
        let time = Float(event.deltaTime)
        entity.position.y += sin(Float(Date().timeIntervalSince1970)) * time * 0.1
    }
}

Platform Boundaries

On visionOS, ARKit provides a different API surface with ARKitSession, WorldTrackingProvider, and PlaneDetectionProvider. These visionOS-specific types are not available on iOS. On iOS, RealityKit handles world tracking automatically through RealityViewCameraContent.

For iOS architecture or migration notes, explicitly name the iOS RealityKit path and handoffs:

Gate AR with ARWorldTrackingConfiguration.isSupported.
Host content with RealityViewCameraContent.
Build scenes from Entity/ModelEntity and place with AnchorEntity.
Include a compact Handoffs line in architecture/review notes:

CollisionComponent + InputTargetComponent handle RealityKit interaction; AccessibilityComponent handles RealityKit entity accessibility metadata; detailed SwiftUI gestures and VoiceOver/Switch Control policy belong to siblings.

Treat existing SCNView/SCNNode work as either a separate SceneKit path or an explicit migration to RealityKit, not a mixed scene graph.

Common Mistakes

DON'T: Skip AR capability checks

Not all devices support AR. Showing a black camera view with no feedback confuses users.

swift

// WRONG -- no device check
struct MyARView: View {
    var body: some View {
        RealityView { content in
            // Fails silently on unsupported devices
        }
    }
}
 
// CORRECT -- check support and show fallback
struct MyARView: View {
    var body: some View {
        if ARWorldTrackingConfiguration.isSupported {
            RealityView { content in
                // AR content
            }
        } else {
            ContentUnavailableView(
                "AR Not Supported",
                systemImage: "arkit",
                description: Text("This device does not support AR.")
            )
        }
    }
}

DON'T: Load heavy models synchronously

Loading large USDZ files on the main thread causes frame drops and hangs. The make closure of RealityView is async -- use it.

swift

// WRONG -- synchronous load blocks the main thread
RealityView { content in
    let model = try! Entity.load(named: "large-scene")
    content.add(model)
}
 
// CORRECT -- async load
RealityView { content in
    if let model = try? await ModelEntity(named: "large-scene") {
        content.add(model)
    }
}

DON'T: Forget collision and input target components for interactive entities

Gestures only work on entities that have both CollisionComponent and InputTargetComponent. Without them, taps and drags pass through.

swift

// WRONG -- entity ignores gestures
let box = ModelEntity(mesh: .generateBox(size: 0.1))
content.add(box)
 
// CORRECT -- add collision and input components
let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))
box.components.set(InputTargetComponent())
content.add(box)

DON'T: Create new entities in the update closure

The update closure runs on every SwiftUI state change. Creating entities there duplicates content on each render pass.

swift

// WRONG -- duplicates entities on every state change
RealityView { content in
    // empty
} update: { content in
    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
    content.add(sphere)  // Added again on every update
}
 
// CORRECT -- create in make, modify in update
RealityView { content in
    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
    sphere.name = "mySphere"
    content.add(sphere)
} update: { content in
    if let sphere = content.entities.first(
        where: { $0.name == "mySphere" }
    ) as? ModelEntity {
        // Modify existing entity
        sphere.position.y = newYPosition
    }
}

DON'T: Ignore camera permission

RealityKit on iOS needs camera access. If the user denies permission, the view shows a black screen with no explanation.

swift

// WRONG -- no permission handling
RealityView { content in
    // Black screen if camera denied
}
 
// CORRECT -- check and request permission
struct ARContainerView: View {
    @State private var cameraAuthorized = false
 
    var body: some View {
        Group {
            if cameraAuthorized {
                RealityView { content in
                    // AR content
                }
            } else {
                ContentUnavailableView(
                    "Camera Access Required",
                    systemImage: "camera.fill",
                    description: Text("Enable camera in Settings to use AR.")
                )
            }
        }
        .task {
            let status = AVCaptureDevice.authorizationStatus(for: .video)
            if status == .authorized {
                cameraAuthorized = true
            } else if status == .notDetermined {
                cameraAuthorized = await AVCaptureDevice
                    .requestAccess(for: .video)
            }
        }
    }
}

Review Checklist

[ ] NSCameraUsageDescription set in Info.plist
[ ] AR device capability checked before presenting AR views
[ ] Camera permission requested and denial handled with a fallback UI
[ ] arkit required-device capability added when AR is the app's core purpose
[ ] 3D models loaded asynchronously in the make closure
[ ] Entities created in make, modified in update (not created in update)
[ ] Interaction notes say CollisionComponent makes entities hittable/pickable and pairs with InputTargetComponent
[ ] Boundary/review notes include a Handoffs line naming AccessibilityComponent and routing detailed SwiftUI/accessibility policy to siblings
[ ] Entity hit tests, RealityViewCameraContent.ray(...), and ARKit real-world surface raycasts are not conflated
[ ] ARKit surface raycast placements use ARSession.raycast(_:) and AnchorEntity(raycastResult:)
[ ] SceneEvents.Update subscriptions used for per-frame logic (not SwiftUI timers)
[ ] Large scenes use ModelEntity(named:) async loading, not Entity.load(named:)
[ ] Anchor entities target appropriate surface types for the use case
[ ] Entity names set for lookup in the update closure

References

Read references/realitykit-patterns.md for physics, animations, lighting, ECS, accessibility, and performance patterns.
RealityKit framework
RealityView
RealityViewCameraContent
RealityViewCamera
Entity
ModelEntity
AnchorEntity
ARKit framework
ARKit in iOS
Verifying Device Support and User Permission
ARWorldTrackingConfiguration
ARRaycastQuery
ARSession.raycast(_:))
Loading entities from a file

← v1.0.3 All versions

Skill v1.0.4

RealityKit

Contents

Setup

Project Configuration

Device Requirements

Key Types

RealityView Basics

Make and Update Pattern

Loading and Creating Entities

Loading from USDZ Files

Adding Components

Anchoring and Placement

AnchorEntity

Anchor Targets

Raycasting

Gestures and Interaction

Drag Gesture on Entities

Scene Understanding

Per-Frame Updates

Platform Boundaries

Common Mistakes

DON'T: Skip AR capability checks

DON'T: Load heavy models synchronously

DON'T: Forget collision and input target components for interactive entities

DON'T: Create new entities in the update closure

DON'T: Ignore camera permission

Review Checklist

References