diff --git a/CHANGELOG.md b/CHANGELOG.md index 26cdd8b30..29a075475 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,11 @@ - **`VROARSession::stopRecording` takes a completion receiving a `VROARRecordingResult`** (`success`, `error`, `durationSec`, `frames`, `depthFrames`, `gpsFixes`, `bytes`), called once the files are complete. The no-argument form remains. On Android, `ARScene.stopRecording()` returns an `ARScene.RecordingResult` with the same fields. - **`VROARSession::isRecordingDepthSupported()` and `ARScene.isRecordingSupported(boolean depth)`.** iOS checks for LiDAR scene depth and ARCore for the Depth API. On Android `ARScene.isRecordingSupported` answers false, with or without depth, until the ARCore session exists. +- **glTF `KHR_texture_transform` and texture coordinate sets (`VROGLTFLoader`, `VROMaterialVisual`, `VROShaderFactory`, `VROMaterialShaderBinding`, `Shaders.metal`).** A texture's offset, rotation and scale apply when the material is drawn, and a texture can sample `TEXCOORD_1` or a later set (`texCoord`, which the extension may override). The base colour texture decides for the whole material, else its first other texture: Viro samples every texture of a material with one set of coordinates through one transform. `VROMaterialVisual::setContentsTransform` sets the transform, and the GL shaders gain a `material_texture_transform` uniform; on Metal it is applied in the vertex stage. +- **glTF `KHR_node_visibility` (`VROGLTFLoader`).** A node with `visible: false` loads hidden, with all its descendants, as the extension asks. +- **glTF `KHR_animation_pointer` (`VROGLTFLoader`).** An animation can drive a node's visibility, a material's `baseColorFactor` and `emissiveFactor`, and its base colour texture's offset, rotation and scale; each glTF animation's pointer channels play under its name, alongside its node and skeletal channels. A pointer at a node's translation, rotation, scale or weights is read as an ordinary channel. Other pointers are skipped with a warning naming them. `VROMaterialVisual::setColorImmediate` sets a colour without the fade or the substrate rebuild `setColor` does. +- **Emissive textures (`VROShaderFactory`, `standard_fsh.glsl`).** A material's emission texture scales its emissive colour, through its own `KHR_texture_transform`; it was bound but no shader sampled it. Not yet on visionOS, whose Metal path draws emission as a colour only. + ### Changed - **Quest: the left Menu button and the palm menu pinch report `MenuButton` and no longer go back (`VROInputControllerOpenXR`).** Meta's guidance gives the Menu button, and the palm-up pinch that stands in for it with hands, to an app's own menu, and B to going back. All three reported `BackButton` and ran the back callback, so the Menu button and the pinch ended a scene. The Menu button and the pinch now report the new `ViroOculus::MenuButton` (13) to the controller's delegates only, so it never clicks or drags a node, and the back callback no longer runs for them. The pinch is read from either hand, since the runtime sets `XR_HAND_TRACKING_AIM_MENU_PRESSED_BIT_FB` on whichever hand is not dominant; only the left hand was read before, so a left-handed wearer's pinch did nothing. B is unchanged. Pressing the Menu button now gives the left controller the short click pulse the other buttons give, which it did not get while it reported `BackButton`. An app that relied on the Menu button ending a scene has to handle `MenuButton` itself. @@ -36,6 +41,16 @@ ### Fixed +- **A GLB with an animation channel that targets no node loads (`tiny_gltf.h`).** The parser required `node` on every channel target, which glTF 2.0 does not, so a file using `KHR_animation_pointer` failed to load at all. Animations made only of such channels no longer cause the skeletal animations of later ones to be skipped. +- **A glTF without meshes or scenes loads (`VROGLTFLoader`).** The parser required scenes, nodes, accessors, buffers and buffer views, though glTF 2.0 requires only `asset`, so a file holding only a camera, or nothing yet, failed to load. +- **Emission shows on physically based materials (`VROShaderFactory`, `Shaders.metal`).** The PBR ambient modifiers rebuilt the output colour without the emissive term, so a glTF `emissiveFactor` (and an emissive colour set on any PBR material) never showed; visionOS's PBR path had no emissive term at all. +- **glTF `alphaMode` OPAQUE is opaque (`VROGLTFLoader`, `VROMaterial`, `VROShaderFactory`, `Shaders.metal`).** The loader mapped it to `RGBZero`, which nothing implements, and every material blends, so the alpha of a base colour texture or of vertex colours (which often hold other data) made opaque meshes see-through. `VROMaterial::setIgnoresDiffuseAlpha` drops that alpha; only the material's transparency and the node's opacity remain, so fades still work. MASK materials use it too: alpha decides only what is discarded, and what passes is opaque. +- **glTF BLEND materials no longer write depth (`VROGLTFLoader`).** A blended surface wrote depth like an opaque one, so a nearly transparent decal hid the mesh drawn after it behind it (whole ears gone on a character with spot decals). three.js, Babylon and Filament do the same. +- **Skins animated through nodes between their joints move (`VROGLTFLoader`).** A joint's parent was looked for only among its node's direct parent, and an animation counted as skeletal only when it targeted a joint, so a rig with an animated node above each joint (`Waist_tr` → `Waist`) had a broken bone tree and played in the bind pose. A joint's world transform is now the product of every node in its chain, animated or at rest, short of the skinned mesh's own ancestors, and an animation of any of those nodes drives the skin. +- **An animation sampler without `interpolation` loads (`tiny_gltf.h`).** It is optional in glTF 2.0 and means LINEAR; Khronos's Fox sample failed to load without it. +- **A skinned mesh is no longer culled while in view (`VRONode`).** The frustum test used its bind-pose bounds, which say nothing of where the bones draw it, so a model posed away from its bind pose lost parts as the camera came close (a character's eyes). A subtree holding a skinned geometry is never culled whole. +- **A glTF with 8-bit indices draws (`VROGLTFLoader`).** The GL substrate drew a 1-byte index buffer as 32-bit indices, reading past its end, and Metal has no 8-bit index type; the loader now widens them to 16 bits. +- **Web: skinned models draw (`VROGeometrySubstrateOpenGL`, `standard_vsh.glsl`).** The vertex shader took bone indices as `ivec4` while glTF's 8- and 16-bit joints were bound unsigned, which WebGL refuses (desktop and mobile GL let it pass). The shaders now take `uvec4`, and every bone index source is bound unsigned. - **Android: a world mesh snapshot no longer reads a mesh the render thread is replacing (`VROARWorldMesh`).** `rvSnapshotWorldMesh` serialises the current mesh on the UI thread while the render thread replaces it every update; a fused room takes long enough to serialise for the old mesh to be freed underneath. The current mesh now has its own lock and the snapshot holds its own reference. - **A reset is not undone by a collision shape still being built (`VROARWorldMesh`).** The shape is built on a background thread and installed later; one started before `resetAccumulation()` put the cleared room back as an invisible collider. A reset, a disable or a newer mesh now drops any build in flight, which also stops two overlapping builds from leaving the newer body out of the physics world. - **`voxelSize` and `maxMemoryMB` take effect after the first fused frame (`VROARWorldMesh`).** The volume was built once, at the first frame, so later values were ignored; a change now starts a new room at the new values. The fusion task no longer reads the config while the render thread writes it, and a frame from before a reset no longer lands in the new scan. diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index aab292e5c..3d725dc2a 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -37,6 +37,7 @@ - **A click goes to the node drawn on top,** and a click in open space reaches the controller's own delegates again. - **Text sits where its vertical alignment puts it,** and a back-culled draw after a front-culled one culls back faces again. - **GLB characters with one skin per mesh animate every part,** not only the first skin's. +- **glTF:** `KHR_texture_transform`, `KHR_node_visibility` and `KHR_animation_pointer` (node visibility, base colour and emissive factors, texture offset, rotation and scale), textures on `TEXCOORD_1` and later sets, and emissive textures. Opaque and masked materials ignore their textures' alpha, blended ones stop writing depth, skins animated through nodes between their joints move and are no longer culled by their bind-pose bounds, samplers without `interpolation` load, files with animation pointers no longer fail to load, emission shows on physically based materials, 8-bit index buffers draw, and skinned models draw on the web. - **Android: a `ViroARCamera` no longer crashes the scene as it mounts.** - **Video:** a paused iOS video stays paused and its late callbacks no longer reach a freed texture; Android play and pause keep their order. - **iOS:** ViroKit targets iOS 15.1, and builds for the iOS Simulator (arm64 and x86_64) as well as the device. The copy into viro now makes `ViroKit.xcframework`, replacing only the slice just built. `ios/build_ios_simulator_libs.sh` adds the simulator builds of LinearMath, protobuf-lite and GVR Audio that ViroKit links. diff --git a/ViroRenderer/Shaders.metal b/ViroRenderer/Shaders.metal index 26fcb01f2..8db7c0f25 100644 --- a/ViroRenderer/Shaders.metal +++ b/ViroRenderer/Shaders.metal @@ -144,6 +144,11 @@ struct VROLightingFragmentOut { static VROLightingFragmentOut VROMakeLightingOut(float4 color, constant VROMaterialUniforms &material) { VROLightingFragmentOut out; + // glTF alphaMode OPAQUE: only the material's transparency and the node's + // opacity remain, not the diffuse color's, texture's or vertex colors' alpha. + if (material.ignores_diffuse_alpha != 0) { + color.a = material.alpha; + } out.color = color; if (kVROHasToneMappingMask) { out.tone_mask = float4(material.tone_mapping_mask); @@ -271,7 +276,7 @@ vertex VROConstantLightingVertexOut constant_lighting_vertex(VRORendererAttribut out.position = _vertex.position; out.instance_id = v_instance_id; - out.texcoord = _geometry.texcoord; + out.texcoord = (material.texture_transform * float4(_geometry.texcoord, 0.0, 1.0)).xy; out.ambient_color = lighting.ambient_light_color; out.material_color = material.diffuse_surface_color; out.diffuse_intensity = material.diffuse_intensity; @@ -440,7 +445,7 @@ vertex VROLambertLightingVertexOut lambert_lighting_vertex(VRORendererAttributes out.position = _vertex.position; out.instance_id = v_instance_id; - out.texcoord = _geometry.texcoord; + out.texcoord = (material.texture_transform * float4(_geometry.texcoord, 0.0, 1.0)).xy; out.color = _geometry.color; out.surface_position = v_position; @@ -721,7 +726,7 @@ vertex VROPhongLightingVertexOut phong_lighting_vertex(VRORendererAttributes att out.position = _vertex.position; out.instance_id = v_instance_id; - out.texcoord = _geometry.texcoord; + out.texcoord = (material.texture_transform * float4(_geometry.texcoord, 0.0, 1.0)).xy; out.color = _geometry.color; out.surface_position = v_position; @@ -1049,7 +1054,7 @@ vertex VROBlinnLightingVertexOut blinn_lighting_vertex(VRORendererAttributes att out.position = _vertex.position; out.instance_id = v_instance_id; - out.texcoord = _geometry.texcoord; + out.texcoord = (material.texture_transform * float4(_geometry.texcoord, 0.0, 1.0)).xy; out.color = _geometry.color; out.surface_position = v_position; @@ -1480,7 +1485,7 @@ vertex VROPBRLightingVertexOut pbr_lighting_vertex(VRORendererAttributes attribu out.position = _vertex.position; out.instance_id = v_instance_id; - out.texcoord = _geometry.texcoord; + out.texcoord = (material.texture_transform * float4(_geometry.texcoord, 0.0, 1.0)).xy; out.color = _geometry.color; out.surface_position = v_position; out.camera_position = view.camera_position; @@ -1577,7 +1582,7 @@ static float4 vro_pbr_shade(VROPBRLightingVertexOut in, ambient = in.ambient_color * base_color * _surface.ao; } - return float4(ambient + radiance_out, _surface.alpha * _surface.diffuse_color.a); + return float4(ambient + radiance_out + material.emissive_color, _surface.alpha * _surface.diffuse_color.a); } fragment VROLightingFragmentOut pbr_lighting_fragment_c(VROPBRLightingVertexOut in [[ stage_in ]], diff --git a/ViroRenderer/VROGLTFLoader.cpp b/ViroRenderer/VROGLTFLoader.cpp index 95868d06c..36e4cdfd4 100644 --- a/ViroRenderer/VROGLTFLoader.cpp +++ b/ViroRenderer/VROGLTFLoader.cpp @@ -57,7 +57,11 @@ #include "VROMorpher.h" #include "VRONodeCamera.h" #include "VROLight.h" +#include "VROTransaction.h" +#include "VROAnimationFloat.h" #include +#include +#include static std::string kVROGLTFInputSamplerKey = "timeInput"; std::map> VROGLTFLoader::_dataCache; @@ -92,6 +96,595 @@ std::map VROGLTFLoader::_nodeBindPos; std::map VROGLTFLoader::_nodeBindScale; std::map VROGLTFLoader::_nodeBindRot; +#pragma mark - glTF extensions (KHR_texture_transform, KHR_node_visibility, KHR_animation_pointer) + +namespace { + +double gltfNumber(const tinygltf::Value &v, double fallback) { + if (v.IsNumber()) { + return v.Get(); + } + if (v.IsInt()) { + return (double) v.Get(); + } + return fallback; +} + +/* + A texture info's KHR_texture_transform, plus the UV set it samples (texCoord, which + the extension may override). + */ +struct GLTFTextureTransform { + float offset[2] = { 0, 0 }; + float rotation = 0; + float scale[2] = { 1, 1 }; + int texCoord = 0; + bool present = false; +}; + +GLTFTextureTransform readTextureTransform(const tinygltf::Parameter &textureInfo) { + GLTFTextureTransform t; + auto tc = textureInfo.json_double_value.find("texCoord"); + if (tc != textureInfo.json_double_value.end()) { + t.texCoord = (int) tc->second; + } + auto ext = textureInfo.extensions.find("KHR_texture_transform"); + if (ext == textureInfo.extensions.end() || !ext->second.IsObject()) { + return t; + } + const tinygltf::Value &v = ext->second; + t.present = true; + if (v.Has("offset") && v.Get("offset").ArrayLen() >= 2) { + t.offset[0] = (float) gltfNumber(v.Get("offset").Get(0), 0); + t.offset[1] = (float) gltfNumber(v.Get("offset").Get(1), 0); + } + if (v.Has("scale") && v.Get("scale").ArrayLen() >= 2) { + t.scale[0] = (float) gltfNumber(v.Get("scale").Get(0), 1); + t.scale[1] = (float) gltfNumber(v.Get("scale").Get(1), 1); + } + if (v.Has("rotation")) { + t.rotation = (float) gltfNumber(v.Get("rotation"), 0); + } + if (v.Has("texCoord")) { + t.texCoord = (int) gltfNumber(v.Get("texCoord"), t.texCoord); + } + return t; +} + +/* + The KHR_texture_transform matrix, T * R * S, as a column-major 4x4 acting on (u, v, 0, 1): + u' = cos(r) * sx * u + sin(r) * sy * v + ox + v' = -sin(r) * sx * u + cos(r) * sy * v + oy + */ +VROMatrix4f textureTransformMatrix(const GLTFTextureTransform &t) { + float c = cosf(t.rotation); + float s = sinf(t.rotation); + float m[16] = { 0 }; + m[0] = c * t.scale[0]; + m[1] = -s * t.scale[0]; + m[4] = s * t.scale[1]; + m[5] = c * t.scale[1]; + m[10] = 1; + m[12] = t.offset[0]; + m[13] = t.offset[1]; + m[15] = 1; + return VROMatrix4f(m); +} + +/* + The texture info whose transform and UV set the material uses. Viro samples every + texture of a material with one set of texture coordinates, so the base color + texture decides, else the first other texture the material has. + */ +const tinygltf::Parameter *primaryTextureInfo(const tinygltf::Material &gMat) { + auto base = gMat.pbrValues.find("baseColorTexture"); + if (base != gMat.pbrValues.end() && base->second.TextureIndex() >= 0) { + return &base->second; + } + for (const char *name : { "emissiveTexture", "normalTexture", "occlusionTexture" }) { + auto it = gMat.additionalValues.find(name); + if (it != gMat.additionalValues.end() && it->second.TextureIndex() >= 0) { + return &it->second; + } + } + auto mr = gMat.pbrValues.find("metallicRoughnessTexture"); + if (mr != gMat.pbrValues.end() && mr->second.TextureIndex() >= 0) { + return &mr->second; + } + return nullptr; +} + +GLTFTextureTransform materialTextureTransform(const tinygltf::Model &gModel, int materialIndex) { + if (materialIndex < 0 || materialIndex >= (int) gModel.materials.size()) { + return GLTFTextureTransform(); + } + const tinygltf::Parameter *info = primaryTextureInfo(gModel.materials[materialIndex]); + return info ? readTextureTransform(*info) : GLTFTextureTransform(); +} + +/* + What a KHR_animation_pointer channel drives. Pointers at a node's translation, + rotation, scale or weights are rewritten into ordinary channels when the file is + parsed (normalizeAnimationPointers), so they never reach this list. + */ +enum class GLTFPointerTarget { + NodeVisibility, + BaseColorFactor, + EmissiveFactor, + TextureOffset, + TextureRotation, + TextureScale, + Unsupported +}; + +struct GLTFPointer { + GLTFPointerTarget target = GLTFPointerTarget::Unsupported; + int index = -1; // node or material +}; + +std::vector splitPointer(const std::string &pointer) { + std::vector parts; + std::string part; + for (size_t i = 1; i <= pointer.size(); i++) { + if (i == pointer.size() || pointer[i] == '/') { + parts.push_back(part); + part.clear(); + } else { + part.push_back(pointer[i]); + } + } + return parts; +} + +bool parseIndex(const std::string &s, int *out) { + if (s.empty() || s.size() > 9) { + return false; + } + for (char c : s) { + if (c < '0' || c > '9') { + return false; + } + } + *out = atoi(s.c_str()); + return true; +} + +std::string channelPointer(const tinygltf::AnimationChannel &channel) { + auto ext = channel.target_extensions.find("KHR_animation_pointer"); + if (ext == channel.target_extensions.end() || !ext->second.IsObject() || + !ext->second.Has("pointer") || !ext->second.Get("pointer").IsString()) { + return ""; + } + return ext->second.Get("pointer").Get(); +} + +GLTFPointer parsePointer(const std::string &pointer) { + GLTFPointer p; + std::vector parts = splitPointer(pointer); + if (parts.size() < 3 || !parseIndex(parts[1], &p.index)) { + return p; + } + if (parts[0] == "nodes" && parts.size() == 5 && parts[2] == "extensions" && + parts[3] == "KHR_node_visibility" && parts[4] == "visible") { + p.target = GLTFPointerTarget::NodeVisibility; + } + else if (parts[0] == "materials") { + if (parts.size() == 3 && parts[2] == "emissiveFactor") { + p.target = GLTFPointerTarget::EmissiveFactor; + } + else if (parts.size() == 4 && parts[2] == "pbrMetallicRoughness" && parts[3] == "baseColorFactor") { + p.target = GLTFPointerTarget::BaseColorFactor; + } + else if (parts.size() == 7 && parts[2] == "pbrMetallicRoughness" && parts[3] == "baseColorTexture" && + parts[4] == "extensions" && parts[5] == "KHR_texture_transform") { + if (parts[6] == "offset") { + p.target = GLTFPointerTarget::TextureOffset; + } else if (parts[6] == "rotation") { + p.target = GLTFPointerTarget::TextureRotation; + } else if (parts[6] == "scale") { + p.target = GLTFPointerTarget::TextureScale; + } + } + } + return p; +} + +int pointerTargetWidth(GLTFPointerTarget target) { + switch (target) { + case GLTFPointerTarget::NodeVisibility: return 1; + case GLTFPointerTarget::BaseColorFactor: return 4; + case GLTFPointerTarget::EmissiveFactor: return 3; + case GLTFPointerTarget::TextureOffset: return 2; + case GLTFPointerTarget::TextureRotation: return 1; + case GLTFPointerTarget::TextureScale: return 2; + default: return 0; + } +} + +/* + Read an accessor as floats: float components as they are, normalized integers mapped + to [0, 1] (or [-1, 1]), other integers as their value (a visibility track is 0 or 1). + */ +bool readAccessorFloats(const tinygltf::Model &gModel, int accessorIndex, int width, std::vector &out) { + if (accessorIndex < 0 || accessorIndex >= (int) gModel.accessors.size()) { + return false; + } + const tinygltf::Accessor &a = gModel.accessors[accessorIndex]; + int components = 1; + switch (a.type) { + case TINYGLTF_TYPE_SCALAR: components = 1; break; + case TINYGLTF_TYPE_VEC2: components = 2; break; + case TINYGLTF_TYPE_VEC3: components = 3; break; + case TINYGLTF_TYPE_VEC4: components = 4; break; + default: return false; + } + if (width > 0 && components != width) { + return false; + } + int componentSize = 0; + switch (a.componentType) { + case TINYGLTF_COMPONENT_TYPE_BYTE: + case TINYGLTF_COMPONENT_TYPE_UNSIGNED_BYTE: componentSize = 1; break; + case TINYGLTF_COMPONENT_TYPE_SHORT: + case TINYGLTF_COMPONENT_TYPE_UNSIGNED_SHORT: componentSize = 2; break; + case TINYGLTF_COMPONENT_TYPE_UNSIGNED_INT: + case TINYGLTF_COMPONENT_TYPE_FLOAT: componentSize = 4; break; + default: return false; + } + if (a.bufferView < 0 || a.bufferView >= (int) gModel.bufferViews.size() || a.sparse.isSparse) { + // Pointer tracks are written dense; a sparse or view-less one is skipped. + return false; + } + const tinygltf::BufferView &view = gModel.bufferViews[a.bufferView]; + if (view.buffer < 0 || view.buffer >= (int) gModel.buffers.size()) { + return false; + } + const tinygltf::Buffer &buffer = gModel.buffers[view.buffer]; + size_t stride = view.byteStride ? view.byteStride : (size_t) (componentSize * components); + size_t start = view.byteOffset + a.byteOffset; + if (a.count == 0 || start + stride * (a.count - 1) + componentSize * components > buffer.data.size()) { + return false; + } + out.resize(a.count * components); + for (size_t i = 0; i < a.count; i++) { + const unsigned char *element = buffer.data.data() + start + stride * i; + for (int c = 0; c < components; c++) { + const unsigned char *src = element + c * componentSize; + float v = 0; + switch (a.componentType) { + case TINYGLTF_COMPONENT_TYPE_FLOAT: { float f; memcpy(&f, src, 4); v = f; break; } + case TINYGLTF_COMPONENT_TYPE_UNSIGNED_BYTE: { v = *src; if (a.normalized) v /= 255.0f; break; } + case TINYGLTF_COMPONENT_TYPE_BYTE: { v = (float) (int8_t) *src; if (a.normalized) v = std::max(v / 127.0f, -1.0f); break; } + case TINYGLTF_COMPONENT_TYPE_UNSIGNED_SHORT: { uint16_t u; memcpy(&u, src, 2); v = u; if (a.normalized) v /= 65535.0f; break; } + case TINYGLTF_COMPONENT_TYPE_SHORT: { int16_t u; memcpy(&u, src, 2); v = u; if (a.normalized) v = std::max(v / 32767.0f, -1.0f); break; } + case TINYGLTF_COMPONENT_TYPE_UNSIGNED_INT: { uint32_t u; memcpy(&u, src, 4); v = (float) u; break; } + } + out[i * components + c] = v; + } + } + return true; +} + +/* + One KHR_animation_pointer channel, read: its key times (seconds) and values, `width` + floats per key (three times that for CUBICSPLINE: in-tangent, value, out-tangent). + */ +struct GLTFPointerTrack { + GLTFPointer pointer; + int width = 0; + bool step = false; + bool cubic = false; + std::vector times; + std::vector values; + + void sample(float t, float *out) const { + size_t n = times.size(); + size_t keyStride = cubic ? width * 3 : width; + size_t valueOffset = cubic ? width : 0; + if (t <= times.front() || n == 1) { + memcpy(out, &values[valueOffset], width * sizeof(float)); + return; + } + if (t >= times.back()) { + memcpy(out, &values[(n - 1) * keyStride + valueOffset], width * sizeof(float)); + return; + } + size_t i = std::upper_bound(times.begin(), times.end(), t) - times.begin() - 1; + const float *a = &values[i * keyStride + valueOffset]; + const float *b = &values[(i + 1) * keyStride + valueOffset]; + if (step) { + memcpy(out, a, width * sizeof(float)); + return; + } + float span = times[i + 1] - times[i]; + float alpha = span > 1e-8f ? (t - times[i]) / span : 0; + for (int c = 0; c < width; c++) { + out[c] = a[c] + (b[c] - a[c]) * alpha; + } + } +}; + +/* + The state the KHR_animation_pointer tracks of one glTF animation drive, shared by + every copy of that animation: the nodes and materials they reach, and each animated + material's texture transform (its parts are animated one by one). + */ +struct GLTFPointerTargets { + std::map> nodes; + std::map>> materials; + std::map transforms; +}; + +/* + Plays the KHR_animation_pointer channels of one glTF animation: node visibility, a + material's base color and emissive factors, and its base color texture's transform. + One float animation runs over the clip, and each frame every track is sampled at + that time and written to its target. + */ +class VROGLTFPointerAnimation : public VROExecutableAnimation, + public std::enable_shared_from_this { +public: + VROGLTFPointerAnimation(std::shared_ptr> tracks, + std::shared_ptr targets, float duration) : + _tracks(tracks), _targets(targets), _duration(duration) {} + virtual ~VROGLTFPointerAnimation() {} + + void setName(std::string name) { _name = name; } + std::string getName() const { return _name; } + + std::shared_ptr copy() { + std::shared_ptr animation = + std::make_shared(_tracks, _targets, _duration); + animation->setName(_name); + animation->setSpeed(_speed); + animation->setTimeOffset(_timeOffset); + return animation; + } + + void execute(std::shared_ptr node, std::function onFinished) { + std::weak_ptr weakSelf = shared_from_this(); + apply(0); + + VROTransaction::begin(); + VROTransaction::setAnimationDuration(_duration); + VROTransaction::setAnimationTimeOffset(_timeOffset); + VROTransaction::setAnimationSpeed(_speed); + VROTransaction::setTimingFunction(VROTimingFunctionType::Linear); + node->animate(std::make_shared([weakSelf](VROAnimatable *const animatable, float t) { + std::shared_ptr self = weakSelf.lock(); + if (self) { + self->apply(t); + } + }, 0.0f, _duration)); + VROTransaction::setFinishCallback([weakSelf, onFinished](bool terminate) { + std::shared_ptr self = weakSelf.lock(); + if (self) { + self->_transaction.reset(); + } + onFinished(); + }); + std::shared_ptr transaction = VROTransaction::commit(); + transaction->holdExecutableAnimation(shared_from_this()); + _transaction = transaction; + } + + void pause() { + std::shared_ptr transaction = _transaction.lock(); + if (transaction) { + VROTransaction::pause(transaction); + } + } + void resume() { + std::shared_ptr transaction = _transaction.lock(); + if (transaction) { + VROTransaction::resume(transaction); + } + } + void terminate(bool jumpToEnd) { + std::shared_ptr transaction = _transaction.lock(); + if (transaction) { + VROTransaction::terminate(transaction, jumpToEnd); + _transaction.reset(); + } + } + void setSpeed(float speed) { + _speed = speed; + std::shared_ptr transaction = _transaction.lock(); + if (transaction) { + VROTransaction::setAnimationSpeed(transaction, _speed); + } + } + void preload() {} + void setDuration(float durationSeconds) { _duration = durationSeconds; } + float getDuration() const { return _duration; } + std::string toString() const { return "[gltf pointer: " + _name + "]"; } + +private: + std::shared_ptr> _tracks; + std::shared_ptr _targets; + float _duration; + std::string _name; + std::weak_ptr _transaction; + + void apply(float t) { + std::set transformsChanged; + float v[4]; + for (const GLTFPointerTrack &track : *_tracks) { + track.sample(t, v); + int index = track.pointer.index; + switch (track.pointer.target) { + case GLTFPointerTarget::NodeVisibility: { + auto it = _targets->nodes.find(index); + std::shared_ptr node = it == _targets->nodes.end() ? nullptr : it->second.lock(); + bool hidden = v[0] < 0.5f; + if (node && node->isHidden() != hidden) { + // A transaction of zero duration: the change shows at once, not faded. + VROTransaction::begin(); + VROTransaction::setAnimationDuration(0); + node->setHidden(hidden); + VROTransaction::commit(); + } + break; + } + case GLTFPointerTarget::BaseColorFactor: + case GLTFPointerTarget::EmissiveFactor: { + bool base = track.pointer.target == GLTFPointerTarget::BaseColorFactor; + VROVector4f color(v[0], v[1], v[2], base ? v[3] : 1.0f); + for (const std::weak_ptr &m_w : _targets->materials[index]) { + std::shared_ptr m = m_w.lock(); + if (m) { + (base ? m->getDiffuse() : m->getEmission()).setColorImmediate(color); + } + } + break; + } + case GLTFPointerTarget::TextureOffset: { + GLTFTextureTransform &tt = _targets->transforms[index]; + tt.offset[0] = v[0]; tt.offset[1] = v[1]; + transformsChanged.insert(index); + break; + } + case GLTFPointerTarget::TextureScale: { + GLTFTextureTransform &tt = _targets->transforms[index]; + tt.scale[0] = v[0]; tt.scale[1] = v[1]; + transformsChanged.insert(index); + break; + } + case GLTFPointerTarget::TextureRotation: { + _targets->transforms[index].rotation = v[0]; + transformsChanged.insert(index); + break; + } + default: + break; + } + } + for (int index : transformsChanged) { + VROMatrix4f matrix = textureTransformMatrix(_targets->transforms[index]); + for (const std::weak_ptr &m_w : _targets->materials[index]) { + std::shared_ptr m = m_w.lock(); + if (m) { + m->getDiffuse().setContentsTransform(matrix); + } + } + } + } +}; + +// Per load (cleared by clearCachedData): KHR_animation_pointer channels by +// animation, the materials whose texture transform they animate, and every node and +// material instance built, so the channels can reach them. +std::map> sPointerTracks; +std::set sAnimatedTransformMaterials; +std::map> sNodesByIndex; +std::map>> sMaterialsByIndex; + +} // namespace + +void VROGLTFLoader::normalizeAnimationPointers(tinygltf::Model &gModel) { + // A KHR_animation_pointer channel at a node's transform or morph weights is an + // ordinary channel written another way: rewrite it into one, so it goes through + // the keyframe and skeletal paths like the rest. + for (tinygltf::Animation &anim : gModel.animations) { + for (tinygltf::AnimationChannel &channel : anim.channels) { + if (channel.target_path != "pointer") { + continue; + } + std::vector parts = splitPointer(channelPointer(channel)); + int nodeIndex = -1; + if (parts.size() == 3 && parts[0] == "nodes" && parseIndex(parts[1], &nodeIndex) && + nodeIndex < (int) gModel.nodes.size() && + (parts[2] == "translation" || parts[2] == "rotation" || + parts[2] == "scale" || parts[2] == "weights")) { + channel.target_node = nodeIndex; + channel.target_path = parts[2]; + } + } + } +} + +bool VROGLTFLoader::isPointerChannel(const tinygltf::AnimationChannel &channel) { + return channel.target_path == "pointer" || channel.target_node < 0; +} + +void VROGLTFLoader::processPointerChannels(const tinygltf::Model &gModel) { + std::set unsupported; + for (int animIndex = 0; animIndex < (int) gModel.animations.size(); animIndex++) { + const tinygltf::Animation &anim = gModel.animations[animIndex]; + for (const tinygltf::AnimationChannel &channel : anim.channels) { + if (!isPointerChannel(channel)) { + continue; + } + std::string pointer = channelPointer(channel); + GLTFPointerTrack track; + track.pointer = parsePointer(pointer); + track.width = pointerTargetWidth(track.pointer.target); + if (track.width == 0 || channel.sampler < 0 || channel.sampler >= (int) anim.samplers.size()) { + unsupported.insert(pointer.empty() ? "(no pointer)" : pointer); + continue; + } + bool isNode = track.pointer.target == GLTFPointerTarget::NodeVisibility; + if (track.pointer.index >= (int) (isNode ? gModel.nodes.size() : gModel.materials.size())) { + continue; + } + const tinygltf::AnimationSampler &sampler = anim.samplers[channel.sampler]; + track.step = VROStringUtil::strcmpinsensitive(sampler.interpolation, "step") || + isNode; // visibility is a boolean: it never blends + track.cubic = VROStringUtil::strcmpinsensitive(sampler.interpolation, "cubicspline"); + if (!readAccessorFloats(gModel, sampler.input, 1, track.times) || + !readAccessorFloats(gModel, sampler.output, track.width, track.values) || + track.times.empty() || + track.values.size() != track.times.size() * track.width * (track.cubic ? 3 : 1)) { + pwarn("glTF: skipping animation pointer %s, its data does not match", pointer.c_str()); + continue; + } + if (track.pointer.target == GLTFPointerTarget::TextureOffset || + track.pointer.target == GLTFPointerTarget::TextureRotation || + track.pointer.target == GLTFPointerTarget::TextureScale) { + sAnimatedTransformMaterials.insert(track.pointer.index); + } + sPointerTracks[animIndex].push_back(std::move(track)); + } + } + for (const std::string &pointer : unsupported) { + pwarn("glTF: animation pointer %s is not supported, its channels are skipped", pointer.c_str()); + } +} + +void VROGLTFLoader::attachPointerAnimations(const tinygltf::Model &gModel, std::shared_ptr host) { + if (sPointerTracks.empty() || !host) { + return; + } + std::shared_ptr targets = std::make_shared(); + for (auto &kv : sNodesByIndex) { + targets->nodes[kv.first] = kv.second; + } + for (auto &kv : sMaterialsByIndex) { + for (auto &m : kv.second) { + targets->materials[kv.first].push_back(m); + } + } + for (int materialIndex : sAnimatedTransformMaterials) { + targets->transforms[materialIndex] = materialTextureTransform(gModel, materialIndex); + } + + for (auto &kv : sPointerTracks) { + int animIndex = kv.first; + float duration = 0; + for (const GLTFPointerTrack &track : kv.second) { + duration = std::max(duration, track.times.back()); + } + std::string name = gModel.animations[animIndex].name; + if (name.empty()) { + name = "animation_" + VROStringUtil::toString(animIndex); + } + std::shared_ptr animation = std::make_shared( + std::make_shared>(std::move(kv.second)), targets, duration); + animation->setName(name); + host->addAnimation(name, animation); + } +} + static int getTypeSize(GLTFType type) { switch (type) { case GLTFType::Scalar: return 1; @@ -451,11 +1044,16 @@ void VROGLTFLoader::loadGLTFFromResource(std::string gltfManifestFilePath, const std::string err; // If we've successfully retrieved the GLTF Manifest, start parsing the file with tinyGLTF. + // glTF 2.0 requires only `asset`: a file with a camera and no mesh has no + // accessors or buffers, and one may hold no scene at all. tinygltf + // requires every section unless told otherwise. bool ret = false; if (isGLTFBinary) { - ret = gLoader.LoadBinaryFromFile(&gModel, &err, cachedFilePath, overwriteResourceMap); + ret = gLoader.LoadBinaryFromFile(&gModel, &err, cachedFilePath, overwriteResourceMap, + tinygltf::NO_REQUIRE); } else { - ret = gLoader.LoadASCIIFromFile(&gModel, &err, cachedFilePath, gltfManifestFilePath, overwriteResourceMap); + ret = gLoader.LoadASCIIFromFile(&gModel, &err, cachedFilePath, gltfManifestFilePath, overwriteResourceMap, + tinygltf::NO_REQUIRE); } // Fail fast if any errors were encountered. @@ -485,6 +1083,7 @@ void VROGLTFLoader::loadGLTFFromResource(std::string gltfManifestFilePath, const // being const it was copied again into the std::function. On WASM, where // the dispatch is synchronous, those copies sat beside the original and // tripled a large GLB's footprint at exactly the moment it peaked. + normalizeAnimationPointers(gModel); std::shared_ptr modelPtr = std::make_shared(std::move(gModel)); VROPlatformDispatchAsyncRenderer([rootNode, modelPtr, driver, onFinish] { @@ -533,6 +1132,13 @@ void VROGLTFLoader::loadGLTFFromResource(std::string gltfManifestFilePath, const } } + // KHR_animation_pointer channels drive nodes and materials across the + // whole model: their animations go on the (first) scene node, where + // a recursive animation lookup from the model's root finds them. + if (success && !gltfRootNode->getChildNodes().empty()) { + attachPointerAnimations(model, gltfRootNode->getChildNodes().front()); + } + for (auto &skeletonPair : _skinIndexToSkeleton) { std::shared_ptr skin = _skinMap[skeletonPair.first]; skeletonPair.second->setSkinnerRootNode(skin->getSkinnerNode()); @@ -577,6 +1183,10 @@ void VROGLTFLoader::clearCachedData() { _nodeBindPos.clear(); _nodeBindScale.clear(); _nodeBindRot.clear(); + sPointerTracks.clear(); + sAnimatedTransformMaterials.clear(); + sNodesByIndex.clear(); + sMaterialsByIndex.clear(); } void VROGLTFLoader::injectBindPoseAnimations() { @@ -697,28 +1307,27 @@ bool VROGLTFLoader::processSkinner(const tinygltf::Model &model) { // Grab the node index corresponding to this joint. int nodeIndex = _skinIndexToJointNodeIndex[skinIndex][jointIndex]; - // Then look up the node with the nodeIndex and grab it's child nodes. - std::vector childrenNodeIndex = model.nodes[nodeIndex].children; - for (int childIndex : childrenNodeIndex) { - // Skip if we have child indexes that are not a joint. - if (skinIndexToNodeJointIndexes[skinIndex].find(childIndex) == skinIndexToNodeJointIndexes[skinIndex].end()) { - continue; + // A joint's parent joint is its nearest ancestor that is a joint of this skin. + // It need not be the joint's direct parent: glTF allows other nodes between + // two joints (a rig may animate an intermediate node per joint). + int ancestor = nodeIndex; + while (_nodeParentMap.count(ancestor)) { + ancestor = _nodeParentMap[ancestor]; + auto parentJoint = skinIndexToNodeJointIndexes[skinIndex].find(ancestor); + if (parentJoint != skinIndexToNodeJointIndexes[skinIndex].end()) { + _skinIndexToJointChildJoints[skinIndex][parentJoint->second].push_back(jointIndex); + skinIndexToJointParentJoint[skinIndex][jointIndex] = parentJoint->second; + break; } - - // For each of the child node index, get it's corresponding joint index. - int subJointIndex = skinIndexToNodeJointIndexes[skinIndex][childIndex]; - - // Parent the all childed joint with it's parent. Also create a reverse map - // in skinIndexToJointParentJoint. - _skinIndexToJointChildJoints[skinIndex][jointIndex].push_back(subJointIndex); - skinIndexToJointParentJoint[skinIndex][subJointIndex] = jointIndex; } } - // Save a reference to the root + // Save a reference to the root: the skeleton node when it is a joint, else the + // first joint (skin.skeleton may name a common ancestor that is not a joint). int nodeIndexOfSkeleton = skin.skeleton; - if (nodeIndexOfSkeleton >= 0) { - _skinIndexToSkeletonRootJoint[skinIndex] = skinIndexToNodeJointIndexes[skinIndex][nodeIndexOfSkeleton]; + auto skeletonJoint = skinIndexToNodeJointIndexes[skinIndex].find(nodeIndexOfSkeleton); + if (nodeIndexOfSkeleton >= 0 && skeletonJoint != skinIndexToNodeJointIndexes[skinIndex].end()) { + _skinIndexToSkeletonRootJoint[skinIndex] = skeletonJoint->second; } else { // Else grab the first item in the joint list and treat it as the root node. _skinIndexToSkeletonRootJoint[skinIndex] = 0; @@ -741,7 +1350,8 @@ bool VROGLTFLoader::processSkinner(const tinygltf::Model &model) { // Create a vec hiearachy of bones, starting at the root bone. for (int jointIndex = 0; jointIndex < skin.joints.size(); jointIndex++) { - int parentJointIndex = skinIndexToJointParentJoint[skinIndex][jointIndex]; + auto parentIt = skinIndexToJointParentJoint[skinIndex].find(jointIndex); + int parentJointIndex = parentIt != skinIndexToJointParentJoint[skinIndex].end() ? parentIt->second : -1; if (jointIndex == rootJoint) { parentJointIndex = -1; } @@ -795,14 +1405,21 @@ bool VROGLTFLoader::processAnimations(const tinygltf::Model &model) { > nodeToAnimDataMap */ + // KHR_animation_pointer channels (materials, node visibility) are read apart: they + // target no node, and become their own animations once the model is built. + processPointerChannels(model); + std::map>> nodeToAnimDataMap; std::map> animToNodeIndexMap; for (int animIndex = 0; animIndex < model.animations.size(); animIndex++) { - tinygltf::Animation anim = model.animations[animIndex]; + const tinygltf::Animation &anim = model.animations[animIndex]; // For each animation, grab the targeted node and its animation channel data. for (int cIndex = 0; cIndex < anim.channels.size(); cIndex++) { - tinygltf::AnimationChannel channel = anim.channels[cIndex]; + const tinygltf::AnimationChannel &channel = anim.channels[cIndex]; + if (isPointerChannel(channel)) { + continue; + } nodeToAnimDataMap[channel.target_node][animIndex].push_back(cIndex); // Also save a reverse map of anim to nodeIndexes for skeletal animation processing. @@ -812,11 +1429,22 @@ bool VROGLTFLoader::processAnimations(const tinygltf::Model &model) { // Group all the node indexes associated with a given skin to be used for // skeletal animation processing. + // A skin's pose depends on its joints and on every ancestor of a joint short of the + // skinned mesh's own ancestors (the renderer applies those): an animation on any of + // them moves the skin. Rigs that animate a node above each joint rely on this. std::map> skinToNodeMap; for (int skinIndex = 0; skinIndex < model.skins.size(); skinIndex ++) { - tinygltf::Skin skin = model.skins[skinIndex]; + const tinygltf::Skin &skin = model.skins[skinIndex]; for (int jointIndex = 0; jointIndex < skin.joints.size(); jointIndex++) { - skinToNodeMap[skinIndex].insert(skin.joints[jointIndex]); + int node = skin.joints[jointIndex]; + skinToNodeMap[skinIndex].insert(node); + while (_nodeParentMap.count(node)) { + node = _nodeParentMap[node]; + if (_skinMeshAncestors.count(skinIndex) && _skinMeshAncestors[skinIndex].count(node)) { + break; + } + skinToNodeMap[skinIndex].insert(node); + } } } @@ -825,11 +1453,14 @@ bool VROGLTFLoader::processAnimations(const tinygltf::Model &model) { // A single glTF animation may drive multiple skins (e.g. a character body + visor), // so we accumulate ALL matching skins rather than keeping only the last one. std::map>>> skeletalAnimToSkinToNodeMap; - for (int animIndex = 0; animIndex < (int)animToNodeIndexMap.size(); animIndex++) { + // By entry, not by index: an animation made only of KHR_animation_pointer channels + // has no entry, and indexing 0..size() would then skip the last animations. + for (auto &animNodes : animToNodeIndexMap) { + int animIndex = animNodes.first; for (int skinIndex = 0; skinIndex < (int)skinToNodeMap.size(); skinIndex++) { std::vector animatedSkeletalJoint; for (auto i : skinToNodeMap[skinIndex]) { - for (auto x : animToNodeIndexMap[animIndex]) { + for (auto x : animNodes.second) { if (i == x) { animatedSkeletalJoint.push_back(i); } @@ -1586,28 +2217,6 @@ static void getNodeRestTRS(const tinygltf::Model &gModel, int nodeIndex, } } -// Helper: build a local TRS matrix for a joint, falling back to the node's rest-pose for -// any channels not present in the animation. Mixamo/Blender rigs typically only animate -// rotation; without the rest-pose translation every non-root bone would collapse to the -// origin, producing severe mesh corruption. -static VROMatrix4f buildJointLocalTransform(const tinygltf::Model &gModel, int nodeIndex, - const VROKeyframeAnimation *anim, - const VROKeyframeAnimationFrame *frame) { - VROVector3f restT; VROVector3f restS; VROQuaternion restR; - getNodeRestTRS(gModel, nodeIndex, restT, restS, restR); - - VROVector3f scale = anim->_hasScale ? frame->scale : restS; - VROQuaternion rotation = anim->_hasRotation ? frame->rotation : restR; - VROVector3f translation = anim->_hasTranslation ? frame->translation : restT; - - VROMatrix4f m; - m.toIdentity(); - m.scale(scale.x, scale.y, scale.z); - m = rotation.getMatrix() * m; - m.translate(translation); - return m; -} - bool VROGLTFLoader::processSkeletalTransformsForFrame(const tinygltf::Model &gModel, int skin, int animationIndex, @@ -1615,107 +2224,60 @@ bool VROGLTFLoader::processSkeletalTransformsForFrame(const tinygltf::Model &gMo int keyFrameTime, int currentJointIndex, std::map &transformsOut) { - // If we are at the root (transform not yet placed by a parent), compute and store it. - // Detect "root" by the absence of a pre-placed transform rather than hardcoding index 0, - // so models whose skeleton root joint is not at index 0 are handled correctly. - if (transformsOut.find(currentJointIndex) == transformsOut.end()) { - int rootNodeIndex = _skinIndexToJointNodeIndex[skin][currentJointIndex]; - - // Walk up the GLTF scene graph collecting non-skin ancestor nodes. - // For multi-skin models the root joint of a secondary skin (e.g. visor's Neck joint) - // may have parent nodes that belong to the primary skin (Hips, Spine…). Those ancestors - // are NOT in this skin's joint list but their animated world transforms must be included - // so the IBM (which encodes the full world-space bind pose) is correctly cancelled out. - std::vector nonSkinAncestors; - { - int cur = rootNodeIndex; - while (_nodeParentMap.count(cur)) { - int par = _nodeParentMap[cur]; - // Check whether the parent node is a joint in THIS skin - bool inThisSkin = false; - for (auto &kv : _skinIndexToJointNodeIndex[skin]) { - if (kv.second == par) { inThisSkin = true; break; } - } - if (inThisSkin) break; // ancestor is already handled by the skin cascade - // Stop if this ancestor is already applied by Viro's renderer as the mesh - // node's modelMatrix (i.e. it's an ancestor of the skin's mesh node). - // Including it here would double-apply it (e.g. Character scale=0.01 twice). - if (_skinMeshAncestors.count(skin) && _skinMeshAncestors[skin].count(par)) break; - nonSkinAncestors.push_back(par); - cur = par; - } - } - - // Build the accumulated world transform from outermost to innermost ancestor. - // For each ancestor, collect its TRS from all animation channels (unflattened T/R/S) - // falling back to the rest-pose for any channel not present in the animation. - VROMatrix4f ancestorWorld; - ancestorWorld.toIdentity(); - for (int i = (int)nonSkinAncestors.size() - 1; i >= 0; i--) { - int ancNode = nonSkinAncestors[i]; - VROVector3f ancT; VROVector3f ancS; VROQuaternion ancR; - getNodeRestTRS(gModel, ancNode, ancT, ancS, ancR); - const auto &ancAnimMap = _nodeKeyFrameAnims[ancNode]; - auto ancAnimIt = ancAnimMap.find(animationIndex); - if (ancAnimIt != ancAnimMap.end()) { - for (const auto &ancAnim : ancAnimIt->second) { - if (keyFrameTime >= (int)ancAnim->getFrames().size()) continue; - const auto &ancFrame = *ancAnim->getFrames()[keyFrameTime]; - if (ancAnim->_hasTranslation) ancT = ancFrame.translation; - if (ancAnim->_hasRotation) ancR = ancFrame.rotation; - if (ancAnim->_hasScale) ancS = ancFrame.scale; + // Each joint's world transform is the product of the local transforms of every node + // from the top of its chain down to the joint, as glTF defines it: animated at this + // frame where the animation drives the node, at rest otherwise. Nodes between two + // joints count, and so do a joint's ancestors outside the skin, but the skinned mesh's + // own ancestors do not: the renderer applies them as the mesh's model matrix. + // (subAnimPropertyIndex and currentJointIndex are kept for the signature: every node + // the animation drives was merged onto one grid, so frame keyFrameTime is the same + // instant for all of them.) + (void) subAnimPropertyIndex; + (void) currentJointIndex; + + std::map worldOfNode; + std::function world = [&](int node) -> VROMatrix4f { + auto cached = worldOfNode.find(node); + if (cached != worldOfNode.end()) { + return cached->second; + } + + VROVector3f t; VROVector3f sc; VROQuaternion r; + getNodeRestTRS(gModel, node, t, sc, r); + auto nodeAnims = _nodeKeyFrameAnims.find(node); + if (nodeAnims != _nodeKeyFrameAnims.end()) { + auto anims = nodeAnims->second.find(animationIndex); + if (anims != nodeAnims->second.end()) { + for (const std::shared_ptr &anim : anims->second) { + if (keyFrameTime >= (int) anim->getFrames().size()) { + continue; + } + const VROKeyframeAnimationFrame &frame = *anim->getFrames()[keyFrameTime]; + if (anim->_hasTranslation) t = frame.translation; + if (anim->_hasRotation) r = frame.rotation; + if (anim->_hasScale) sc = frame.scale; } } - VROMatrix4f ancLocal; - ancLocal.toIdentity(); - ancLocal.scale(ancS.x, ancS.y, ancS.z); - ancLocal = ancR.getMatrix() * ancLocal; - ancLocal.translate(ancT); - ancestorWorld = ancestorWorld.multiply(ancLocal); - } - - // Compute root joint's local transform, then pre-multiply by ancestor world. - const auto &anims = _nodeKeyFrameAnims[rootNodeIndex][animationIndex]; - VROMatrix4f rootLocal; - if (anims.empty()) { - rootLocal = getTransformOfNode(gModel, rootNodeIndex); - } else { - const auto &anim = anims.at(subAnimPropertyIndex); - const auto &frame = *anim->getFrames()[keyFrameTime]; - rootLocal = buildJointLocalTransform(gModel, rootNodeIndex, anim.get(), &frame); } - transformsOut[currentJointIndex] = ancestorWorld.multiply(rootLocal); - } - - // Grab the transform of the current joint to be cascaded and multiplied on the child. - VROMatrix4f currentMatrix = transformsOut[currentJointIndex]; + VROMatrix4f local; + local.toIdentity(); + local.scale(sc.x, sc.y, sc.z); + local = r.getMatrix() * local; + local.translate(t); - // Grab all the child joints for this current joint. - std::vector childJoints = _skinIndexToJointChildJoints[skin][currentJointIndex]; - for (int childJointIndex : childJoints) { - int childNodeIndex = _skinIndexToJointNodeIndex[skin][childJointIndex]; - const auto &anims = _nodeKeyFrameAnims[childNodeIndex][animationIndex]; - - VROMatrix4f localTransform; - if (anims.empty()) { - localTransform = getTransformOfNode(gModel, childNodeIndex); - } else { - const auto &anim = anims.at(subAnimPropertyIndex); - const auto &frame = *anim->getFrames()[keyFrameTime]; - localTransform = buildJointLocalTransform(gModel, childNodeIndex, anim.get(), &frame); + VROMatrix4f result = local; + auto parent = _nodeParentMap.find(node); + if (parent != _nodeParentMap.end() && + !(_skinMeshAncestors.count(skin) && _skinMeshAncestors[skin].count(parent->second))) { + result = world(parent->second).multiply(local); } + worldOfNode[node] = result; + return result; + }; - // Cascade: world transform = parent_world * local - transformsOut[childJointIndex] = currentMatrix.multiply(localTransform); - - // Continue going down the skeletal tree - if (!processSkeletalTransformsForFrame(gModel, skin, animationIndex, subAnimPropertyIndex, - keyFrameTime, childJointIndex, transformsOut)) { - return false; - } + for (auto &joint : _skinIndexToJointNodeIndex[skin]) { + transformsOut[joint.first] = world(joint.second); } - - // We have reached the leaf of the skeletal tree. return true; } @@ -1808,6 +2370,15 @@ bool VROGLTFLoader::processNode(const tinygltf::Model &gModel, std::shared_ptrsetScale(scale); node->setRotation(rot); node->setName(gNode.name); + sNodesByIndex[gNodeIndex] = node; + + // KHR_node_visibility: a node with visible false is hidden with all its descendants. + auto visibility = gNode.extensions.find("KHR_node_visibility"); + if (visibility != gNode.extensions.end() && visibility->second.IsObject() && + visibility->second.Has("visible") && visibility->second.Get("visible").IsBool() && + !visibility->second.Get("visible").Get()) { + node->setHidden(true); + } // Record the VRONode and its bind-pose TRS for nodes that have keyframe animations. // injectBindPoseAnimations() uses this after all nodes are processed to add single-frame @@ -2127,10 +2698,14 @@ bool VROGLTFLoader::processMesh(const tinygltf::Model &gModel, std::shared_ptr &gPrimitives = gMesh.primitives; for (tinygltf::Primitive gPrimitive : gPrimitives) { + // The material's texture transform and the UV set its textures sample + // (KHR_texture_transform, texCoord). + GLTFTextureTransform textureTransform = materialTextureTransform(gModel, gPrimitive.material); // Grab vertex indexing information needed for creating meshes. bool successVertex = processVertexElement(gModel, gPrimitive, elements); - bool successAttributes = processVertexAttributes(gModel, gPrimitive.attributes, sources, elements.size() - 1, driver); + bool successAttributes = processVertexAttributes(gModel, gPrimitive.attributes, sources, elements.size() - 1, driver, + textureTransform.texCoord); processTangent(elements, sources, elements.size() - 1); if (!successVertex || !successAttributes) { @@ -2144,6 +2719,10 @@ bool VROGLTFLoader::processMesh(const tinygltf::Model &gModel, std::shared_ptr= 0) { const tinygltf::Material &gMat = gModel.materials[gMatIndex]; material = getMaterial(gModel, gMat); + if (textureTransform.present || sAnimatedTransformMaterials.count(gMatIndex)) { + material->getDiffuse().setContentsTransform(textureTransformMatrix(textureTransform)); + } + sMaterialsByIndex[gMatIndex].push_back(material); } // Process Morph targets for each primitive. @@ -2499,6 +3078,22 @@ bool VROGLTFLoader::processVertexElement(const tinygltf::Model &gModel, const unsigned char *srcData = gBuffer.data.data() + dataOffset; int bytesPerIndex = getComponentTypeSize(gTypeComponent); + // 8-bit indices (legal glTF) are widened to 16 bits: the GL substrate draws a + // 1-byte index buffer as GL_UNSIGNED_INT, reading past its end, and Metal has + // no 8-bit index type at all. + std::vector widenedIndices; + if (gTypeComponent == GLTFTypeComponent::UnsignedByte) { + widenedIndices.resize(elementCount); + for (size_t i = 0; i < elementCount; i++) { + widenedIndices[i] = srcData[i * bufferViewStride]; + } + srcData = (const unsigned char *) widenedIndices.data(); + gTypeComponent = GLTFTypeComponent::UnsignedShort; + bytesPerIndex = 2; + bufferViewStride = 2; + dataLength = elementCount * 2; + } + // Handle LINE_LOOP: Convert to Line by appending the first index at the end if (gPrimitive.mode == TINYGLTF_MODE_LINE_LOOP) { size_t newElementCount = elementCount + 1; @@ -2593,9 +3188,9 @@ bool VROGLTFLoader::processVertexElement(const tinygltf::Model &gModel, // Standard case: directly use the buffer data int primitiveCount = VROGeometryUtilGetPrimitiveCount((int) gIndicesAccessor.count, primitiveType); - std::shared_ptr data = std::make_shared((void *)gBuffer.data.data(), - dataLength, - dataOffset); + std::shared_ptr data = widenedIndices.empty() + ? std::make_shared((void *)gBuffer.data.data(), dataLength, dataOffset) + : std::make_shared((void *)widenedIndices.data(), dataLength, VRODataOwnership::Copy); std::shared_ptr element = std::make_shared(data, primitiveType, @@ -2611,11 +3206,17 @@ bool VROGLTFLoader::processVertexAttributes(const tinygltf::Model &gModel, std::map &gAttributes, std::vector> &sources, size_t geoElementIndex, - std::shared_ptr driver) { + std::shared_ptr driver, + int texcoordSet) { if (gAttributes.size() <= 0) { return false; } + // The UV set the material's textures sample. When it is not TEXCOORD_0 and the + // primitive has it, that set is the texture coordinates, in place of TEXCOORD_0. + std::string texcoordName = "TEXCOORD_" + VROStringUtil::toString(texcoordSet); + bool otherTexcoordSet = texcoordSet > 0 && gAttributes.find(texcoordName) != gAttributes.end(); + // Iterate through each Vertex Attribute (Position / Normals / etc) to parse its data for (auto const& gAttribute : gAttributes) { std::string attributeName = gAttribute.first; @@ -2623,6 +3224,14 @@ bool VROGLTFLoader::processVertexAttributes(const tinygltf::Model &gModel, // Ensure we only parse valid attributes, gracefully skip over the ones we do not yet support VROGeometrySourceSemantic attributeType = getGeometryAttribute(attributeName); + if (otherTexcoordSet) { + if (attributeName == "TEXCOORD_0") { + continue; + } + if (attributeName == texcoordName) { + attributeType = VROGeometrySourceSemantic::Texcoord; + } + } if (attributeType == VROGeometrySourceSemantic::Invalid) { continue; } @@ -2961,6 +3570,9 @@ std::shared_ptr VROGLTFLoader::getMaterial(const tinygltf::Model &g if (VROStringUtil::strcmpinsensitive(mode, "OPAQUE")) { vroMat->setTransparencyMode(VROTransparencyMode::RGBZero); + // OPAQUE ignores alpha: the base color's, the texture's and the vertex + // colors' (a texture's alpha often holds something else). + vroMat->setIgnoresDiffuseAlpha(true); } else if (VROStringUtil::strcmpinsensitive(mode, "MASK")) { vroMat->setTransparencyMode(VROTransparencyMode::Mask); // Parse alphaCutoff, default is 0.5 per glTF spec @@ -2969,8 +3581,14 @@ std::shared_ptr VROGLTFLoader::getMaterial(const tinygltf::Model &g alphaCutoff = (float)gAdditionalMap["alphaCutoff"].Factor(); } vroMat->setAlphaCutoff(alphaCutoff); + // What passes the cutoff is opaque: the alpha decides only what is discarded. + vroMat->setIgnoresDiffuseAlpha(true); } else if (VROStringUtil::strcmpinsensitive(mode, "BLEND")) { vroMat->setTransparencyMode(VROTransparencyMode::AOne); + // A blended surface does not write depth (as three.js, Babylon and Filament do + // for glTF BLEND): it would hide whatever is drawn behind it later, such as a + // decal that is nearly transparent hiding the mesh under it. + vroMat->setWritesToDepthBuffer(false); } // Process emissive material properties @@ -2988,6 +3606,12 @@ std::shared_ptr VROGLTFLoader::getMaterial(const tinygltf::Model &g std::shared_ptr emissiveTexture = getTexture(gModel, gAdditionalMap, "emissiveTexture", true); if (emissiveTexture != nullptr) { vroMat->getEmission().setTexture(emissiveTexture); + // The emissive texture samples through its own KHR_texture_transform: a glow + // map is often a small region of an atlas, placed apart from the base color. + GLTFTextureTransform emissiveTransform = readTextureTransform(gAdditionalMap["emissiveTexture"]); + if (emissiveTransform.present) { + vroMat->getEmission().setContentsTransform(textureTransformMatrix(emissiveTransform)); + } } // Check for KHR_materials_unlit extension diff --git a/ViroRenderer/VROGLTFLoader.h b/ViroRenderer/VROGLTFLoader.h index e62aae981..b1c201a8a 100644 --- a/ViroRenderer/VROGLTFLoader.h +++ b/ViroRenderer/VROGLTFLoader.h @@ -64,6 +64,7 @@ namespace tinygltf { class Skin; class Animation; class AnimationSampler; + class AnimationChannel; class Buffer; class BufferView; class Camera; @@ -128,7 +129,8 @@ class VROGLTFLoader { static bool processVertexAttributes(const tinygltf::Model &gModel, std::map &gAttributes, std::vector> &sources, size_t geoElementIndex, - std::shared_ptr driver); + std::shared_ptr driver, + int texcoordSet = 0); static void processTangent(std::vector> &elements, std::vector> &sources, size_t geoElementIndex); static void regenerateTangent(std::vector &posArray, @@ -197,6 +199,14 @@ class VROGLTFLoader { // Processing of Animation Data static bool processAnimations(const tinygltf::Model &gModel); + + // KHR_animation_pointer: rewrite pointers at a node's transform or weights into + // ordinary channels (before the model is shared), read the remaining ones, and + // once the nodes and materials exist, attach one animation per glTF animation. + static void normalizeAnimationPointers(tinygltf::Model &gModel); + static bool isPointerChannel(const tinygltf::AnimationChannel &channel); + static void processPointerChannels(const tinygltf::Model &gModel); + static void attachPointerAnimations(const tinygltf::Model &gModel, std::shared_ptr host); static bool processKeyFrameAnimations(const tinygltf::Model &gModel, std::map>> &gltfAnimatedNodes); // Resample all of a skin's animation channels onto a single common time-grid and merge each diff --git a/ViroRenderer/VROGeometrySubstrateOpenGL.cpp b/ViroRenderer/VROGeometrySubstrateOpenGL.cpp index 7d9d3f972..5b098c12f 100644 --- a/ViroRenderer/VROGeometrySubstrateOpenGL.cpp +++ b/ViroRenderer/VROGeometrySubstrateOpenGL.cpp @@ -179,6 +179,12 @@ VROVertexDescriptorOpenGL VROGeometrySubstrateOpenGL::configureVertexDescriptor( int attrIdx = VROGeometryUtilParseAttributeIndex(source->getSemantic()); std::pair format = parseVertexFormat(source); + // Bone indices are never negative, and the shaders take them as uvec4: bind + // every integer size unsigned, so the input and the attribute types agree + // (WebGL refuses a mismatch; desktop and mobile GL let it pass). + if (source->getSemantic() == VROGeometrySourceSemantic::BoneIndices && format.first == GL_INT) { + format.first = GL_UNSIGNED_INT; + } vd.attributes[vd.numAttributes].index = attrIdx; vd.attributes[vd.numAttributes].size = format.second; @@ -217,7 +223,8 @@ void VROGeometrySubstrateOpenGL::createVAO() { for (int i = 0; i < vd.numAttributes; i++) { if (!vd.attributes[i].normalized && - (vd.attributes[i].type == GL_INT || vd.attributes[i].type == GL_SHORT || + (vd.attributes[i].type == GL_INT || vd.attributes[i].type == GL_UNSIGNED_INT || + vd.attributes[i].type == GL_SHORT || vd.attributes[i].type == GL_UNSIGNED_BYTE || vd.attributes[i].type == GL_UNSIGNED_SHORT)) { GL( glVertexAttribIPointer(vd.attributes[i].index, vd.attributes[i].size, vd.attributes[i].type, vd.stride, (GLvoid *) vd.attributes[i].offset) ); diff --git a/ViroRenderer/VROMaterial.cpp b/ViroRenderer/VROMaterial.cpp index ab7d16902..1ac248403 100644 --- a/ViroRenderer/VROMaterial.cpp +++ b/ViroRenderer/VROMaterial.cpp @@ -46,6 +46,7 @@ VROMaterial::VROMaterial() : VROThreadRestricted(VROThreadName::Renderer), _transparency(1.0), _transparencyMode(VROTransparencyMode::AOne), _alphaCutoff(0.5f), // glTF 2.0 default value + _ignoresDiffuseAlpha(false), _lightingModel(VROLightingModel::Constant), _litPerPixel(true), _cullMode(VROCullMode::Back), @@ -99,6 +100,7 @@ VROMaterial::VROMaterial(std::shared_ptr material) : VROThreadRestr _transparency(material->_transparency), _transparencyMode(material->_transparencyMode), _alphaCutoff(material->_alphaCutoff), + _ignoresDiffuseAlpha(material->_ignoresDiffuseAlpha), _lightingModel(material->_lightingModel), _litPerPixel(material->_litPerPixel), _cullMode(material->_cullMode), @@ -171,6 +173,7 @@ void VROMaterial::copyFrom(std::shared_ptr material) { _transparency = material->_transparency; _transparencyMode = material->_transparencyMode; _alphaCutoff = material->_alphaCutoff; + _ignoresDiffuseAlpha = material->_ignoresDiffuseAlpha; _lightingModel = material->_lightingModel; _litPerPixel = material->_litPerPixel; _cullMode = material->_cullMode; @@ -359,6 +362,11 @@ bool VROMaterial::bindShader(int lightsHash, } bool VROMaterial::hasDiffuseAlpha() const { + // An opaque glTF material draws its diffuse alpha as 1: it sorts with the opaque + // objects, ahead of anything blended. + if (_ignoresDiffuseAlpha) { + return false; + } if (_diffuse->getTextureType() == VROTextureType::None) { return _diffuse->getColor().w < (1.0 - kEpsilon); } diff --git a/ViroRenderer/VROMaterial.h b/ViroRenderer/VROMaterial.h index 0e8dd40b0..81403ddb7 100644 --- a/ViroRenderer/VROMaterial.h +++ b/ViroRenderer/VROMaterial.h @@ -231,6 +231,21 @@ class VROMaterial : public VROAnimatable, public VROThreadRestricted { return _alphaCutoff; } + /* + When true, the alpha of the diffuse color, its texture and the vertex colors is + ignored: only the material's transparency and the node's opacity make it see + through (glTF alphaMode OPAQUE, where a texture's alpha may hold other data). + */ + void setIgnoresDiffuseAlpha(bool ignores) { + if (_ignoresDiffuseAlpha != ignores) { + _ignoresDiffuseAlpha = ignores; + updateSubstrate(); + } + } + bool getIgnoresDiffuseAlpha() const { + return _ignoresDiffuseAlpha; + } + /* Lighting models. */ @@ -577,6 +592,11 @@ class VROMaterial : public VROAnimatable, public VROThreadRestricted { Alpha cutoff threshold for Mask transparency mode. */ float _alphaCutoff; + + /* + Whether the diffuse alpha is ignored (glTF alphaMode OPAQUE). + */ + bool _ignoresDiffuseAlpha; /* The lighting model to use to compute the interaction between diff --git a/ViroRenderer/VROMaterialShaderBinding.cpp b/ViroRenderer/VROMaterialShaderBinding.cpp index cecb2981e..72dc77ac2 100644 --- a/ViroRenderer/VROMaterialShaderBinding.cpp +++ b/ViroRenderer/VROMaterialShaderBinding.cpp @@ -55,6 +55,8 @@ VROMaterialShaderBinding::VROMaterialShaderBinding(std::shared_ptrgetUniform("material_ao"); _alphaCutoffUniform = program->getUniform("material_alpha_cutoff"); _emissiveColorUniform = program->getUniform("material_emissive_color"); + _textureTransformUniform = program->getUniform("material_texture_transform"); + _emissiveTextureTransformUniform = program->getUniform("material_emissive_texture_transform"); _normalMatrixUniform = program->getUniform("normal_matrix"); _modelMatrixUniform = program->getUniform("model_matrix"); @@ -310,6 +314,12 @@ void VROMaterialShaderBinding::bindMaterialUniforms(const VROMaterial &material, if (_aoUniform != nullptr) { _aoUniform->setFloat(material.getAmbientOcclusion().getColor().x); } + if (_textureTransformUniform != nullptr) { + _textureTransformUniform->setMat4(material.getDiffuse().getContentsTransform()); + } + if (_emissiveTextureTransformUniform != nullptr) { + _emissiveTextureTransformUniform->setMat4(material.getEmission().getContentsTransform()); + } if (_emissiveColorUniform != nullptr) { VROVector4f emissiveColor = material.getEmission().getColor(); if (driver->isLinearRenderingEnabled()) { diff --git a/ViroRenderer/VROMaterialShaderBinding.h b/ViroRenderer/VROMaterialShaderBinding.h index c4fe64eea..ba1aee724 100644 --- a/ViroRenderer/VROMaterialShaderBinding.h +++ b/ViroRenderer/VROMaterialShaderBinding.h @@ -114,6 +114,8 @@ class VROMaterialShaderBinding { VROUniform *_aoUniform; VROUniform *_alphaCutoffUniform; VROUniform *_emissiveColorUniform; + VROUniform *_textureTransformUniform; + VROUniform *_emissiveTextureTransformUniform; VROUniform *_normalMatrixUniform; VROUniform *_modelMatrixUniform; diff --git a/ViroRenderer/VROMaterialSubstrateMetal.cpp b/ViroRenderer/VROMaterialSubstrateMetal.cpp index 091c04990..71efe6a24 100644 --- a/ViroRenderer/VROMaterialSubstrateMetal.cpp +++ b/ViroRenderer/VROMaterialSubstrateMetal.cpp @@ -484,6 +484,10 @@ VROConcurrentBuffer &VROMaterialSubstrateMetal::bindMaterialUniforms(float opaci uniforms->ao = _material.getAmbientOcclusion().getColor().x; uniforms->tone_mapping_mask = _material.needsToneMapping() ? 1.0f : 0.0f; uniforms->bloom_threshold = _material.getBloomThreshold(); + uniforms->texture_transform = toMatrixFloat4x4(_material.getDiffuse().getContentsTransform()); + VROVector4f emission = _material.getEmission().getColor(); + uniforms->emissive_color = toVectorFloat3(VROVector3f(emission.x, emission.y, emission.z)); + uniforms->ignores_diffuse_alpha = _material.getIgnoresDiffuseAlpha() ? 1 : 0; // Fill custom uniforms buffer based on the layout created during inflation if (!_material.getShaderModifiers().empty()) { diff --git a/ViroRenderer/VROMaterialVisual.cpp b/ViroRenderer/VROMaterialVisual.cpp index f33a14eeb..42d17120a 100644 --- a/ViroRenderer/VROMaterialVisual.cpp +++ b/ViroRenderer/VROMaterialVisual.cpp @@ -35,7 +35,8 @@ VROMaterialVisual::VROMaterialVisual(VROMaterial &material, const VROMaterialVis _contentsColor(visual._contentsColor), _contentsTexture(visual._contentsTexture), _intensity(visual._intensity), - _contentsTransform(visual._contentsTransform) + _contentsTransform(visual._contentsTransform), + _hasContentsTransform(visual._hasContentsTransform) {} void VROMaterialVisual::deleteGL() { @@ -50,6 +51,15 @@ void VROMaterialVisual::copyFrom(const VROMaterialVisual &visual) { _contentsTexture = visual._contentsTexture; _intensity = visual._intensity; _contentsTransform = visual._contentsTransform; + _hasContentsTransform = visual._hasContentsTransform; +} + +void VROMaterialVisual::setContentsTransform(VROMatrix4f transform) { + _contentsTransform = transform; + if (!_hasContentsTransform) { + _hasContentsTransform = true; + _material.updateSubstrate(); + } } void VROMaterialVisual::clear() { diff --git a/ViroRenderer/VROMaterialVisual.h b/ViroRenderer/VROMaterialVisual.h index d37fb2fa2..572b5b35c 100644 --- a/ViroRenderer/VROMaterialVisual.h +++ b/ViroRenderer/VROMaterialVisual.h @@ -44,14 +44,16 @@ class VROMaterialVisual { _material(material), _permissibleContentsMask(permissibleContentsMask), _contentsColor({ 1.0, 1.0, 1.0, 1.0 }), - _intensity(1.0) + _intensity(1.0), + _hasContentsTransform(false) {} VROMaterialVisual(VROMaterial &material, int permissibleContentsMask, VROVector4f color) : _material(material), _permissibleContentsMask(permissibleContentsMask), _contentsColor(color), - _intensity(1.0) + _intensity(1.0), + _hasContentsTransform(false) {} /* @@ -96,6 +98,30 @@ class VROMaterialVisual { VROVector4f getColor() const { return _contentsColor; } + + /* + Set the color without the fade or the substrate rebuild setColor() does, for a + value driven every frame by an animation (glTF KHR_animation_pointer). The + shaders read the color when they bind the material, so it shows on the next draw. + */ + void setColorImmediate(VROVector4f color) { + _contentsColor = color; + } + + /* + The transform applied to the texture coordinates before this visual's texture is + sampled (glTF KHR_texture_transform). Only the diffuse visual's is used, and it + applies to every texture of the material, which all sample the diffuse texture + coordinates. The first call rebuilds the substrate, so the shader gains the + transform; later calls, such as an animation's, only change the value. + */ + void setContentsTransform(VROMatrix4f transform); + const VROMatrix4f &getContentsTransform() const { + return _contentsTransform; + } + bool hasContentsTransform() const { + return _hasContentsTransform; + } std::shared_ptr getTexture() const { if (_contentsTexture) { @@ -144,6 +170,7 @@ class VROMaterialVisual { the material is attached to. */ VROMatrix4f _contentsTransform; + bool _hasContentsTransform; }; diff --git a/ViroRenderer/VRONode.cpp b/ViroRenderer/VRONode.cpp index a5840affb..a2b017495 100644 --- a/ViroRenderer/VRONode.cpp +++ b/ViroRenderer/VRONode.cpp @@ -702,6 +702,13 @@ void VRONode::updateVisibility(const VRORenderContext &context) { result = frustum.intersectAllOpt(_worldUmbrellaBoundingBox, &_umbrellaBoxMetadata); } + // A skinned mesh is drawn where its bones put it, which its bind-pose bounds do not + // tell: never cull a subtree holding one (a model posed away from its bind pose lost + // its eyes when the camera came close). + if (result == VROFrustumResult::Outside && hasSkinnedGeometryInSubtree()) { + result = VROFrustumResult::Intersects; + } + // Process the results of the frustum test, iterating down the tree if there // was an intersection, or else wholesale including or excluding all child nodes // in the other two cases. @@ -719,6 +726,18 @@ void VRONode::updateVisibility(const VRORenderContext &context) { } } +bool VRONode::hasSkinnedGeometryInSubtree() const { + if (_geometry && _geometry->getSkinner()) { + return true; + } + for (const std::shared_ptr &childNode : _subnodes) { + if (childNode->hasSkinnedGeometryInSubtree()) { + return true; + } + } + return false; +} + void VRONode::setVisibilityRecursive(bool visible) { _visible = visible; diff --git a/ViroRenderer/VRONode.h b/ViroRenderer/VRONode.h index 27a9b17df..5a835771f 100644 --- a/ViroRenderer/VRONode.h +++ b/ViroRenderer/VRONode.h @@ -1075,6 +1075,13 @@ class VRONode : public VROAnimatable, public VROThreadRestricted { given value. */ void setVisibilityRecursive(bool visible); + + /* + True if this node or a descendant draws a skinned geometry. Its bounds come from + the bind pose, not from where the bones place it, so a frustum test on them can + cull a mesh that is in view. + */ + bool hasSkinnedGeometryInSubtree() const; /* Recursively expand the given bounding box by this node's _worldBoundingBox. diff --git a/ViroRenderer/VROShaderCapabilities.cpp b/ViroRenderer/VROShaderCapabilities.cpp index 7b4e3c576..3e06cbb92 100644 --- a/ViroRenderer/VROShaderCapabilities.cpp +++ b/ViroRenderer/VROShaderCapabilities.cpp @@ -43,6 +43,9 @@ VROMaterialShaderCapabilities VROShaderCapabilities::deriveMaterialCapabilitiesK cap.roughnessMap = false; cap.metalnessMap = false; cap.aoMap = false; + cap.textureTransform = false; + cap.emissiveTexture = false; + cap.ignoresDiffuseAlpha = material.getIgnoresDiffuseAlpha(); cap.diffuseTextureStereoMode = VROStereoMode::None; cap.bloom = false; cap.postProcessMask = false; @@ -87,6 +90,14 @@ VROMaterialShaderCapabilities VROShaderCapabilities::deriveMaterialCapabilitiesK cap.diffuseEGLModifier = false; } + // Texture coordinate transform (glTF KHR_texture_transform), applied before any + // texture is sampled + cap.textureTransform = diffuse.hasContentsTransform(); + + // Emissive Map (glTF emissiveTexture), scaled by the emissive color + cap.emissiveTexture = material.getEmission().getTextureType() == VROTextureType::Texture2D && + lightingModel != VROLightingModel::Constant; + // Specular Map if (specular.getTextureType() == VROTextureType::Texture2D && (lightingModel == VROLightingModel::Blinn || lightingModel == VROLightingModel::Phong)) { diff --git a/ViroRenderer/VROShaderCapabilities.h b/ViroRenderer/VROShaderCapabilities.h index 4089db951..7ab2f4207 100644 --- a/ViroRenderer/VROShaderCapabilities.h +++ b/ViroRenderer/VROShaderCapabilities.h @@ -62,6 +62,9 @@ struct VROMaterialShaderCapabilities { bool normalTexture; bool reflectiveTexture; bool roughnessMap, metalnessMap, aoMap; + bool textureTransform; + bool emissiveTexture; + bool ignoresDiffuseAlpha; bool bloom; bool postProcessMask; bool receivesShadows; @@ -72,13 +75,13 @@ struct VROMaterialShaderCapabilities { bool operator< (const VROMaterialShaderCapabilities& r) const { return std::tie(lightingModel, diffuseTexture, diffuseTextureStereoMode, diffuseEGLModifier, specularTexture, normalTexture, reflectiveTexture, - roughnessMap, metalnessMap, aoMap, bloom, postProcessMask, + roughnessMap, metalnessMap, aoMap, textureTransform, emissiveTexture, ignoresDiffuseAlpha, bloom, postProcessMask, receivesShadows, chromaKeyFiltering, chromaKeyRed, chromaKeyGreen, chromaKeyBlue, additionalModifierKeys) < std::tie(r.lightingModel, r.diffuseTexture, r.diffuseTextureStereoMode, r.diffuseEGLModifier, r.specularTexture, r.normalTexture, r.reflectiveTexture, - r.roughnessMap, r.metalnessMap, r.aoMap, r.bloom, r.postProcessMask, + r.roughnessMap, r.metalnessMap, r.aoMap, r.textureTransform, r.emissiveTexture, r.ignoresDiffuseAlpha, r.bloom, r.postProcessMask, r.receivesShadows, r.chromaKeyFiltering, r.chromaKeyRed, r.chromaKeyGreen, r.chromaKeyBlue, r.additionalModifierKeys); diff --git a/ViroRenderer/VROShaderFactory.cpp b/ViroRenderer/VROShaderFactory.cpp index 83f57fa32..9fd1c8804 100644 --- a/ViroRenderer/VROShaderFactory.cpp +++ b/ViroRenderer/VROShaderFactory.cpp @@ -40,8 +40,11 @@ #include "VRODriverOpenGL.h" #include +static thread_local std::shared_ptr sTextureTransformModifier; static thread_local std::shared_ptr sDiffuseTextureModifier; static thread_local std::shared_ptr sSpecularTextureModifier; +static thread_local std::shared_ptr sEmissiveTextureModifier; +static thread_local std::shared_ptr sOpaqueAlphaModifier; static thread_local std::shared_ptr sNormalMapTextureModifier; static thread_local std::shared_ptr sReflectiveTextureModifier; static thread_local std::shared_ptr sRoughnessTextureModifier; @@ -156,6 +159,12 @@ std::shared_ptr VROShaderFactory::buildShader(VROShaderCapabil std::vector samplers; std::vector> modifiers; + // The texture coordinate transform goes first: every texture modifier below + // samples _surface.diffuse_texcoord + if (materialCapabilities.textureTransform) { + modifiers.push_back(createTextureTransformModifier()); + } + // Stereo mode must be placed prior to diffuse texture (because it modifies // the texture coordinates used when sampling the diffuse texture) if (materialCapabilities.diffuseTextureStereoMode != VROStereoMode::None) { @@ -194,6 +203,12 @@ std::shared_ptr VROShaderFactory::buildShader(VROShaderCapabil materialCapabilities.chromaKeyBlue)); } + // Emissive Map. Only the standard fragment shader has an emissive term to scale. + if (materialCapabilities.emissiveTexture && lightingModel != VROLightingModel::Constant) { + samplers.push_back("emissive_texture"); + modifiers.push_back(createEmissiveTextureModifier()); + } + // Normal Map (note this must be placed before the PBR surface modifier, which uses the normal) if (materialCapabilities.normalTexture) { samplers.push_back("normal_texture"); @@ -271,6 +286,12 @@ std::shared_ptr VROShaderFactory::buildShader(VROShaderCapabil samplers.push_back("shadow_map"); } + // Opaque alpha (glTF alphaMode OPAQUE): after every lighting modifier, so none + // can bring the diffuse alpha back, and before bloom and tone mapping, which read it + if (materialCapabilities.ignoresDiffuseAlpha) { + modifiers.push_back(createOpaqueAlphaModifier()); + } + // Bloom. Gate on the pipeline-level bloom flag (not the per-material one): // when the HDR target has a bloom color attachment, EVERY shader drawn into // it must write that output, or strict backends (WebGL2) raise @@ -376,6 +397,25 @@ std::shared_ptr VROShaderFactory::buildShader(VROShaderCapabil #pragma mark - Texture Modifiers +std::shared_ptr VROShaderFactory::createTextureTransformModifier() { + /* + Modifier that transforms the texture coordinates every texture of the material + samples (glTF KHR_texture_transform: offset, rotation and scale). The matrix is + the diffuse visual's contents transform, bound with the material. + */ + if (!sTextureTransformModifier) { + std::vector modifierCode = { + "uniform highp mat4 material_texture_transform;", + "_surface.diffuse_texcoord = (material_texture_transform * vec4(_surface.diffuse_texcoord, 0.0, 1.0)).xy;", + "_surface.specular_texcoord = _surface.diffuse_texcoord;" + }; + sTextureTransformModifier = std::make_shared(VROShaderEntryPoint::Surface, + modifierCode); + sTextureTransformModifier->setName("texture_transform"); + } + return sTextureTransformModifier; +} + std::shared_ptr VROShaderFactory::createDiffuseTextureModifier() { /* Modifier that multiplies the material's surface color by a diffuse texture. @@ -392,6 +432,40 @@ std::shared_ptr VROShaderFactory::createDiffuseTextureModifie return sDiffuseTextureModifier; } +std::shared_ptr VROShaderFactory::createEmissiveTextureModifier() { + /* + Modifier that scales the material's emissive color by its emissive texture, sampled + through the emission's own texture transform (identity unless glTF gives one). + */ + if (!sEmissiveTextureModifier) { + std::vector modifierCode = { + "uniform sampler2D emissive_texture;", + "uniform highp mat4 material_emissive_texture_transform;", + "_emissive *= texture(emissive_texture, (material_emissive_texture_transform * vec4(v_texcoord, 0.0, 1.0)).xy).rgb;" + }; + sEmissiveTextureModifier = std::make_shared(VROShaderEntryPoint::Surface, + modifierCode); + sEmissiveTextureModifier->setName("emissive"); + } + return sEmissiveTextureModifier; +} + +std::shared_ptr VROShaderFactory::createOpaqueAlphaModifier() { + /* + Modifier that drops the diffuse color's, texture's and vertex colors' alpha from + the output: only the material's transparency and the node's opacity remain. + */ + if (!sOpaqueAlphaModifier) { + std::vector modifierCode = { + "_output_color.a = _surface.alpha;" + }; + sOpaqueAlphaModifier = std::make_shared(VROShaderEntryPoint::Fragment, + modifierCode); + sOpaqueAlphaModifier->setName("opaque_alpha"); + } + return sOpaqueAlphaModifier; +} + std::shared_ptr VROShaderFactory::createSpecularTextureModifier() { /* Modifier that multiplies the material's specular color by a specular texture. @@ -757,7 +831,7 @@ std::shared_ptr VROShaderFactory::createPBRConstantAmbientFra // diminish a shadow (in VROARShadow), we have to set this term to the final // ambient contribution. "_ambient = _ambient * albedo * _surface.ao;", - "_output_color = vec4(_ambient + _diffuse, _output_color.a);", + "_output_color = vec4(_ambient + _diffuse + _emissive, _output_color.a);", }; sPBRConstantAmbientFragmentModifier = std::make_shared(VROShaderEntryPoint::Fragment, modifierCode); @@ -785,7 +859,7 @@ std::shared_ptr VROShaderFactory::createPBRDiffuseIrradianceF // diminish a shadow (in VROARShadow), we have to set this term to the final // ambient contribution. "_ambient = (_ambient * albedo + ambient_kD * irradiance * albedo) * _surface.ao;", - "_output_color = vec4(_ambient + _diffuse, _output_color.a);", + "_output_color = vec4(_ambient + _diffuse + _emissive, _output_color.a);", }; sPBRDiffuseIrradianceFragmentModifier = std::make_shared(VROShaderEntryPoint::Fragment, modifierCode); @@ -825,7 +899,7 @@ std::shared_ptr VROShaderFactory::createPBRDiffuseAndSpecular // Because the _ambient term is used by AR shadows to compute how much to diminish a shadow // (in VROARShadow), we have to set this term to the final ambient contribution. "_ambient = (_ambient * albedo + ambient_kD * irradiance * albedo + ambient_specular) * _surface.ao;", - "_output_color = vec4(_ambient + _diffuse, _output_color.a);", + "_output_color = vec4(_ambient + _diffuse + _emissive, _output_color.a);", }; sPBRDiffuseAndSpecularIrradianceFragmentModifier = std::make_shared(VROShaderEntryPoint::Fragment, modifierCode); diff --git a/ViroRenderer/VROShaderFactory.h b/ViroRenderer/VROShaderFactory.h index 3bff756ee..fc3cd8f79 100644 --- a/ViroRenderer/VROShaderFactory.h +++ b/ViroRenderer/VROShaderFactory.h @@ -114,7 +114,10 @@ class VROShaderFactory { const std::vector> &modifiers, std::shared_ptr &driver); + std::shared_ptr createTextureTransformModifier(); std::shared_ptr createDiffuseTextureModifier(); + std::shared_ptr createEmissiveTextureModifier(); + std::shared_ptr createOpaqueAlphaModifier(); std::shared_ptr createSpecularTextureModifier(); std::shared_ptr createNormalMapTextureModifier(); std::shared_ptr createReflectiveTextureModifier(); diff --git a/ViroRenderer/VROShaderProgram.cpp b/ViroRenderer/VROShaderProgram.cpp index 5f4365231..93ee751fa 100644 --- a/ViroRenderer/VROShaderProgram.cpp +++ b/ViroRenderer/VROShaderProgram.cpp @@ -106,7 +106,7 @@ VROShaderProgram::VROShaderProgram(std::string vertexShader, std::string fragmen vertexAssignments += "_geometry.tangent = tangent;\n"; } if ((_attributes & (int)VROShaderMask::BoneIndex) != 0) { - vertexAssignments += "_geometry.bone_indices = bone_indices;\n"; + vertexAssignments += "_geometry.bone_indices = ivec4(bone_indices);\n"; } if ((_attributes & (int)VROShaderMask::BoneWeight) != 0) { vertexAssignments += "_geometry.bone_weights = bone_weights;\n"; diff --git a/ViroRenderer/VROSharedStructures.h b/ViroRenderer/VROSharedStructures.h index 4778cdd68..10a0d5ff1 100644 --- a/ViroRenderer/VROSharedStructures.h +++ b/ViroRenderer/VROSharedStructures.h @@ -81,6 +81,13 @@ typedef struct { // 0 otherwise; bloom_threshold is negative when the material contributes no bloom. float tone_mapping_mask; float bloom_threshold; + + // glTF KHR_texture_transform, applied to the texture coordinates in the vertex + // stage (identity when the material has none), and the emissive colour the PBR + // path adds to its output. + matrix_float4x4 texture_transform; + vector_float3 emissive_color; + int ignores_diffuse_alpha; } VROMaterialUniforms; typedef struct { diff --git a/ViroRenderer/standard_fsh.glsl b/ViroRenderer/standard_fsh.glsl index a51407ade..422a51c15 100644 --- a/ViroRenderer/standard_fsh.glsl +++ b/ViroRenderer/standard_fsh.glsl @@ -47,6 +47,7 @@ void main() { highp vec3 _ambient = ambient_light_color.xyz; highp vec3 _diffuse = vec3(0, 0, 0); highp vec3 _specular = vec3(0, 0, 0); + highp vec3 _emissive = material_emissive_color; #pragma surface_modifier_body @@ -68,7 +69,7 @@ void main() { highp vec4 _output_color = vec4(_ambient * _surface.diffuse_color.xyz + _diffuse * _surface.diffuse_color.xyz * _surface.diffuse_intensity + _specular * _surface.specular_color + - material_emissive_color, + _emissive, _surface.alpha * _surface.diffuse_color.a); if (material_alpha_cutoff > 0.0 && _output_color.a < material_alpha_cutoff) { diff --git a/ViroRenderer/standard_vsh.glsl b/ViroRenderer/standard_vsh.glsl index 55a934efd..e0dde3210 100644 --- a/ViroRenderer/standard_vsh.glsl +++ b/ViroRenderer/standard_vsh.glsl @@ -32,7 +32,9 @@ in vec3 normal; in vec2 texcoord; in vec4 tangent; in vec4 bone_weights; -in ivec4 bone_indices; +// Unsigned: glTF joints are unsigned bytes or shorts, and WebGL refuses an +// unsigned attribute bound to a signed input. The substrate binds them unsigned. +in uvec4 bone_indices; in vec3 morph_0; in vec3 morph_1; in vec3 morph_2; diff --git a/ViroRenderer/standard_vsh_adreno.glsl b/ViroRenderer/standard_vsh_adreno.glsl index 8c6ba71e9..a5847b00f 100644 --- a/ViroRenderer/standard_vsh_adreno.glsl +++ b/ViroRenderer/standard_vsh_adreno.glsl @@ -26,7 +26,7 @@ in vec3 normal; in vec2 texcoord; in vec4 tangent; in vec4 bone_weights; -in ivec4 bone_indices; +in uvec4 bone_indices; in vec3 morph_0; in vec3 morph_1; in vec3 morph_2; @@ -57,7 +57,7 @@ void main() { _geometry_texcoord = texcoord; _geometry_tangent = tangent; _geometry_bone_weights = bone_weights; - _geometry_bone_indices = bone_indices; + _geometry_bone_indices = ivec4(bone_indices); _transforms_model_matrix = model_matrix; _transforms_view_matrix = view_matrix; diff --git a/ViroRenderer/tiny_gltf.h b/ViroRenderer/tiny_gltf.h index 3994a2b86..0761d1261 100755 --- a/ViroRenderer/tiny_gltf.h +++ b/ViroRenderer/tiny_gltf.h @@ -384,6 +384,9 @@ struct Parameter { std::string string_value; std::vector number_array; std::map json_double_value; + // ReactVision: the "extensions" of a JSON-object parameter, such as a texture + // info's KHR_texture_transform. json_double_value keeps only its numbers. + std::map extensions; // context sensitive methods. depending the type of the Parameter you are // accessing, these are either valid or not @@ -431,9 +434,11 @@ typedef std::map ExtensionMap; struct AnimationChannel { int sampler; // required - int target_node; // required (index of the node to target) + int target_node; // optional (-1 when absent, as for KHR_animation_pointer) std::string target_path; // required in ["translation", "rotation", "scale", - // "weights"] + // "weights", "pointer"] + // ReactVision: the target's "extensions" (KHR_animation_pointer's pointer). + ExtensionMap target_extensions; Value extras; AnimationChannel() : sampler(-1), target_node(-1) {} @@ -1990,6 +1995,8 @@ static bool ParseJSONProperty(std::map *ret, return true; } +static bool ParseExtensionsProperty(ExtensionMap *ret, std::string* err, const json &o); + static bool ParseParameterProperty(Parameter *param, std::string *err, const json &o, const std::string &prop, bool required) { @@ -2012,6 +2019,7 @@ static bool ParseParameterProperty(Parameter *param, std::string *err, return true; } else if (ParseJSONProperty(¶m->json_double_value, err, o, prop, false)) { + ParseExtensionsProperty(¶m->extensions, err, o.find(prop).value()); return true; } else if (ParseBooleanProperty(¶m->bool_value, err, o, prop, false)) { return true; @@ -2732,12 +2740,9 @@ static bool ParseAnimationChannel(AnimationChannel *channel, std::string *err, if ((targetIt != o.end()) && targetIt.value().is_object()) { const json &target_object = targetIt.value(); - if (!ParseNumberProperty(&targetIndex, err, target_object, "node", true)) { - if (err) { - (*err) += "`node` field is missing in animation.channels.target\n"; - } - return false; - } + // `node` is optional in glTF 2.0: a KHR_animation_pointer target has none. + ParseNumberProperty(&targetIndex, err, target_object, "node", false); + ParseExtensionsProperty(&channel->target_extensions, err, target_object); if (!ParseStringProperty(&channel->target_path, err, target_object, "path", true)) { @@ -2792,12 +2797,10 @@ static bool ParseAnimation(Animation *animation, std::string *err, } return false; } + // `interpolation` is optional in glTF 2.0 and defaults to LINEAR. if (!ParseStringProperty(&sampler.interpolation, err, s, - "interpolation", true)) { - if (err) { - (*err) += "`interpolation` field is missing in animation.sampler\n"; - } - return false; + "interpolation", false)) { + sampler.interpolation = "LINEAR"; } if (!ParseNumberProperty(&outputIndex, err, s, "output", true)) { if (err) { diff --git a/android/sharedCode/src/main/assets/standard_fsh.glsl b/android/sharedCode/src/main/assets/standard_fsh.glsl index a51407ade..422a51c15 100644 --- a/android/sharedCode/src/main/assets/standard_fsh.glsl +++ b/android/sharedCode/src/main/assets/standard_fsh.glsl @@ -47,6 +47,7 @@ void main() { highp vec3 _ambient = ambient_light_color.xyz; highp vec3 _diffuse = vec3(0, 0, 0); highp vec3 _specular = vec3(0, 0, 0); + highp vec3 _emissive = material_emissive_color; #pragma surface_modifier_body @@ -68,7 +69,7 @@ void main() { highp vec4 _output_color = vec4(_ambient * _surface.diffuse_color.xyz + _diffuse * _surface.diffuse_color.xyz * _surface.diffuse_intensity + _specular * _surface.specular_color + - material_emissive_color, + _emissive, _surface.alpha * _surface.diffuse_color.a); if (material_alpha_cutoff > 0.0 && _output_color.a < material_alpha_cutoff) { diff --git a/android/sharedCode/src/main/assets/standard_vsh.glsl b/android/sharedCode/src/main/assets/standard_vsh.glsl index 55a934efd..e0dde3210 100644 --- a/android/sharedCode/src/main/assets/standard_vsh.glsl +++ b/android/sharedCode/src/main/assets/standard_vsh.glsl @@ -32,7 +32,9 @@ in vec3 normal; in vec2 texcoord; in vec4 tangent; in vec4 bone_weights; -in ivec4 bone_indices; +// Unsigned: glTF joints are unsigned bytes or shorts, and WebGL refuses an +// unsigned attribute bound to a signed input. The substrate binds them unsigned. +in uvec4 bone_indices; in vec3 morph_0; in vec3 morph_1; in vec3 morph_2; diff --git a/android/sharedCode/src/main/assets/standard_vsh_adreno.glsl b/android/sharedCode/src/main/assets/standard_vsh_adreno.glsl index 8c6ba71e9..a5847b00f 100644 --- a/android/sharedCode/src/main/assets/standard_vsh_adreno.glsl +++ b/android/sharedCode/src/main/assets/standard_vsh_adreno.glsl @@ -26,7 +26,7 @@ in vec3 normal; in vec2 texcoord; in vec4 tangent; in vec4 bone_weights; -in ivec4 bone_indices; +in uvec4 bone_indices; in vec3 morph_0; in vec3 morph_1; in vec3 morph_2; @@ -57,7 +57,7 @@ void main() { _geometry_texcoord = texcoord; _geometry_tangent = tangent; _geometry_bone_weights = bone_weights; - _geometry_bone_indices = bone_indices; + _geometry_bone_indices = ivec4(bone_indices); _transforms_model_matrix = model_matrix; _transforms_view_matrix = view_matrix; diff --git a/wasm/preload/standard_fsh.glsl b/wasm/preload/standard_fsh.glsl index a51407ade..422a51c15 100644 --- a/wasm/preload/standard_fsh.glsl +++ b/wasm/preload/standard_fsh.glsl @@ -47,6 +47,7 @@ void main() { highp vec3 _ambient = ambient_light_color.xyz; highp vec3 _diffuse = vec3(0, 0, 0); highp vec3 _specular = vec3(0, 0, 0); + highp vec3 _emissive = material_emissive_color; #pragma surface_modifier_body @@ -68,7 +69,7 @@ void main() { highp vec4 _output_color = vec4(_ambient * _surface.diffuse_color.xyz + _diffuse * _surface.diffuse_color.xyz * _surface.diffuse_intensity + _specular * _surface.specular_color + - material_emissive_color, + _emissive, _surface.alpha * _surface.diffuse_color.a); if (material_alpha_cutoff > 0.0 && _output_color.a < material_alpha_cutoff) { diff --git a/wasm/preload/standard_vsh.glsl b/wasm/preload/standard_vsh.glsl index 55a934efd..e0dde3210 100644 --- a/wasm/preload/standard_vsh.glsl +++ b/wasm/preload/standard_vsh.glsl @@ -32,7 +32,9 @@ in vec3 normal; in vec2 texcoord; in vec4 tangent; in vec4 bone_weights; -in ivec4 bone_indices; +// Unsigned: glTF joints are unsigned bytes or shorts, and WebGL refuses an +// unsigned attribute bound to a signed input. The substrate binds them unsigned. +in uvec4 bone_indices; in vec3 morph_0; in vec3 morph_1; in vec3 morph_2; diff --git a/wasm/preload/standard_vsh_adreno.glsl b/wasm/preload/standard_vsh_adreno.glsl index 8c6ba71e9..a5847b00f 100644 --- a/wasm/preload/standard_vsh_adreno.glsl +++ b/wasm/preload/standard_vsh_adreno.glsl @@ -26,7 +26,7 @@ in vec3 normal; in vec2 texcoord; in vec4 tangent; in vec4 bone_weights; -in ivec4 bone_indices; +in uvec4 bone_indices; in vec3 morph_0; in vec3 morph_1; in vec3 morph_2; @@ -57,7 +57,7 @@ void main() { _geometry_texcoord = texcoord; _geometry_tangent = tangent; _geometry_bone_weights = bone_weights; - _geometry_bone_indices = bone_indices; + _geometry_bone_indices = ivec4(bone_indices); _transforms_model_matrix = model_matrix; _transforms_view_matrix = view_matrix;