{"id":714,"date":"2026-09-28T23:04:13","date_gmt":"2026-09-28T14:04:13","guid":{"rendered":"https:\/\/www.dtc.teu.ac.jp\/?p=714"},"modified":"2026-09-28T23:20:04","modified_gmt":"2026-09-28T14:20:04","slug":"tokkio-custom-avatar-runtime","status":"publish","type":"post","link":"https:\/\/www.dtc.teu.ac.jp\/ja\/tokkio-custom-avatar-runtime\/","title":{"rendered":"NVIDIA Tokkio\u3067\u30ab\u30b9\u30bf\u30e0\u30a2\u30d0\u30bf\u30fc\u3092\u52d5\u304b\u3059\uff1a\u30e9\u30f3\u30bf\u30a4\u30e0\u7d71\u5408\u3001\u65e5\u672c\u8a9e\u97f3\u58f0\u3001\u672c\u756a\u904b\u7528"},"content":{"rendered":"<p class=\"has-link-color wp-elements-1 wp-block-paragraph\">This is the second of two articles on running a custom Character Creator 5 (CC5) character as an NVIDIA Tokkio avatar. It starts from <a href=\"https:\/\/www.dtc.teu.ac.jp\/cc5-tokkio-avatar\/\">Part 1<\/a>&#8216;s hand-off package <a href=\"#ref-1\">[1]<\/a> and ends with the avatar holding Japanese conversations in a browser, as it did over two days of a live event in September 2026.<\/p>\n\n<p class=\"has-link-color wp-elements-2 wp-block-paragraph\">We ran Tokkio 5.0 GA (general availability) <a href=\"#ref-2\">[2]<\/a>, kept NVIDIA&#8217;s container images as shipped, and replaced the language model (LLM) and text-to-speech (TTS) services behind the ACE Controller with our own. Our own measurements carry a sample size and a date. We publish no performance figures for NVIDIA components.<\/p>\n\n\n<nav aria-label=\"Contents\" style=\"border-top:1px solid #e4e4e7;margin:0 0 40px 0;padding:16px 0 0 0\">\n<p style=\"margin:0 0 10px 0;font-weight:700;color:#18181b\">Contents<\/p>\n<div style=\"display:grid;grid-template-columns:repeat(auto-fit,minmax(260px,1fr));gap:0 40px\">\n<div>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-1\" style=\"color:#4d7c0f;text-decoration:none\">1. Define the runtime deliverable<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-2\" style=\"color:#4d7c0f;text-decoration:none\">2. Pin the stack<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-3\" style=\"color:#4d7c0f;text-decoration:none\">3. Keep the ACE Controller, replace what sits behind it<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-4\" style=\"color:#4d7c0f;text-decoration:none\">4. Accept the packaged renderer as a runtime contract<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-5\" style=\"color:#4d7c0f;text-decoration:none\">5. Two pipecat details that decide what the avatar remembers<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-6\" style=\"color:#4d7c0f;text-decoration:none\">6. Design against dead air<\/a><\/p>\n<\/div>\n<div>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-7\" style=\"color:#4d7c0f;text-decoration:none\">7. Make Japanese speech sound right<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-8\" style=\"color:#4d7c0f;text-decoration:none\">8. Fix the video path, not the skin<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-9\" style=\"color:#4d7c0f;text-decoration:none\">9. When GPU pods silently lose the GPU<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-10\" style=\"color:#4d7c0f;text-decoration:none\">10. Common problems and where to fix them<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-11\" style=\"color:#4d7c0f;text-decoration:none\">11. What we have not measured<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-next\" style=\"color:#4d7c0f;text-decoration:none\">Where this leaves the series<\/a><\/p>\n<p style=\"margin:0 0 6px 0;font-size:0.95em;line-height:1.5\"><a href=\"#section-references\" style=\"color:#4d7c0f;text-decoration:none\">References<\/a><\/p>\n<\/div>\n<\/div>\n<\/nav>\n\n\n<h2 class=\"wp-block-heading\" id=\"section-1\">1. Define the runtime deliverable<\/h2>\n\n<p class=\"wp-block-paragraph\">Our definition of done was a Japanese conversation with the avatar in a browser, with lip sync. Every startup check had to pass, the renderer had to survive a soak of at least 12 hours without a restart, and no silence inside a turn could last longer than five seconds.<\/p>\n\n<p class=\"wp-block-paragraph\">Running is not ready: we saw the stage dead while every pod reported Running. On 2026-09-17 the startup check had 48 items and passed all of them on the revision we took to the event. Gaps remain: the exact renderer binary we shipped has no continuous 12-hour run on file, and no turn was ever timed end to end against the five-second limit (section 11).<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-2\">2. Pin the stack<\/h2>\n\n<p class=\"wp-block-paragraph\">As of September 2026, 5.0 GA is the latest published Tokkio release and its documentation has moved to the archive site, so pin everything.<\/p>\n\n\n<div style=\"overflow:auto;margin:20px 0 28px 0\">\n<table style=\"width:100%;border-collapse:collapse;font-size:0.93em;line-height:1.5;margin:0\">\n<thead><tr style=\"border:0\"><th style=\"text-align:left;padding:8px 20px 8px 0;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b\">Component<\/th><th style=\"text-align:left;padding:8px 0 8px 0;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b\">What we ran<\/th><\/tr><\/thead>\n<tbody>\n<tr style=\"border:0\"><td style=\"padding:8px 20px 4px 0;vertical-align:top\">Tokkio app chart<\/td><td style=\"padding:8px 0 4px 0;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">tokkio-1stream-with-ui 5.0.0-GA<\/code>, one-click install, single node, bare metal<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">ACE Controller<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">5.0.0<\/code>, built on pipecat <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">0.0.68<\/code> <a href=\"#ref-2\" style=\"color:#4d7c0f\">[2]<\/a><\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">Audio2Face-3D (A2F-3D)<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">1.3.16<\/code> <a href=\"#ref-3\" style=\"color:#4d7c0f\">[3]<\/a><\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">Riva<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">2.19.0<\/code>, with the ja-JP streaming Conformer swapped in (section 7)<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">Unreal Renderer microservice<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">0.1.3<\/code><\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">Unreal Engine build<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">5.5.4<\/code>, Linux, Development configuration<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">Video Storage Toolkit (VST, also called VMS)<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">1.3.0<\/code><\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">Animation Graph<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">1.1.0<\/code><\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">Kubernetes<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\">kubeadm <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">1.31.2<\/code>, containerd <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">1.7.23<\/code> with the systemd cgroup driver on cgroup v2<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">GPU Operator<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">v24.9.2<\/code>, NVIDIA Container Toolkit <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">1.17.4<\/code>, Container Device Interface (CDI) off<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top\">Host<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\">Ubuntu 22.04 LTS, kernel 5.15, 2x NVIDIA RTX 6000 Ada Generation<\/td><\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n<p class=\"has-link-color wp-elements-3 wp-block-paragraph\">NVIDIA&#8217;s ACE plugins for Unreal are built for Unreal Engine 5.3 and must be rebuilt for other versions <a href=\"#ref-4\">[4]<\/a>. The renderer author rebuilt and modified them for our 5.5.4 build.<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-3\">3. Keep the ACE Controller, replace what sits behind it<\/h2>\n\n<p class=\"has-link-color wp-elements-4 wp-block-paragraph\">The ACE Controller, Riva, Audio2Face-3D, the Animation Graph, the renderer, Stream Distribution and Routing (SDR), VST and the UI all run from their stock images. The controller sends the answer audio to Audio2Face-3D, whose animation reaches the renderer through the Animation Graph <a href=\"#ref-5\">[5]<\/a>. We replaced what sits behind the controller: the LLM and the TTS (Figure 1).<\/p>\n\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;6adbd4801097f&quot;}\" data-wp-interactive=\"core\/image\" data-wp-key=\"6adbd4801097f\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1788\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on--click=\"actions.showLightbox\" data-wp-on--load=\"callbacks.setButtonStyles\" data-wp-on--pointerdown=\"actions.preloadImage\" data-wp-on--pointerenter=\"actions.preloadImageWithDelay\" data-wp-on--pointerleave=\"actions.cancelPreload\" data-wp-on-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig1-architecture-scaled.png\" alt=\"Architecture on one server: in the Tokkio cluster, the ACE Controller holds our LLM and TTS classes and reaches our gateway and TTS service on the host via a reverse proxy. The controller sends answer audio to A2F-3D, then animation and audio to the Animation Graph and renderer; video returns via VST to the browser.\" class=\"wp-image-703\" srcset=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig1-architecture-scaled.png 2560w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig1-architecture-300x210.png 300w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig1-architecture-1024x715.png 1024w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig1-architecture-2000x1397.png 2000w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig1-architecture-scaled-18x12.png 18w\" sizes=\"auto, (max-width: 2560px) 100vw, 2560px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\tdata-wp-bind--aria-label=\"state.thisImage.triggerButtonAriaLabel\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"state.thisImage.buttonRight\"\n\t\t\tdata-wp-style--top=\"state.thisImage.buttonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><figcaption class=\"wp-element-caption\">Figure 1. Runtime architecture on our single-node server, with our answer-path code in green: the LLM and TTS classes in the ACE Controller, and the provider gateway and TTS service on the host behind a reverse proxy that admits only the pod network and the host. Thick lines trace the answer audio, as PCM up to the renderer and as a video stream after it.<\/figcaption><\/figure>\n\n<p class=\"has-link-color wp-elements-5 wp-block-paragraph\">The controller is pipecat plus NVIDIA&#8217;s nvidia-pipecat extension <a href=\"#ref-6\">[6]<\/a>, with its code and config on a persistent volume claim mounted at <code>\/code<\/code>. NVIDIA documents <code>llm_processor<\/code> in <code>config.yaml<\/code> for the LLM, and edits to <code>bot.py<\/code> for other processors <a href=\"#ref-7\">[7]<\/a>. In practice, <code>llm_processor<\/code> accepts only the three stock class names, and the shipped <code>bot.py<\/code> builds the stock cloud TTS unconditionally; on an install without a key for it, you get transcripts with no voice. On the controller volume, we added our class name and a <code>tts_processor<\/code> key to <code>config.py<\/code>, and changed <code>bot.py<\/code> to build our classes and to fail loudly when their config section is missing. The same file also sets the Riva language and model, the voice activity detection (VAD) parameters, and the automatic speech recognition (ASR) word list.<\/p>\n\n<p class=\"wp-block-paragraph\">Our provider gateway and TTS service listen on the host&#8217;s loopback, which the controller pod cannot reach. A reverse proxy on the host forwards to them, admits only the pod network and the host itself, and turns buffering off for streaming. That is segmentation, not authentication. The gateway calls a cloud LLM API first, with local open-weight models served by Ollama (a 32B primary and a 7B standby) as the fallback. It emits one newline-delimited JSON (NDJSON) line per sentence, so TTS starts on the first sentence while the model is still writing. Our TTS service calls a cloud streaming TTS service.<\/p>\n\n<p class=\"wp-block-paragraph\">The controller volume sits outside git and review, so the startup check hashes each injected module against the release and checks that <code>bot.py<\/code> constructs each class; its first run found two drifted modules.<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-4\">4. Accept the packaged renderer as a runtime contract<\/h2>\n\n<p class=\"wp-block-paragraph\">The author of Part 1 also built the renderer package, including the stage actor, the facial mapping and the patches to NVIDIA&#8217;s Unreal plugins. The renderer author verified HTTP stream registration and removal and the face-animation receive path on an earlier Windows package with a mock Animation Graph. The earlier Linux package was verified up to the build and transfer integrity, and a later Linux build was started and rendered on a Linux GPU. This article covers the rest of the acceptance: running inside Tokkio on our server, real Audio2Face-3D output and dialogue, browser video and audio over WebRTC, and long runs.<\/p>\n\n<p class=\"wp-block-paragraph\">Each frame, the stage actor reads the interpolated ACE weights and maps the 52 ACE inputs to 62 animation curves. A dedicated AnimInstance outputs the curves through the animation proxy, and the character&#8217;s Reallusion Rig Logic post-process Animation Blueprint turns them into morph targets and jaw and eye bone poses.<\/p>\n\n<p class=\"has-link-color wp-elements-6 wp-block-paragraph\">Part 1&#8217;s <a href=\"https:\/\/www.dtc.teu.ac.jp\/cc5-tokkio-avatar\/#section-7\">section 7<\/a> maps the same 52 inputs to 54 CC5 targets <a href=\"#ref-1\">[1]<\/a>. Four mouth inputs gained outputs, and the output names follow the character&#8217;s facial profile.<\/p>\n\n\n<div style=\"overflow:auto;margin:20px 0 24px 0\">\n<table style=\"width:100%;border-collapse:collapse;font-size:0.93em;line-height:1.5;margin:0\">\n<thead><tr style=\"border:0\"><th style=\"text-align:left;padding:8px 20px 8px 0;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b\">ACE input<\/th><th style=\"text-align:left;padding:8px 20px 8px 0;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b\">Outputs in Part 1<\/th><th style=\"text-align:left;padding:8px 0 8px 0;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b\">Outputs in the stage build<\/th><\/tr><\/thead>\n<tbody>\n<tr style=\"border:0\"><td style=\"padding:8px 20px 4px 0;vertical-align:top;font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">MouthFunnel<\/td><td style=\"padding:8px 20px 4px 0;vertical-align:top\">1<\/td><td style=\"padding:8px 0 4px 0;vertical-align:top\">4, one per quadrant<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top;font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">MouthPucker<\/td><td style=\"padding:4px 20px 4px 0;vertical-align:top\">1<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\">4, one per quadrant<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top;font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">MouthRollLower<\/td><td style=\"padding:4px 20px 4px 0;vertical-align:top\">1<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\">2, left and right<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:4px 20px 4px 0;vertical-align:top;font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.92em\">MouthRollUpper<\/td><td style=\"padding:4px 20px 4px 0;vertical-align:top\">1<\/td><td style=\"padding:4px 0 4px 0;vertical-align:top\">2, left and right<\/td><\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">That adds eight outputs, so 54 becomes 62. Two inputs are approximations: MouthClose drives a lip-tightening shape, so closed lips are not guaranteed, and MouthShrugUpper maps to an approximate control. MouthShrugLower uses the character&#8217;s existing bone-driven pose. This mapping comes from an earlier renderer build. Later builds, the stage build among them, moved the same mapping into a new actor class. At startup the stage build&#8217;s log shows all 62 bindings resolved, which does not mean that every expression or real speech looks right.<\/p>\n\n<p class=\"wp-block-paragraph\">The per-frame step below is an excerpt from that earlier renderer build, with logging and verification code omitted.<\/p>\n\n\n<div style=\"background:#fafafa;border:1px solid #e4e4e7;border-radius:6px;margin:24px 0 28px 0;overflow:hidden\">\n<div style=\"padding:8px 14px;border-bottom:1px solid #e4e4e7;font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.78em;color:#71717a;line-height:1.5\">CC5PackagingStageActor.cpp, earlier renderer build (excerpt; logging and verification code omitted)<\/div>\n<pre style=\"margin:0;padding:14px 16px;overflow:auto;background:#fafafa;border:0;border-radius:0;font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.84em;line-height:1.55;color:#18181b;white-space:pre\">void ACC5PackagingStageActor::Tick(float DeltaSeconds)\n{\n    Super::Tick(DeltaSeconds);\n    ApplyACECurves();\n}\n\nvoid ACC5PackagingStageActor::ApplyACECurves()\n{\n    if (!ACECurveSource || !CharacterMesh)\n        return;\n    CachedACECurveWeights.Reset();\n    ACECurveSource-&gt;GetCurveOutputsInterp(CachedACECurveWeights);\n    if (CachedACECurveWeights.Num() &lt; RequiredACECurveCount)\n    {\n        if (bHadActiveACECurves)\n            ResetACEMorphTargets();\n        return;\n    }\n    auto* FaceAnimation =\n        Cast&lt;UCC5ACEAnimInstance&gt;(CharacterMesh-&gt;GetAnimInstance());\n    if (!FaceAnimation)\n        return;\n    for (const FACEToCC5MorphBinding&amp; Binding : ACEToCC5MorphBindings)\n    {\n        const float Weight = FMath::Clamp(\n            CachedACECurveWeights[Binding.CurveIndex], 0.0f, 1.0f);\n        FaceAnimation-&gt;SetFaceCurve(Binding.MorphTarget, Weight);\n    }\n    bHadActiveACECurves = true;\n}<\/pre>\n<\/div>\n\n\n<p class=\"has-link-color wp-elements-7 wp-block-paragraph\">At runtime, the actor that the stream helper&#8217;s <code>GetActor()<\/code> returns must carry a <code>UACEAudioCurveSourceComponent<\/code> <a href=\"#ref-4\">[4]<\/a>. Without it, the build looks healthy but answers SDR&#8217;s <code>add_stream<\/code> with HTTP 400 and &#8220;Failed to subscribe character to stream!&#8221;, and the browser gets audio without video until the UI reports <code>VST_INBOUND_STREAM_ERROR<\/code> <a href=\"#ref-8\">[8]<\/a>.<\/p>\n\n<p class=\"wp-block-paragraph\">A build can start cleanly and register with the signalling server, then still fail at the first subscription or after roughly a day of uptime.<\/p>\n\n\n<div style=\"overflow:auto;margin:20px 0 28px 0\">\n<table style=\"width:100%;border-collapse:collapse;font-size:0.93em;line-height:1.5;margin:0\">\n<thead><tr style=\"border:0\"><th style=\"text-align:left;padding:8px 20px 8px 0;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b;width:20%\">Step<\/th><th style=\"text-align:left;padding:8px 20px 8px 0;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b;width:42%\">What we check<\/th><th style=\"text-align:left;padding:8px 0 8px 0;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b\">What failure looks like<\/th><\/tr><\/thead>\n<tbody>\n<tr style=\"border:0\"><td style=\"padding:8px 20px 6px 0;vertical-align:top\">Startup log checks<\/td><td style=\"padding:8px 20px 6px 0;vertical-align:top\">Log lines for the validated mapping, the actor registered with its audio curve source and the frame-rate cap, and the setting <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.9em\">r.GPUStatsEnabled=0<\/code> in effect. Then port 8021 <a href=\"#ref-9\" style=\"color:#4d7c0f\">[9]<\/a> is listening and answers <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.9em\">GET \/health<\/code> from inside the cluster.<\/td><td style=\"padding:8px 0 6px 0;vertical-align:top\">A line is missing, or port 8021 does not answer while the pod stays Running.<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:6px 20px 8px 0;vertical-align:top\">Soak<\/td><td style=\"padding:6px 20px 8px 0;vertical-align:top\">12 hours or more, sampled every 30 minutes: container restarts, the GPU memory trend, and name-table (FName) growth while idle.<\/td><td style=\"padding:6px 0 8px 0;vertical-align:top\">Sustained FName growth is a failure signal to investigate. The earlier long-run crash has not been reproduced and attributed to this cause.<\/td><\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">For FName growth, the renderer author traced a per-frame name to Unreal Engine 5.5&#8217;s realtime GPU profiler. The growth reproduced while idle and stopped in a short test with <code>r.GPUStatsEnabled=0<\/code>. That does not show it caused the earlier crash, so we also recreate the renderer pod before each show day.<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-5\">5. Two pipecat details that decide what the avatar remembers<\/h2>\n\n<p class=\"wp-block-paragraph\">Two behaviors of pipecat 0.0.68 decide what the model later sees as its own answers.<\/p>\n\n<p class=\"has-link-color wp-elements-8 wp-block-paragraph\">The first is a contract. A TTS service forwards <code>LLMFullResponseEndFrame<\/code> only when <code>push_text_frames<\/code> is on; otherwise the &#8220;subclass must do it&#8221; <a href=\"#ref-10\">[10]<\/a>. Our TTS class turns <code>push_text_frames<\/code> off because <code>run_tts<\/code> emits its own text frames, so the end frame was dropped and the assistant context aggregator never got its designed commit point.<\/p>\n\n<p class=\"wp-block-paragraph\">We reproduced this in the controller pod on 2026-09-04 with a reduced pipeline: three uninterrupted turns, one run per variant. All three answers stayed uncommitted. Forwarding the frame once, after synthesis, committed all three. From the pipecat code, the full pipeline&#8217;s aggregator also commits when the bot stops speaking, including between sentences, so without the fix commits follow speaking gaps rather than the ends of answers. We did not isolate this on the live stack.<\/p>\n\n\n<div style=\"background:#fafafa;border:1px solid #e4e4e7;border-radius:6px;margin:24px 0 28px 0;overflow:hidden\">\n<div style=\"padding:8px 14px;border-bottom:1px solid #e4e4e7;font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.78em;color:#71717a;line-height:1.5\">custom_tts.py (excerpt)<\/div>\n<pre style=\"margin:0;padding:14px 16px;overflow:auto;background:#fafafa;border:0;border-radius:0;font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.84em;line-height:1.55;color:#18181b;white-space:pre\">from pipecat.frames.frames import LLMFullResponseEndFrame, TTSSpeakFrame\nfrom pipecat.services.tts_service import TTSService  # pipecat 0.0.68\n\n\nclass CustomTTSService(TTSService):\n    async def process_frame(self, frame, direction):\n        # Filler lines ride TTSSpeakFrame and were never counted as queued\n        # model sentences, so their synthesis must not dequeue one.\n        if isinstance(frame, TTSSpeakFrame):\n            self._skip_dequeue = True\n            try:\n                await super().process_frame(frame, direction)\n            finally:\n                self._skip_dequeue = False\n            return\n        await super().process_frame(frame, direction)\n        # With push_text_frames=False, 0.0.68 flushes the last sentence on\n        # this frame but does not forward it. The assistant context\n        # aggregator commits the answer on it, so re-emit it once, after\n        # the base class has synthesized, so audio precedes the commit.\n        # pipecat &gt;= 0.0.108 emits it in the base class: drop this branch there.\n        if isinstance(frame, LLMFullResponseEndFrame):\n            await self.push_frame(frame, direction)<\/pre>\n<\/div>\n\n\n\n<div style=\"border-left:3px solid #4d7c0f;padding:2px 0 2px 16px;margin:24px 0 28px 0\">\n<p style=\"margin:0;line-height:1.65\"><strong style=\"color:#18181b\">Note.<\/strong> From pipecat 0.0.108 (pull request #4145), the base class emits the end frame itself <a href=\"#ref-11\" style=\"color:#4d7c0f\">[11]<\/a>. On those versions a turn with no audio still loses the frame, an issue still open on 2026-09-28 <a href=\"#ref-12\" style=\"color:#4d7c0f\">[12]<\/a>; with this override on 0.0.68, the frame is forwarded whether or not the turn produced audio.<\/p>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">The second behavior is how a barge-in is filed. On an interruption, pipecat commits the sentences whose text frame already reached the aggregator and drops the rest, which we reproduced in the pod on 2026-09-04, in one run. So our TTS service yields each sentence&#8217;s text frame after its audio has fully streamed out, and a sentence cut mid-audio is not filed as said. Precision is one sentence, because the service returns audio without word timestamps. The next request&#8217;s history marks a cut answer with a prefix meaning &#8220;answer cut off by the listener&#8221;.<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-6\">6. Design against dead air<\/h2>\n\n<p class=\"wp-block-paragraph\">Five seconds of silence inside a turn was our hard limit, the dead-air limit. Every design choice was checked against it (Figure 2).<\/p>\n\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;6adbd480110de&quot;}\" data-wp-interactive=\"core\/image\" data-wp-key=\"6adbd480110de\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1646\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on--click=\"actions.showLightbox\" data-wp-on--load=\"callbacks.setButtonStyles\" data-wp-on--pointerdown=\"actions.preloadImage\" data-wp-on--pointerenter=\"actions.preloadImageWithDelay\" data-wp-on--pointerleave=\"actions.cancelPreload\" data-wp-on-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig2-turn-timeline-scaled.png\" alt=\"Timeline with the dead-air limit at 5 s. After the configured 1.2 s VAD delay, our 2026-09-18 medians are 0.89 s to the LLM first token in the gateway (n=19) and 1.07 s to the first sentence at our LLM service in the controller (n=25). A filler line is due 1.2 s into the turn; other segments carry no figure.\" class=\"wp-image-704\" srcset=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig2-turn-timeline-scaled.png 2560w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig2-turn-timeline-300x193.png 300w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig2-turn-timeline-1024x658.png 1024w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig2-turn-timeline-2000x1286.png 2000w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig2-turn-timeline-scaled-18x12.png 18w\" sizes=\"auto, (max-width: 2560px) 100vw, 2560px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\tdata-wp-bind--aria-label=\"state.thisImage.triggerButtonAriaLabel\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"state.thisImage.buttonRight\"\n\t\t\tdata-wp-style--top=\"state.thisImage.buttonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><figcaption class=\"wp-element-caption\">Figure 2. One turn against the 5 s dead-air limit: the VAD bar is the configured stop_secs, and the dark bars are medians timed inside our own components on 2026-09-18 (n=19 and n=25), small-sample observations rather than benchmarks. Segments without a figure were not measured or not isolated.<\/figcaption><\/figure>\n\n<p class=\"wp-block-paragraph\">The two measured series in Figure 2 start at different points and do not add up. The first-sentence series can also read low, because a barge-in before the first sentence stops its clock. We never measured microphone to speaker.<\/p>\n\n<p class=\"wp-block-paragraph\">A filler watchdog in our LLM service speaks the first filler line <code>time_delay<\/code> seconds after the last text sent to TTS; we ran it at 1.2 s. It then speaks one more for every further four seconds of silence, at most four per turn, and stops at the first model sentence.<\/p>\n\n<p class=\"wp-block-paragraph\">A filler line plays before the model has said anything, so it must not assert anything. In testing,\u300c\u3048\u3048\u3001\u3048\u3048\u300d(&#8220;yeah, yeah&#8221;) after a yes-or-no question was a false &#8220;yes&#8221;. We deleted six agreeing lines and keyed the filler pools by the listener&#8217;s intent, which deterministic suffix rules classify without a model call. Questions never hear an acknowledgement, statements never hear &#8220;good question&#8221;, and unknown intent gets neutral interjections. A misclassification can add an acknowledgement or a &#8220;good question&#8221;, never an answer; we have not measured how often one happens. Keep <code>filler: []<\/code> rather than deleting it, or the stock English fillers come back.<\/p>\n\n<p class=\"wp-block-paragraph\">LLM failover is allowed only inside the switch window, from turn start until the model&#8217;s own first sentence leaves the gateway. Filler lines do not close it. After it closes, the provider is sticky for the turn. An LLM stream over server-sent events (SSE) succeeds only with <code>[DONE]<\/code> or a <code>finish_reason<\/code>; a stream that just stops, or an empty answer, is a failure. With an invalid cloud key (2026-09-02, n=2 turns), the local 32B model answered with fallback reason <code>auth<\/code>.<\/p>\n\n\n<figure class=\"wp-block-video\"><video height=\"720\" style=\"aspect-ratio: 1280 \/ 720;\" width=\"1280\" controls poster=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/turn-dead-air-poster.png\" src=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/turn-dead-air.mp4\" playsinline><track src=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/turn-dead-air.en_.vtt\" label=\"English\" srclang=\"en\" kind=\"subtitles\" default \/><track src=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/turn-dead-air.ja_.vtt\" label=\"\u65e5\u672c\u8a9e\" srclang=\"ja\" kind=\"subtitles\" \/><track src=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/turn-dead-air.zh_.vtt\" label=\"\u7b80\u4f53\u4e2d\u6587\" srclang=\"zh-Hans\" kind=\"subtitles\" \/><\/video><figcaption class=\"wp-element-caption\">Video 1. One turn against the 5 s dead-air limit, in three illustrative scenes with AI-generated narration: a fast turn, a slow turn in which filler lines cover the wait while the gateway fails over to a fallback model inside the switch window, and a barge-in. Only the labeled values are configured (VAD <code>stop_secs<\/code> 1.2 s, filler <code>time_delay<\/code> 1.2 s, then every 4 s, at most 4 per turn); every other duration is illustrative, not measured.<\/figcaption><\/figure>\n\n\n<p class=\"wp-block-paragraph\">Video 1&#8217;s narration is AI-generated with a stock voice of an open-source text-to-speech model; it is not the avatar&#8217;s voice, and the system itself does not use that model. In the slow turn, no silence reaches 5 s, but the primary fails fast. If it hung until its timeout instead, the wait would outlast all four filler lines, so the watchdog alone does not bound dead air. In the barge-in, only sentences whose audio had fully streamed out of the TTS service are kept, and the next request marks the answer as cut off by the listener (section 5).<\/p>\n\n<p class=\"wp-block-paragraph\">Reasoning models are banned, because their <code>content<\/code> stays empty while they think. OpenAI&#8217;s <code>gpt-5.6-terra<\/code> passed our admission check in 3 runs on 2026-08-17, each with a prompt first content delta, no reasoning deltas and a clean finish.<\/p>\n\n<p class=\"wp-block-paragraph\">Over two days of a live event in September 2026, rehearsals and staff tests included, the gateway&#8217;s log recorded 405 conversation requests, all answered with HTTP 200, and no warning or error entry. That does not prove that every turn completed or that no failover happened: an empty answer can fail over without a warning, and the log keeps no per-turn outcome.<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-7\">7. Make Japanese speech sound right<\/h2>\n\n<p class=\"has-link-color wp-elements-9 wp-block-paragraph\">Switching Riva from English to the Japanese streaming Conformer <a href=\"#ref-13\">[13]<\/a> takes two coordinated edits. The model argument of the <code>triton0<\/code> Deployment, in both the main and the model-init containers, becomes <code>rmir_asr_conformer_unified_ja_jp_str:2.19.0<\/code>, and the ASR service in <code>bot.py<\/code> gets <code>language=\"ja-JP\"<\/code> and the model <code>conformer-unified-ja-JP-asr-streaming-asr-bls-ensemble<\/code>. Apply triton0 first, wait for Ready, restart riva-speech, then edit <code>bot.py<\/code> and recreate the controller pod. Copy the old model out first, because a failed deploy empties the model directory.<\/p>\n\n<p class=\"has-link-color wp-elements-10 wp-block-paragraph\">We pass 36 event-specific names and terms as boosted words <a href=\"#ref-14\">[14]<\/a> at nvidia-pipecat&#8217;s default score of 4.0; their effect is unmeasured. Room noise was sometimes finalized as short English words such as &#8220;ok&#8221;. Now a pure-ASCII final of at most 12 characters, or a single breath-like kana, never becomes a turn or a history entry. The cost is that a genuine &#8220;NVIDIA&#8221; is dropped too.<\/p>\n\n<p class=\"wp-block-paragraph\">Our prompts forbid Arabic numerals, so the model writes years like \u4e8c\u3007\u4e8c\u516d\u5e74, and the cloud TTS read them digit by digit. The TTS service rewrites three or four kanji digits before \u5e74 into positional form, \u4e8c\u5343\u4e8c\u5341\u516d\u5e74. The spoken sentence lasted 4.29 s in digit form and 3.44 s in positional form (one clip each, 2026-09-04).<\/p>\n\n<p class=\"wp-block-paragraph\">Other words go through a reading table. An entry needs a person to hear the misreading, and then to hear the kana rewrite read correctly, each on two separate samples, because the cloud TTS is not deterministic. Two samples were not always enough: the rewrite for the character&#8217;s own name passed and then misread on the deployed stack.<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-8\">8. Fix the video path, not the skin<\/h2>\n\n<p class=\"wp-block-paragraph\">In the browser, the avatar looked darker than the renderer&#8217;s frames, most of all in the mid-tones, and the shift looked like a gamma curve, not an offset (Figure 3). We measured this on 2026-09-17, on one client and two builds that share the shipped binary.<\/p>\n\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;6adbd4801176e&quot;}\" data-wp-interactive=\"core\/image\" data-wp-key=\"6adbd4801176e\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1243\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on--click=\"actions.showLightbox\" data-wp-on--load=\"callbacks.setButtonStyles\" data-wp-on--pointerdown=\"actions.preloadImage\" data-wp-on--pointerenter=\"actions.preloadImageWithDelay\" data-wp-on--pointerleave=\"actions.cancelPreload\" data-wp-on-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig4-tone-curve-scaled.png\" alt=\"Illustrative chart, not measured data, of luma seen in the browser against reference luma before compensation. The observed curve sags below the identity diagonal in the mid-tones and changes little near black and white. The intended compensation, r.TonemapperGamma=2.75, is drawn close to the diagonal.\" class=\"wp-image-705\" srcset=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig4-tone-curve-scaled.png 2560w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig4-tone-curve-300x146.png 300w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig4-tone-curve-1024x497.png 1024w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig4-tone-curve-2000x971.png 2000w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig4-tone-curve-scaled-18x9.png 18w\" sizes=\"auto, (max-width: 2560px) 100vw, 2560px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\tdata-wp-bind--aria-label=\"state.thisImage.triggerButtonAriaLabel\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"state.thisImage.buttonRight\"\n\t\t\tdata-wp-style--top=\"state.thisImage.buttonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><figcaption class=\"wp-element-caption\">Figure 3. Illustrative, not measured data: in the browser, mid-tones came out darker than the renderer produced, while black and white moved much less. Raising r.TonemapperGamma to 2.75 brightens the renderer&#8217;s output to offset that darkening; we accepted the result by eye, without a measurement after the change.<\/figcaption><\/figure>\n\n<p class=\"wp-block-paragraph\">The renderer was not the cause: its Linux render matched the Windows render of the same build and the editor viewport. We did not split the shift between VST and the client. Brightening the skin was the wrong fix: doubling the face&#8217;s base color multiplier washed the face toward white.<\/p>\n\n<p class=\"has-link-color wp-elements-11 wp-block-paragraph\">The shipped fix is the startup argument <code>-ini:Engine:[ConsoleVariables]:r.TonemapperGamma=2.75<\/code>, and we accepted the result by eye. Resolution and encoder are env vars of the renderer StatefulSet: <code>IAUEMS_WINDOW_WIDTH<\/code>, <code>IAUEMS_WINDOW_HEIGHT<\/code> and <code>IAUEMS_DEPLOYMENT_ADDITIONAL_STARTUP_ARGS<\/code>, which also carries the gamma flag <a href=\"#ref-9\">[9]<\/a>. The chart default is 1280&#215;720 with VP8, and ours was 1920&#215;1080. A helm upgrade would re-render them; our startup check prints the <code>kubectl set env<\/code> command on drift.<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-9\">9. When GPU pods silently lose the GPU<\/h2>\n\n<p class=\"has-link-color wp-elements-12 wp-block-paragraph\">Tokkio 5.0&#8217;s GPU pods (VST, Audio2Face-3D, Triton, Riva, the Animation Graph and the renderer) get their GPU through the NVIDIA runtime and <code>NVIDIA_VISIBLE_DEVICES<\/code>, set in the pod spec for four of them; Riva and the Animation Graph set none and see every GPU. None of them requests <code>nvidia.com\/gpu<\/code>. Our node ran containerd with the systemd cgroup driver on cgroup v2, the toolkit&#8217;s legacy hook, and CDI off. In that setup the GPU grant exists only in device rules that the NVIDIA hook writes behind systemd&#8217;s back <a href=\"#ref-15\">[15]<\/a>. Any <code>systemctl daemon-reload<\/code> rebuilds the rules without it <a href=\"#ref-16\">[16]<\/a>. Our reloads came from the daily package-update timer and snap refreshes.<\/p>\n\n\n<div style=\"border-left:3px solid #4d7c0f;background:#f7fee7;padding:12px 16px;margin:24px 0 28px 0;border-radius:0 6px 6px 0\">\n<p style=\"margin:0;line-height:1.65\"><strong style=\"color:#18181b\">Caution.<\/strong> Disabling the timer that causes the reloads is itself a reload, because <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.88em;background:#ffffff;padding:1px 5px;border-radius:6px\">systemctl disable<\/code> and <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.88em;background:#ffffff;padding:1px 5px;border-radius:6px\">enable<\/code> reload the systemd daemon. Do it only while the Tokkio workloads are scaled to zero.<\/p>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">Nothing breaks at reload time: the kernel checks device permission only at open, so open handles keep working and every pod stays Running (Figure 4). The next <code>open()<\/code> fails, with a misleading error. VST opens a new hardware encoder per browser session, so the first session after a reload crashed it, repeatedly. Its log blamed <code>\/dev\/v4l2-nvenc<\/code>, but on x86 that library loops over <code>\/dev\/nvidia0<\/code> to <code>\/dev\/nvidia15<\/code>. We never caught the permission error at the crash instant, so this is our best-supported explanation.<\/p>\n\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;6adbd48011b51&quot;}\" data-wp-interactive=\"core\/image\" data-wp-key=\"6adbd48011b51\" class=\"wp-block-image size-full wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1584\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on--click=\"actions.showLightbox\" data-wp-on--load=\"callbacks.setButtonStyles\" data-wp-on--pointerdown=\"actions.preloadImage\" data-wp-on--pointerenter=\"actions.preloadImageWithDelay\" data-wp-on--pointerleave=\"actions.cancelPreload\" data-wp-on-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig5-gpu-access-scaled.png\" alt=\"Causal chain: a systemd daemon-reload drops the GPU grants of pods without an nvidia.com\/gpu request; they stay Running until the next open() fails. Our fixes: reload only at zero replicas, else recreate the six GPU pods, and probe devices at startup. A resource request that passes device specifications, or CDI, was designed, not deployed.\" class=\"wp-image-706\" srcset=\"https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig5-gpu-access-scaled.png 2560w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig5-gpu-access-300x186.png 300w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig5-gpu-access-1024x633.png 1024w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig5-gpu-access-2000x1237.png 2000w, https:\/\/www.dtc.teu.ac.jp\/wp-content\/uploads\/2026\/09\/fig5-gpu-access-scaled-18x12.png 18w\" sizes=\"auto, (max-width: 2560px) 100vw, 2560px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\tdata-wp-bind--aria-label=\"state.thisImage.triggerButtonAriaLabel\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"state.thisImage.buttonRight\"\n\t\t\tdata-wp-style--top=\"state.thisImage.buttonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><figcaption class=\"wp-element-caption\">Figure 4. How a GPU pod silently loses its device: a systemd daemon-reload drops the device grants of pods that get their GPU through NVIDIA_VISIBLE_DEVICES rather than an nvidia.com\/gpu request, and nothing fails until the next open(). The green boxes are what we did; the dashed box is the structural fix we designed but did not deploy.<\/figcaption><\/figure>\n\n<p class=\"has-link-color wp-elements-13 wp-block-paragraph\">On 2026-09-18 we disabled the apt timers with the stage up, and five of six GPU pods lost access. Our rule is that anything that reloads systemd runs at zero replicas. If a reload slips through, freeze its source and delete all GPU pods. The startup check opens every <code>\/dev\/nvidia*<\/code> inside each GPU pod and counts reloads since VST started. The structural fixes we designed were an <code>nvidia.com\/gpu<\/code> request with the device plugin set to pass device specifications, or CDI, so that the devices enter the container specification <a href=\"#ref-15\">[15]<\/a>; we judged both too risky before a show and deployed neither.<\/p>\n\n<p class=\"wp-block-paragraph\">A manual reboot into a new kernel added a race. The kubelet restarted the Tokkio pods before the toolkit DaemonSet had rewritten containerd&#8217;s config, so their sandboxes had no <code>\/dev\/nvidia*<\/code>. CrashLoop restarts containers, not sandboxes, so wait for the toolkit DaemonSet to report Ready, then delete the pods.<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-10\">10. Common problems and where to fix them<\/h2>\n\n<p class=\"wp-block-paragraph\">Part 1 closed with a table of asset symptoms. This is the runtime counterpart.<\/p>\n\n\n<div style=\"overflow:auto;margin:20px 0 28px 0\">\n<table style=\"width:100%;border-collapse:collapse;font-size:0.93em;line-height:1.5;margin:0\">\n<thead><tr style=\"border:0\"><th style=\"text-align:left;padding:8px 20px 8px 10px;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b;width:44%\">Symptom<\/th><th style=\"text-align:left;padding:8px 10px 8px 0;border-bottom:1px solid #d4d4d8;font-weight:700;color:#18181b\">Where to fix<\/th><\/tr><\/thead>\n<tbody>\n<tr style=\"border:0\"><td colspan=\"2\" style=\"padding:8px 10px;background:#fafafa;font-weight:700\">Renderer and video<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:8px 20px 6px 10px;vertical-align:top\"><code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.9em\">add_stream<\/code> returns 400; audio but no video<\/td><td style=\"padding:8px 10px 6px 0;vertical-align:top\">The actor from <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.9em\">GetActor()<\/code> lacks <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.9em\">UACEAudioCurveSourceComponent<\/code>. Fix the Unreal build.<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:6px 20px 8px 10px;vertical-align:top\">Pod Running, every session blank after a renderer crash<\/td><td style=\"padding:6px 10px 8px 0;vertical-align:top\">The engine can fail to rebind port 8021. Delete the pod, and probe <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.9em\">:8021\/health<\/code> from inside the cluster.<\/td><\/tr>\n<tr style=\"border:0\"><td colspan=\"2\" style=\"padding:8px 10px;background:#fafafa;font-weight:700;border-top:1px solid #e4e4e7\">GPU and cluster<\/td><\/tr>\n<tr style=\"border:0\"><td style=\"padding:8px 20px 8px 10px;vertical-align:top\">VST crashes on the first session after host maintenance; its log blames <code style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;font-size:0.9em\">\/dev\/v4l2-nvenc<\/code><\/td><td style=\"padding:8px 10px 8px 0;vertical-align:top\">A daemon-reload revoked GPU access. Reload only at zero replicas; otherwise delete all GPU pods (section 9).<\/td><\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\" id=\"section-11\">11. What we have not measured<\/h2>\n\n<p class=\"wp-block-paragraph\">We never measured microphone-to-speaker latency, ASR character error rate, what word boosting changed, or how often the filler intent rules misclassify. We have not verified that playback and Audio2Face-3D receive the same PCM on the Tokkio path. The exact renderer binary we shipped has no continuous 12-hour soak on file. For the event, the controller logs that feed our latency report were lost when the stage was scaled to zero.<\/p>\n\n<h2 class=\"wp-block-heading\" id=\"section-next\">Where this leaves the series<\/h2>\n\n<p class=\"has-link-color wp-elements-14 wp-block-paragraph\">Part 1 ended with a hand-off package <a href=\"#ref-1\">[1]<\/a>, and this article took that package onto a running stage. Part 1 owns the asset and its facial contract; this part owns the runtime around it. The ACE runtime connection is in sections 3 and 4, and the renderer actor&#8217;s per-frame path is in section 4, seen from the integration side. The camera component inside the Unreal renderer comes with the packaged build. VST, which handles the browser&#8217;s camera and audio streams, is in sections 8 and 9. Write down what ready means, check it on every start, and keep the evidence.<\/p>\n\n\n<div id=\"section-references\" style=\"border-top:1px solid #e4e4e7;margin:40px 0 0 0;padding:16px 0 0 0\">\n<p style=\"margin:0 0 4px 0;font-weight:700;font-size:1.05em;color:#18181b\">References<\/p>\n<p style=\"margin:0 0 16px 0;font-size:0.85em;color:#71717a\">Links checked September 2026. Tokkio and ACE documentation links point to the archive site; the Riva and Container Toolkit links point to current documentation, not to the versions we ran.<\/p>\n<p id=\"ref-1\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[1]<\/span> DTC Tech Blog. <a href=\"https:\/\/www.dtc.teu.ac.jp\/cc5-tokkio-avatar\/\" style=\"color:#4d7c0f;text-decoration:none\">Creating a Custom Avatar for NVIDIA Tokkio with Character Creator 5 and Unreal Engine \u2192<\/a><br><span style=\"color:#52525b\">Part 1 of this series: the hand-off package and, in its section 7, 52 ACE inputs mapped to 54 CC5 targets.<\/span><\/p>\n<p id=\"ref-2\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[2]<\/span> NVIDIA. <a href=\"https:\/\/archive.docs.nvidia.com\/ace\/tokkio\/5.0\/release-notes\/release-notes-5.0-ga.html\" style=\"color:#4d7c0f;text-decoration:none\">Tokkio 5.0 GA release notes \u2192<\/a><br><span style=\"color:#52525b\">Tokkio 5.0 GA uses pipecat 0.0.68.<\/span><\/p>\n<p id=\"ref-3\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[3]<\/span> NVIDIA. <a href=\"https:\/\/archive.docs.nvidia.com\/ace\/audio2face-3d-microservice\/latest\/text\/changelog.html\" style=\"color:#4d7c0f;text-decoration:none\">Audio2Face-3D Microservice: Release Notes \u2192<\/a><br><span style=\"color:#52525b\">Includes the 1.3.16 entry.<\/span><\/p>\n<p id=\"ref-4\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[4]<\/span> NVIDIA. <a href=\"https:\/\/archive.docs.nvidia.com\/ace\/unreal-renderer-microservice\/latest\/avatar-customization\/fully-custom-avatar-workflow.html\" style=\"color:#4d7c0f;text-decoration:none\">Unreal Renderer Microservice: Custom Avatar, Arbitrary Asset \u2192<\/a><br><span style=\"color:#52525b\">ACE plugins written and built for Unreal Engine 5.3; an ACEAudioCurveSource on the avatar actor.<\/span><\/p>\n<p id=\"ref-5\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[5]<\/span> NVIDIA. <a href=\"https:\/\/archive.docs.nvidia.com\/ace\/tokkio\/5.0\/overview\/architecture.html\" style=\"color:#4d7c0f;text-decoration:none\">Tokkio 5.0: Architecture \u2192<\/a><br><span style=\"color:#52525b\">The answer path from Audio2Face-3D through the Animation Graph to the renderer.<\/span><\/p>\n<p id=\"ref-6\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[6]<\/span> NVIDIA. <a href=\"https:\/\/archive.docs.nvidia.com\/ace\/ace-controller-microservice\/latest\/user-guide.html\" style=\"color:#4d7c0f;text-decoration:none\">ACE Controller Microservice: User Guide \u2192<\/a><br><span style=\"color:#52525b\">The controller as pipecat plus the nvidia-pipecat extension.<\/span><\/p>\n<p id=\"ref-7\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[7]<\/span> NVIDIA. <a href=\"https:\/\/archive.docs.nvidia.com\/ace\/tokkio\/5.0\/customization\/customization-options.html\" style=\"color:#4d7c0f;text-decoration:none\">Tokkio 5.0: Customization Options \u2192<\/a><br><span style=\"color:#52525b\">llm_processor in config.yaml with three stock LLM services; other processors, TTS included, in bot.py.<\/span><\/p>\n<p id=\"ref-8\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[8]<\/span> NVIDIA. <a href=\"https:\/\/archive.docs.nvidia.com\/ace\/tokkio\/5.0\/microservices\/ui\/troubleshooting.html\" style=\"color:#4d7c0f;text-decoration:none\">Tokkio 5.0: Tokkio UI Troubleshooting \u2192<\/a><br><span style=\"color:#52525b\">VST_INBOUND_STREAM_ERROR: the UI cannot connect to the inbound avatar stream; TURN problems are the most common cause.<\/span><\/p>\n<p id=\"ref-9\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[9]<\/span> NVIDIA. <a href=\"https:\/\/archive.docs.nvidia.com\/ace\/unreal-renderer-microservice\/latest\/parameters.html\" style=\"color:#4d7c0f;text-decoration:none\">Unreal Renderer Microservice: Parameters \u2192<\/a><br><span style=\"color:#52525b\">HTTP API port 8021; IAUEMS_WINDOW_WIDTH and IAUEMS_WINDOW_HEIGHT, 1280 and 720 by default; additional startup arguments.<\/span><\/p>\n<p id=\"ref-10\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[10]<\/span> pipecat. <a href=\"https:\/\/github.com\/pipecat-ai\/pipecat\/blob\/v0.0.68\/src\/pipecat\/services\/tts_service.py\" style=\"color:#4d7c0f;text-decoration:none\">tts_service.py at v0.0.68 \u2192<\/a><br><span style=\"color:#52525b\">With push_text_frames off, the subclass must push LLMFullResponseEndFrame.<\/span><\/p>\n<p id=\"ref-11\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[11]<\/span> pipecat. <a href=\"https:\/\/github.com\/pipecat-ai\/pipecat\/pull\/4145\" style=\"color:#4d7c0f;text-decoration:none\">Pull request #4145: TTS improvements \u2192<\/a><br><span style=\"color:#52525b\">From 0.0.108, the base class emits the end frame for turns with audio.<\/span><\/p>\n<p id=\"ref-12\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[12]<\/span> pipecat. <a href=\"https:\/\/github.com\/pipecat-ai\/pipecat\/issues\/5654\" style=\"color:#4d7c0f;text-decoration:none\">Issue #5654: TTSService with push_text_frames=False drops the LLMFullResponseEndFrame of any turn that produces no audio \u2192<\/a><br><span style=\"color:#52525b\">Open as of 2026-09-28.<\/span><\/p>\n<p id=\"ref-13\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[13]<\/span> NVIDIA. <a href=\"https:\/\/docs.nvidia.com\/deeplearning\/riva\/user-guide\/docs\/asr\/asr-overview.html\" style=\"color:#4d7c0f;text-decoration:none\">Riva ASR Overview \u2192<\/a><br><span style=\"color:#52525b\">Language table including Japanese (ja-JP) Conformer models.<\/span><\/p>\n<p id=\"ref-14\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[14]<\/span> NVIDIA. <a href=\"https:\/\/docs.nvidia.com\/deeplearning\/riva\/user-guide\/docs\/asr\/asr-customizing.html\" style=\"color:#4d7c0f;text-decoration:none\">Basics of Speech Recognition and Customization of Riva ASR \u2192<\/a><br><span style=\"color:#52525b\">Word boosting at request time.<\/span><\/p>\n<p id=\"ref-15\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[15]<\/span> NVIDIA. <a href=\"https:\/\/docs.nvidia.com\/datacenter\/cloud-native\/container-toolkit\/latest\/troubleshooting.html\" style=\"color:#4d7c0f;text-decoration:none\">NVIDIA Container Toolkit: Troubleshooting \u2192<\/a><br><span style=\"color:#52525b\">A daemon-reload under systemd-managed cgroups drops GPU access that the legacy hook granted; explicit device nodes or CDI prevent this.<\/span><\/p>\n<p id=\"ref-16\" style=\"margin:0 0 14px 0;padding:0 0 0 12px;border-left:2px solid #4d7c0f;font-size:0.92em;line-height:1.55\"><span style=\"font-family:&apos;JetBrains Mono&apos;,&apos;SFMono-Regular&apos;,Menlo,Consolas,monospace;color:#71717a\">[16]<\/span> NVIDIA. <a href=\"https:\/\/github.com\/NVIDIA\/gpu-operator\/issues\/503\" style=\"color:#4d7c0f;text-decoration:none\">gpu-operator issue #503: Kubernetes containers using NVIDIA_VISIBLE_DEVICES lose device access after systemctl daemon-reload \u2192<\/a><br><span style=\"color:#52525b\">Containers using NVIDIA_VISIBLE_DEVICES lose GPU access on daemon-reload; pods with resource requests keep it.<\/span><\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"<p>How we took the Character Creator 5 avatar from Part 1, packaged as a Linux renderer build, onto a Tokkio 5.0 stage: the renderer&#8217;s runtime contract, a replaced LLM and TTS behind the stock ACE Controller, Japanese speech fixes, and the GPU and video failures we met. Every checkpoint names the log or measurement behind it.<\/p>","protected":false},"author":5,"featured_media":717,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45],"tags":[],"class_list":["post-714","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech-blog"],"_links":{"self":[{"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/posts\/714","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/comments?post=714"}],"version-history":[{"count":4,"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/posts\/714\/revisions"}],"predecessor-version":[{"id":721,"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/posts\/714\/revisions\/721"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/media\/717"}],"wp:attachment":[{"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/media?parent=714"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/categories?post=714"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dtc.teu.ac.jp\/ja\/wp-json\/wp\/v2\/tags?post=714"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}